Ahead-of-Time P-Tuning
Authors: Daniil Gavrilov
Venue: Submitted to ICLR 2023 (2023)
Keywords: Efficient Fine-Tuning
P-Tuning
Multi-Task Inference
Transformers
GLUE
SuperGLUE

Abstract: This paper proposes a new parameter-efficient method for fine-tuning, AoT P-Tuning. This method adds input-dependent biases before evaluating the Transformer layer, reducing the required evaluation time when compared to P-Tuning. Same as P-Tuning, AoT P-Tuning allows multi-task inference with a single backbone model for evaluating different tasks in a single batch.
We experimented with the proposed method on the GLUE and SuperGLUE benchmarking datasets using RoBERTa-Base, RoBERTa-Large, and DeBERTa-XL backbone models. Our observations show that AoT P-tuning performed on par with or better than P-Tuning v2 while being up to $1.3\times$ times faster during inference.

Under review as a conference paper at ICLR 2023
AHEAD-OF-TIME - P-TUNING
Anonymous authors Paper under double-blind review
ABSTRACT
This paper proposes a new parameter-efficient method for fine-tuning, AoT P- Tuning. This method adds input-dependent biases before evaluating the Trans- former layer, reducing the required evaluation time while allowing multi-task inference with single backbone model for evaluating different tasks single batch. We experimented with the proposed method on the GLUE and SuperGLUE benchmarking datasets using RoBERTa- Base, RoBERTa-Large and DeBERTa-X0 XL backbone models. Our findings show that AOT P-tuning performed par with better than P-Tuning and comparable to other baselines for efficient fine-tuning while being faster during inference.
60
LoBA
SuperGLUE Macro Score
Lott
Adapters
Adapters
*Tuiiing V2
Plluning 92
AOT P.luning louns)
P.Tuning (ouns)
GLUE Macro
07
57
RoBEITT-BRI Base
RcBERTu-Large
Figure 1: GLUE and SuperGLUE Macro scores for different backbone model scales. See Section 4.2 for more details.
INTRODUCTION
P-Tuning (Liu et al. 202lb a Lester et al. 2021) a promising way to fine-tune large Language Models (LMs) (Devlin et al. 2019 Lan et al. 2020 Liu et al. 2209 Radford et al. (019). While it currently underperforms compared to other methods for parameter-efficient efficient fine-tuning (Hu et al.) 2022 Houlsby et al. 2019)
wide range of tasks (Ding et al. 2022) has practical, valuable property that allows to evaluate different trained prompts parallet in a multi-task manner (i.e., a single backbone LM could be used for different tasks during inference, which simplify model serving in real-world applications) (Lester et al. 2021). This property is why researchers aim to further develop P-Tuning methods.
Although is possible to perform multi-task evaluation with P-Tuning, it introduces significant computational overhead due to the concatenation of prefixes sequences and the evaluation of the attention mechanism (Vaswani et al. 2017) on longer sequences.
We propose simple mechanism for parameter-efficient fine-tuning of Language Models, namely Ahead-of-Time (AoT) P-Tuning, for which we add input-dependent bias before each Transformer layer. Same as P- P-Tuning. it is possible use AoT P-Tuning in multi-task inference setups when single backbone LM is used for several downstream tasks.
The contributions of this paper can be summarized follows:
We described the intuition behind AoT P-Tuning, which illustrates the connection of the proposed method with P-Tuning.

---

Under review as a conference paper at ICLR 2023
Px
K
Pv
V
P.WW
Figure 2: Schematic comparison of P-Tuning v2 (left), and AoT P-Tuning (right). Since the sequence length is not increased, AoT P-Tuning takes significantly less time to evaluate, only requiring the overhead of adding biases to the input sequence (See Section for experiments with inference speed).
2. We proposed two reparameterizations of AoT P- P-Tuning weights; first based on a factorized matrix trained from scratch, and second based on LM's embeddings matrix passed through a trainable Fully Connected network.
3. We experimented with the proposed method on GLUE, and SuperGLUE Benchmarking Datasets (Wang et al. 2018 2019) with the RoBERTa (Liu al. 2019) and DeBERTa (He (et al.] (020) models and observed that AoT P-Tuning performed on par with or better than P-Tuning v2, comparable other baselines for efficient fine-tuning while being faster than them.
RECENT WORKS
Currently, a wide range of different methods could referenced with P-Tuning, Liu et al. (202 lb) proposed to add soft prompts to the embeddings of GPT-2's input sequence (Radford et al. 2019) to train it on classification tasks. [ester al. (202T) proposed scheme similar to the one used in Liu al. 2021b), but trained T5 model (Raffel et al. (020) with P-Tuning to show how the performance of the method changes with the increased scale of the backbone model.
Recently, Qin & Eisner (202T;; Li & Liang (201): Liu al. (021a) proposed add prefixes not only to input embeddings but also at each layer of the Transformer model. In addition, Liu et al. (2021a) suggested training linear classification head top of the backbone model instead of utilizing a LM head to obtain classification results.
Due to this range of similar methods, will follow the naming used by Liu al. (2021a) and refer to Prompt-Tiune Tuning (adding soft prompts to the input emheddings) as P- Tunnnng v1 and to Prefix-Tuning (adding soft prefixes at each layer of Transformer backbone) P-Tuning v2.
Hu et al. (0222) proposed to train low-rank changes of attention weights, while Houlsby et al. (2019) fine-tuned additional model layers, which can also be considered parameter-efficient Ben Zaken et al. (002) proposed to fine-tune only bias terms of the model.
3 AHEAD-OF-TIME P-TUNING
For readers' convenience, we provided background to Transformer evaluation and P-Tuning v1/v2 methods, which are relatable to the proposed method in Appendix Section A
3.1 ON THE OVERHEAD OF RECENT METHODS
While the Transformer model has O(n?) time complexity and GPU memory consumption for sequence length n. For P-Tuning v1, this complexity transforms into O((n + p) since the length of input sequence is increased by the length of the prompt p. while for P-Tuning v2 the complexity is equal 0(n(n p)).
2

---

Under review as a conference paper at ICLR 2023
Method
Parameter Efficient
Zero-Cost
Multi-Tann Inference
Fine-Tuning
X
X
LoRA
X
LoRA Fused
X
Adapters
X
BitFit
X
P-Tuning v1/v2
AoT P-Tuning (ours)
Table 1: Schematic comparison of recent fine-tuning methods with AoT P-Tuning. Recent fine-tuning approaches either allow inference with no computational overhead or multi-task inference. See Section 3.1 for details.
Liu et al. (2221a) showed that for some tasks, the prompt length p could reach values of 100, increasing time and memory footprint during evaluation.
For the Adapters approach (Houlsby et al. 2019), the overhead appears since additional layers are injected into the computation of model. However, it is possible to perform multi-task inference with it by passing the weights these layers with the batch.
We could consider two usage scenarios for LoRA (Hu et al. 2022). For the first one, we fuse original weights with a trained low-rank addition. In such a case, we will see no inference speed overhead since the obtained model will have the same structure as the original pre-traine pre-trained model. While is theoretically possible to perform multi-task inference with a fused model similar to Adapters, doing will require allocating massive amount of GPU memory. E.g., for the DeBERTa-XI model with layers of hidden size d = 1024 and = 48, passing a batch with b sequences will require one to store 1024 1024# 48 4 6 parameters be given to the model, where is number of parameter matrices of Attention module. It is easy to note that with b = 4, we will already exceed the number of original parameters in the model, which is impractical. To overcome this, is possible fuse LoRA weights to perform multi-task inference. However, doing so can introduce computational overhead.
See Table D for the schematic comparison of recent methods terms of inference speed overhead and ability to perform multi-task inference.
3.2 PROPOSED MECHANISM
With AoT P-Tuiing, we propose to augment each Transformer layer with a simple procedure. We define trainable matrices p E R for each layer. Then, before the evaluation of the i-th layer, modify the hidden states follows:
H" ="
p Zn
E Rad
where P E Rd is lookup of zy-th prompt embedding from Pi. Such scheme allows us to save significant amount time during evaluation since AoT P-Tuning does not imply an increase in sequence length. While pi in naive implementation will require lot of memory to store parameters, we describe reparametrizations which make training more tractable.
in the following Section 3.3
Note that AoT P- P-Tuning, same as plain P- P-Tuning, could be evaluated in parallel with several tasks batch due to the fact that performing look-up from P can easily parallelized. See Appendix Section B for explained intuition behind the proposed method.
As for P-Tuning v1 and P-Tuning v2, we only optimize parameters of P and Classification Head during fine-tuning.
3.3 ON THE PARAMETER EFFICIENCY OF AOT P-TUNING
is notable that, in most cases, it is not feasible to optimize the weight P E [V for each layer.
If we consider training RoBERTa-Laage with such scheme (which has V 50265, 1024 and
3

---

Under review as a conference paper at ICLR 2023
RoBERTa-Bass
Model
STS-B
SST-2
RTE
QQP
Fine-Tuning
90.6 + 0.3
95.0 = 0.2
81.2 = 0.7
89.6 + 0.2
Adapters LoRA
90.7 0.2
94.4 0.3
80.5 + 2.0
89.2 0.1
90.1
94.3 0.5
80.5 - 1.8
86.3
BitFit
90.3
94.5
80.9 - 1.4
P-Tuning vl
86.9 0.9
9.3
60.3 + 2.4
82.2 1.5
P-Tuning v2
8.3
94.6 0.2
80.5 + 3.4
86.4 + 3.3
Kron. AoT P-Tuning (ours)
89.7
94.0 0.2
77.6 + 1.4
88.2 # 0.1
FC AoT P-Tuning (ours)
90.0
94.4 = 0.3
78.0 1.3
87.9 + 0.2
QNLI
MRPC
MNLI
CoLA
Macro
Fine-Tuning
92.4 + 0.1
90.8 # 0.5
87.0 + 0.3
63.8 + 1,4
86.3
Adapters
0.2
91
86.8 + 0.1
63.0 # 1.3
86.0
LoRA
91.6 0.3
9.7
84.7 + 0.3
60.3 + 1.0
84.8
BirFit
90.9 = 0.5
85.0 0.1
60.4 + 1.2
84.7
P-Tuning vl
88.3 0.5
82.0 1.7 89.1
80.8 0.6
4
77.5
P-Tuning v2
91.9 + 1.6
85.3 + 0.2
60.7 2.6
84.7
Kron. AoT P-Tuning (ours) FC AoT P-Tuning (ours)
90.7 = 0.4
89.5 + 1.1
84.6 + 0.1
59.3 # 1.2
84.2
91.3 = 0.4
90.3 = 0.3
85.4 0.1
60.3 + 2.2
84.7
RoBERTa-Large
Model
STS-B
SST-2
RTE
QQP
Fine-Tuning
0.2
96.1 # 0..
88.1 + 1.5
90.3 + 0.2
Adapters LoRA
92.1 0.2
96.3 + 0.4
90.0 = 0.1
94.2 = 0.1
91.4
95.9 0.2
87.2 18.7
# 93.7
BitFit
91.8 + 0.2
96.2 0.4
87.7 = 0.8
87.2 + 0.6
P-Tuning vl
75.5 = 6.3
94.4 0.4 9.3
62.8 = 2.3
76.9 + 2.5
P-Tuning v2
91.0 0.4
87.4 1.5
86.6 + 0.6
Kron. AoT P-Tuning (ours) FC AoT P-Tuning (ours)
91.1 # 0.8
96.2 # 0.2
84.8 + 1.3
89.4 # 0.1
91.7 = 0.4
96.7 0.1
88.4 0.9
88.7 # 0.2
QNLI
MRPC
MNLI
CoLA
Macro
Fine-Tuning
94.3 + 0.2
91.6 + 0.6
89.9 + 0.2
68.1 + 1.9
88.8
Adapters
91.3 + 0.4
90.1 # 0.2
67.2 1.3
87.7 + 18.5
88.6
LoRA
01.077.
88.9 + 24.0
66.3
87.4 + 1.6
87.7
BitFit
94.1 0.4
1.0
8.14
69.8 + 3.1
88.4
P- P-Tnning v1 T-Tuning v2
79.1 2.4
1.1
75.9 18.3 0.7
24.7 + 17.6
71.0
94.0 1.1
91.2
66.9 + 1.5
87.8
Kron. AoT P-Tuning (ours)
94.2 + 0.1
89.7 + 0.9
89.3 + 0.1
65.5 + 1.9
87.5
FC AoT P-Tuning (ours)
94.1 + 0.2
91.6 0.8
89.6 0.1
69.2 0.9
88.8
Table 2: Results the GLUE Dev set. Each result is median and std across several seeds, and the Macro column is mean score across all tasks. We bolded the best results and underlined the second best results. Fine-tuning is omitted from comparison with other methods and was not bolded for visibility. See Section 4.2 for details.
1 - 24), then storing all biases will exceed 1.2B parameters, while the model itself has roughly 350M parameters.
To overcome this limitation, we propose two reparametrizations of P so that can use fewer parameters during training.
4

---

Under review as a conference paper at ICLR 2023
The first is based the Kronecker product (namely, Kronecker AoT P-Tuning). More specifically, we reparametrize P as
P = (WE @ WM)
(2)
where Wz E Raxr WM Rxr WR E Rxd a and b are selected in such a way so a bb= V is the factorization rank which is hyperparameter to tune, and @ denotes the Kronecker product.
With this reparametrization, training AoT P-Tuning becomes tractable. E.g., for RoBERTa-Large, with 256, b 200, and 20, P will contain roughly 10M parameters, which is less than 3% of the total number of parameters the mode !
The second approach work with P. which we used in experiments, is based on passing the embeddings matrix E through a learnable Fully Connected network (namely, FC AoT P-Tuning). Thus, reparametrize P as
P= f(EWi + W22 + b2.
where Wi E Rdxr bj E W2 E Rexd b2 E Rdd f is non-linearity, and r is the mapping rank, which is also hyperparameter to tune, same as for Kronecker AOT P-Tuning.
With FC AoT P-Tuning, we utilize knowledge stored the pre- pre-trained embeddings matrix E, which should hypothetically perform better than training P from scratch Kronecker AoT P-Tuning.
Note that for both Kronecker and FC AoT P-Tuning, we can evaluate only specific rows (Pr,,.... Pa.. for input sequence (nn) making training more efficient.
For both reparametrizations, P could be fused once training complete, and thus the rank of factorization does not affect inference speed. During the evaluation, there is need to store the full P GPU memory. Instead, it could be stored RAM, and only rows of these matrices should be placed in GPU memory to be added to the hidden states before each layer.
From a certain perspective, choosing between AOT P- Tuming and P- P-Tuming is a trade-off between evaluation speed and RAM consumption during inference. If RAM is limited, then usual Tunnng could be used at the cost of slower inference. In other cases, AoT P-Tuning is viable if there is enough RAM and inference speed is crucial. Although, in most cases, P matrices for different tasks could be easily stored in the RAM. For RoBERTa-Large, a single task parameter will require roughly 2.4Gb if stored in half-precision.
EXPERIMENTS
4.1 EXPERIMENTAL DETAILS
We compared AoT P-Tuning (Kronecker and FC reparametrizations of P) with other fine- tinettunigg methods capable of performing multi-task inference: P-Tuning vl, P-Tuning v2 on GLUE and SuperGLUUE WWang et al. 2018 2019) Benchmarking Datasets. We also evaluated plain fine-tuning, LoRA, Adapters, and BitFit for reference. For each fine-tuning approach, we experimented with the RoBERTa-Base, RoBeRTa-Large, and DeBERTa- XL backbone models.
For each task, we performed a grid hyperparameter search (see Appendix Table 5 for hyperparameter ranges). For RoBERTa models, we evaluated each hyperparameter set with 5 different seed values and reported median and std score values for each task. For DeBERTa-XX. we used assess each hyperparameter assignment with single seed due to longer training time. See Appendix Table for a list of metrics used for each task.
OOne may note that 256 * 200 51200 * 50265. However, 50265 is difficult to factorize efficiently since 50265 = 1117 3 5. Because of this, we chose to mostly factorize P in such a way as to make it slightly larger than the original vocabulary size. Doing allows us to select more appropriate and b from the perspective of parameter and computational efficiency.
5

---

Under review as a conference paper at ICLR 2023
RoBERTa-Larg
Model
RTE
COPA
WSC
WiC
Fine-Tuning
88.1 + 1.5
87.0 = 10.2
80.8 + 6.3
73.8 + 1.6
Adapters LoRA
87.7 + 18.5
89.0 10.1
7.89
73.5 = 1.0
87.4 1.6
9
79.8 10.6
71.9 1.2
BitFit
87.7 0.8
91.
7.. 6.7
T-Tuning vl
62.8 2.3
7..0
66.3 1.3
6..1
P-Tuming v2
87.4 + 1.5
8.30
75.0
70.8 + 1.5
Kron. AoT P-Tuning (ours)
84.8 # 1.3
72.0 2
67.3 + 3.0
71.0 + 1.0
FC AoT P-Tuning (ours)
88.4 0.9
85.0 = 10.1
79.8 + 4.1
72.1 + 1.5
MultiRC
CB
BoolQ
Macro
Fine-Tuning
83.3 # 1.1
97.3 2.8
85.6 + 0.3
85.1
Adapters LoRA
83.7 # 20.3
100.0 0.0
85.7 + 10.6
85.4
75.7 17.4
100.0 2.6
84.6 + 0.6
84.3
BitFit
82.5 0.6
100.0 0.7
85.4 1.0
84.2
T-Tnning vl
54.3
81.4 = 3.0
64.3 + 1.2
66.9
P-Tuning v2
82.4 + 0.6
100.0 + 0.8
85.0 + 0.6
83.9
Kron. AoT P-Tuning (ours) FC AoT P-Tuning (ours)
82.8 + 0.8
97.3 2.3
84.8 + 0.5
80.0
82.7 + 19.3
100.0 0.0
85.5 10.3
84.8
DeBERTa-XL
Model
RTE
COPA
WSC
WiC
Fine-Tunnig
89.9
96.0
76.9
75.9
Adapters
90.3
96.0
89.4
77.3
LoRA
90.3
97.0
89.4
75.5
BitFit
89.2
97.0
86.5
73.7
P-Tuning vl
78.3
90.0
67.3
66.8
P-Tuning v2
90.6
97.0
89.4
76.5
Kron. AoT P-Tuning (ours)
88.8
96.0
87.5
71.8
FC AoT P-Tuning (ours)
91.0
98.0
94.2
74.1
MultiRC
CB
BoolQ
Macro
Tune-Tuning
84.3
98.4
86.7
86.9
Adapters
86.7
97.3
88.9
89.4
LoRA BitFit
86.0
100.0
88.3
89.5
85.2
100.0
86.5
88.3
P- Tunnin v1
82.1
93.8
79.4
79.7
P-Tuiing v2
87.1
97.3
87.0
89.3
Kron. AoT P-Tuning (ours)
86.3
83.1
87.3
85.8
FC AoT P-Tuning (ours)
86.5
92.3
88.1
89.2
Table 3: Results the SuperGLUE Dev set. For RoBERTa-Large each result is median and std across several seeds, and the Macro column is a mean score across all tasks. For DeBERTa-XI.. we evaluated each hyperparameter assignment with single seed and reported its metric score. We bolded the best results and underlined the second best results. Fine-tuning is omitted from comparison with other methods and was bolded for visibility. See Section 4.2 for details.
We used the Adam (Kingma & Ba 2015) optimizer with constant learning rate for each task. We stopped training once the validation metric stopped increasing (see the "patience" parameter in Appendix T).

---

Under review as a conference paper at ICLR 2023
RoBERTh Base Hacro Score
RoBERth Large Mocro Score
0.1
0.7
Marre Store
6.7
Marre
0.5
Accptirs
Adapters
6.5
FC AoT pluning
Pllanin
Cren, Pftuiing
Tunnng
0.4
0.4
Parameters
Parameters
(b)
Moco Spore
0.75
Macro Seore
MiX Accetors
Biht
Adastore
0.65
intuning
PTaning
Prieluring
10'
10
Trainale Parameters
Trainable Parameters
(c)
(d)
Figure (a-b) GLUE macro scores for AoT P-Tuning, P-Tuning v1, and P-Tuning v2 with RoBERTa- Base and RoBERTa-Large models. (c-d) SuperGLUE macro score for RoBERTa-Baa and DeBERTa- XL models. P-Tuning v2 performing on par with or worse than AoT P-Tuning across different prefix
sizes. See Section 4.2 for details.
For Kronecker AoT P-Tuning with RoBERTa models, we parametrized the matrix P (W, @ WR R with a = 256, and = 200, while for DeBERTa, we used a = b = 360. W,, and War were initialized randomly, while WR was initialized as zero matrix. For FC AoT P-Tuning, we initialized W1 randomly, while W2. bj. and by were initialized with zeros. For Kronecker AoT P-Tuming, we applied dropout (Srivastava et al., (2144) to the P. with fixed probability equal to 0.1. In contrast, for FC AoT P-Tuning, we applied dropout to E before multiplying it with W,
Each experiment was run on single NVIDIA A100 GPU with total computation time of roughly 1200 days.
4.2 RESULTS
See Tables 3 for the results of trained models. We observed that FC AoT P-Tuning performed better than Kronecker AoT P-Tuning, and hypothesize that this result is mostly caused by the fact that FC reparametrization utilized a pre-trained embedding matrix rather than learning biases from scratch.
For RoBERTa-Baa Base, FC AoT P-Tuning performed on par with P-Tuning v2 and produced the same Maero score. For RoBERTa- Large, FC AoT P-Tuning outperformed P-Tuning v2 on GLUE tasks and showed a Macro score equal to plain Fine- Fine-Tuning. AoT P-Tuning with DeBERTa-X0 performed on par with P-Tuning v2 (89.2 vs 89.3 macro scores respectively).
We also observed that both AoT P-Tuming reparametrizations mainly showed a lower variance of metrics across different seeds. Note that P-Tuning v1 showed unstable performance and improved
7

---

Under review as a conference paper at ICLR 2023
10
G
Lage, kuel, CHARS
Talkeita LengL, WANT
Packeera Larg,, WARRE
Avates
Thlmuble
(d)
(I)
be3D 24 1--11-30
C40CSOUL bE 26 201
DASCST-OL DE 64 7-0
(h)
Figure 4: Speed measurements for haseline methods with sequence length equal to 384 for different back- models. See Appendix Figure 9 for results with other sequence lengths and Section 4.3 for details.
results with RoBERTa-Base (although still underperforming by a large margin when compared to other methods).
See Figure for macro scores of P- P-Tuning v2 and AoT P-Tuning with different prefix lengths p and prefix ranks We observed that P-Tuning v2 performed worse for RoBERTa-Bass with shorter prompt lengths and was comparable to or better than AoT P-Tuning when p > 50. For GLUE tasks with RoBERTa-Large, FC AoT P-Tuning performed better for all prefixes p. while dropping performance for large rank r. For DeBERTa-XX XL, both P-Tuning v2 and FC AoT P-Tuning performed on par. We also provide per-task results with differrnnt prefix scales (sec Appendix Figures 5 7 It is notable that in most cases, P-Tuning v2 suffers from a small prefix size p for Base and Large models, and achieves results comparable with AoT P-Tuning with larger p (which corresponds with the results in Figure 33. At the same time, FC AoT P- Tuning mostly showed stable performance across different ranks r, only performing unstably on MultiRC task with large rank r.
Compared with LoRA and Adapters methods, RoBERTa-I arrge Adapters showed a higher Macro score with SuperGL.UE, than all P-Tuning methods, including AoT P-Tuning, even while showing larger variance across task scores. For DeBERTa-XL, LoRA and Adapters marginally outperformed P-Tuning methods on the macro score. BitFit performed worse than FC AoT P-Tuning, except for RoBERTa-Baa back-bone where it performed the same. From such a perspective, if the ability to
Note that the best macro result across different scales of prefixes in these Figures differs from the macro result from Tables 2 and 3 since the macro score from Tables 2 and 3 aggregates scores with different prefix scales.
8

---

Under review as a conference paper at ICLR 2023
perform multi-task inference without computational overhead is crucial, then AoT P-Tuning will not dramatically reduce performance compared to LoRA and Adapters and could be used. Note that the gap in performance between AoT P- P-Tuning and Adapters and LoRA became marginal with model growth.
With per-task Expected Validation Performance (EVP) (Dodge et al.) 2019), observed that AoT P-Tuning highly depends on the number of hyperparameter assignments (see Appendix Figures 6
8p. Although, in most cases, using less than 100 hyperparameter assignments for AoT P-Tuning enough for it to outperform P-Tuning v2, which is not crucial in most cases.
We also analyzed trained P matrices for FC AoT P-Tuning with the DeBERTa-XI. model. See Appendix Section C for more details.
4.3 INFERENCE SPEED OVERHEAD
In Figure El and Appendix Figure we investigated the computational overhead of AoT P-Tuning compared to other baselines.
We estimated inference time for RoBERTa-Base, RoBERTa-Large, and DeBERTa-XI. models with batch sizes [1, 16, 64] and sequence lengths E [64, 128, 384). For batch size equal to 1, evaluated the model 300 times, and 100 times for other values. We report mean values of inference time normalized by the inference time of the vanilla model (i.e., plain fine-tuning).
We evaluated AoT P-Tuning on two setups. For the first setup, we fused P so that the model can perform on its top speed. We did not perform fusing for the second setup. While a lack of fusing not required to use AoT P-Tuming, we report these results for reference. We also report LoRA results for the experiments in which we did not fuse weights. This setup makes possible to use LoRA in multi-task setup (see Section 3.1 for details).
We observed that the proposed method performed differently depending on the experiment's parame- ters. AoT performed with computational overhead for small model (i.e., RoBERTa-Baas Base)) small sequence length, and small batch size. We observed 12% overhead on inference speed compared to plain fine-tuning for RoBERTa-Base with batch size and sequence length 64. However, AoT P-Tuning still performed faster than other baselines by a large margin (i.c., LoRA for RoBERTa-Baae with batch size 1 and sequence length 384 added 50 70% computational overhead compared to fine-tuning).
Once model size input size is increased, observed that the overhead of adding biases for AoT P-Tuning becomes negligible. The proposed method performed the same as plain fine-tuning (for some experiments,
2% overhead, while for others, AoT P-Tuning performed even faster than fine-tuning, which is a variation in inference time measurements). For the most practical setup with large models (DeBERTa-XIL) small batch size (= 1) and long sequence length (= 384), AoT P- T-Tuning performed slightly faster than fine-tuning, while other methods performed 12 25% slower.
observed
5 CONCLUSION AND FUTURE WORK
In this paper, we proposed AoT P-Tuning, which is a new method for parameter-efficient fine-tuning of pre-trained models, and two reparametrizations of learnable weights for this method.
We observed that AoT P-Tuning performed on par or better than P-Tuning v2 based on the macro scores of GLUE and SuperGLUUF Benchmarking Datasets. In addition, compared to the LoRA and Adapters methods with large backbone model (i.c., DeBERTa-XL), we observed speed increase at the cost of only a marginal performance drop.
We experimented with two reparametrizationn based on the Kronecker product and FC network. It is possible to explore other possible reparametrizations for weight P, which could further increase the performance of the proposed method. In addition, while proposed simple method, there many possible architectural changes which could also boost the performance of AoT P-Tuning and reduce the number of necessary hyperparameter assignments.

---

Under review as a conference paper at ICLR 2023
6 REPRODUCIBILITY
We organized our experiments in such a way that makes it possible to quickly reproduce all the results reported in our paper.
tools/ run https//essswwsssssseeeeeee py runs wandb sweeps used in our hyperparameter search and outputs wandb agent commands run these sweeps. Once the hyperparameter search is finished, tools/ s glve_ resull 1 tss. Py and too 1 s/ supergl ue_ resu ts py scripts are used to parse results from these sweeps and output plots reported by us.
The data processing pipeline for all datasets used in this paper can be found in the zarya data__ processingg py script. It could be helpful for tasks such as WSC, where several ways to tokenize input sequences could affect the resulting performance.
We also provided requi i rements. txt (with fixed dependencies versions) and a Dockerfile with source files to make it possible to quickly reproduce our training setup without issues with versions of libraries used for our experiments.
REFERENCES
Elad Ben Zaken, Yoav Goldberg, and Shauli Raviogel. BitFit: Simple parameter-efficient fine-tuning for transformer-based masked language-models, In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 1-9, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/w1/2022.acc short. URL /ttp://llaatttloooyooo/////22222 acl.s222..llshrr...
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volune (Long and Short Papers), pp. 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 10.18653/11/119-44222 URL https //aclanthologg www.yyorggNn111111
Ning Ding, Yujia Qin, Guang Yang. Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, Jing Yi, Weilin Zhao, Xiaozii Wang. Zhiyuan Liu, Hai-Tao Zheng, Jianfei Chen, Yang Liu, Jie Tang, Juanzi Li, and Maosong Sun. Delta tuning: A comprehensive study of parameter efficient methods for pre-trained language models. 03 2022. doi: 30.21203//s..rs-15555 15535551/11
Jesse Dodge, Suchin Gururangan, Dallas Card, Roy Schwartz, and Noah A. Smith. Show your work: Improved reporting of experimental results. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNI.P-IJCNLL. pp. 2185-2194, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10. 10.18653/v1/DD1912222 URI. https //aclantho ogg.
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Decoding-enhanced bert with disentangled attention. CoRR, abs/2006.03e0 003544, 2020. HRRlhttp://ddt uni -trier de/ htt/////oounaaasccrrr///// cat.se//oorr0000 html abss 2006 03654
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for NLP. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 2790-2799, PMLR, 09-15 Jun 2019. URL htttt::// htttps://proceedinggs mlr
stress/97/hhull.
html
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In ICLR 2022,
http:::/wwwwmmiccooooooo April 2022. off www.////ne/eeeeeeeeeeeeeeee
10

---

Under review as a conference paper at ICLR 2023
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.). 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings. 2015. URL http: 1/arxiv. org. abs / 412.44..6 6980
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: A lite bert for self-supervised learning of language representations. In International Conference on Learning Representations, 2020. URL
wtwp:/..ee/oouuueeeeeeeo umrum?
id=HleA7AEts
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efftcient prompt tuning. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 3045- 3045-3059, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. 10.11633/v1/2021 www.022eeemnlpmmeeeee monllp-main225 243. URLhttps
7/aclanthe ho ogy. org 2021 emn 1apllppmmms ma 243
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Anmual Meeting of the Association for Computational Linguistics and the I Itth International Joint Conference Natural Language Processing (Volume 1: Long Papers), pp. 1582-45977 Online, August 2021. Association for Computational Linguistics. doi: 10.18653/ v1/2021 aal.2...c--ooo.35500 333. URL http::://aaaattttlllaaa// wrt.loogoorr//g2e/ 201.1/201.....coogggo
Xiao Liu, Kaixuan Ji, Yicheng Fu, Zhengxiao Du, Zhilin Yang, and Jie Tang. P-tuning v2: Prompt tuning can be comparable to tine-tuming universally across scales and tasks. CoRR, abs/2110.07602. 2021a. URLhttps: httt:::///arxiv... arxia.aiiv...... org/ wbtpaas........1
Xiao Liu, Yanan Zheng. Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. Gpt understands, too. arXiv:2103.10385, 2021b.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach, 2019. URL htt.tp:/aaxxv..og////o
1929 cite
Guanghui Qin and Jason Eisner. Learning how ask: Querying LMs with mixtures of soft prompts. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 5203-5212, Online, June 2021. Association for Computational Linguistics. doi: 10.18653/v1/202. 1 www.cl-main.41111 URL https
//aclantho log.l org/2021 nww.laaa...... main..111
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. Language models are unsupervised multitask learners. 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67, 2020. URL
wttp:ppprr.ppppp..p.e/pp. v2 /t/////112222222222
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(56) 1958, 2014. URL http://tp//// httl:/mmroorrr wwmlr.org/pan www..rrrpppppp/.......
srivastaval4a.hot
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, pp. 5998-6008, 2017.
Alex Wang, Amapreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bow- man. Glue: A multi-task benchmark and analysis platform for natural language understand- ing, 2018. UtLphtt:://aaet org/ abs/ 1804. 07461 cite wtw.v:.100..7461110 https://ghuehennmmarkccomm
11

---

Under review as a conference paper at ICLR 2023
Alex Wang, Yada Pruksachatkum, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: stickier benchmark for emerralpurposs. language understanding systems. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran As- sociates, Inc., 2019. URL ht / http:///ooceeeddeeeeeeeoeo ngs neur pps wttpe/eerrr.ccccpppprr.e.oooe wttpccppppppe.111111 9/file/ Paper.
A BACKGROUND
A.1 EVALUATION OF TRANSFORMER
Having an input sequence a {s1..... .,, In where z; token index, the embeddings of input texts are evaluated as H (Es....., Ex,) } where E R is the embeddings matrix, |V] the vocabulary size, d is the size of the hidden state of the model, and E,, is an embedding of the token ry- Hidden states H' are then passed to the (i + 1)-th layer of the Transformer to evaluate Hi+1 with a total I number of layers. To do so, H' are first mapped through three matrices Wo. W Wv E Raxd to get Q, K and V, which are then used to evaluate the attention layer's results as:
QK T V E Raxd
A = attention Q,K, V) softmax
Vd
After A is evaluated, is passed through the remaining layer including residual connections and FC layers to get H'+1, Here and later, we omit the layer index for attention result A for visibility.
A.2 P-TUNING V1
Having a pre-trained Transformer LM with parameters O instead of fine-tuning all parameters of this model downstream task, it is possible to define soft prompts P Rpxd (Lx et al. 2021b), where is the length of prompt. P is then concatenated to input sequence embeddings as:
H 10 concat( H E (e+n))dd
(5)
Then, only P and Classification Head are fine-tuned on downstream task, while θ remains frozer" Such parametrization of fine-tuning makes it possible to perform multi-task inference.
A.3 P-TUNING V2
Instead of concatenation of single prompt P the H Liu et al. (2021a) proposed to concatenate soft prefixes at each layer of the Transformer model. To apply P-Tuning v2, soft prefixes Pk, Pv Rexd are defined for each layer and concatenated to the K and V matrices before evaluating the attention K" = concat( PK, K)), V concat( Pv. V). Then, Attention is evaluated as follows:
A' attention Q, V'),
(6)
where i-th component of A' could be then written as:
A/= Σ a Q. + Σ Extp(Qu K'
1=1
'In fact, Transformer architecture implies evaluation of multi- head Attention. We omit this in this paper for simplicity since all derivations could be casily extended on the multi-head case.
'Original implementation of P- P-Tuning vl (Liu et al. 2022b) implied utilizing the LM Head of pre-trained model instead of training a Classification Head. However, Liu et al. (201a) later showed that using separate Classification Head performs marginally better.
12

---

Under review as a conference paper at ICLR 2023
Note that R attention weights for the i-th token (we omit the i-th index for simplicity) and thus 2j=1 a =1
As for P-Tuning vl, only parameters of soft prefixes Pk, Pv and Classification Head are optimized on a downstream task while freezing the parameters backbone model.
B INTUITION BEHIND AOT P-TUNING AND CONNECTION TO THE P-TUNING
Having H', after passing through Wo. WK. and Wv we obtain Q', K', and V'. Note that
V HWv + (P... ........ Ps, WV del V + P.PW
The result of evaluating Attention with AoT P-Tuning could be seen as:
A = Σ a
Wr E a,
(8)
j=1
From such a perspective, there is clear connection between AoT P-Tuning (Equation 8 and P-Tuning with the following changes:
v2 (Appendix Equation 7
1. For AoT P-Tuning, attention weights aj. j E 1,1 are used for both terms Equation S
2. For AoT P-Tuning, attention is evaluated on modified Q'. In addition, there is difference in the form of dependency of K' and V on prefix weight. For AoT P-Tuning. we add prefixes to K and V, while for P-Tuning v2, prefixes
concatenated to these matrices.
3. For AoT P-Tuning, the first term of Equation 8 implies evaluation of Attention with prompt which dependent on the input text, while for P-Tuning v2, the prompt Py constant.
Considering Equation g AoT can be seen as a form of the P- P-Tuning method, for which we embed prefixes before evaluating the attention layer")
C ANALYSIS OF TRAINED WEIGHTS
We investigated trained P matrices for WSC, COPA, CB, and RTE tasks with the DeBERTa-XX model. Since FC AoT P-Tuning performed better than Kronecker factorization, we selected this reparametrization method report the results.
More specifically, we sorted rows of P matrices for each layer measured by the L2 norm and reported the appropriate tokens for these rows. See Tables 7 10 for results.
For the WSC task, there is a clear interpretation of trained rows for P, since rows with a large La norm represent tokens responsible for pronouns and names, which is crucial for solving WSC. For the COPA task, we observed that the model tends to assign large norms for verb tokens. For the RTE and CB tasks, P also assigns large norms for name tokens, which often occur the training data, while CB primarily modifies adverbs for later layers.
'It is possible to think of AoT P-Tuning as method which adds bias after the evaluation of the Transformer layer. In this case, it could be seen method that directly models the result of the evaluation of P-Tuning v2 with slightly different computation order. However, believe that this way more difficult consider.
13

---

Under review as a conference paper at ICLR 2023
Task
Metric
Task
Metric
CoLA
Mattews Correlation
BoolQ
Accuracy
MRPC
CB
Accuracy
2
RTE
Accuracy
RTE
Accuracy
SST-2
Accuracy
COPA
Accuracy
MNLI
Accuracy
MultiRC
Accurncy
QNLI
Accuracy
WSC
Accuracy
QQP
F
WIC Accuracy
2
STSB
Pearson Spearman
Table Metries used in experiments for each task. See Section 4.1 for more details.
Parameter
Range
All Tasks, except RTE
P-Tuning v1/v2/AoT
Parameter
Range
batch size
16, 64
P-Tuning v1/v2/AoT
learning rate
1e-4, 5e-4, 4, 3 le-3
batch size
16, 32, 64
5. 10, 20, 50, 100
5e-5, 1e-4. 4, 3e- 5e-4,
learning rate
1e-3, 2e 2--3, 5e-3
LoRA
2, 4, 16, 32, 64
5, 10, 20, 50, 100
Adapters
16, 32, 64, 128, 256
LoRA
2, 4, 16, 32, 64
Kron.
5, 10, 25, 30, 50
Adapters
16, 32, 64, 128, 256
FC
32, 64, 128, 256, 512
Kron.
5, 10, 25, 30, 50
Fine-Tuning
FCr
32, 64, 128, 256, 512
le-5, 5, 5e 5, le-4, 5e-4. 5e-3
Fine- Tine-Tunigg
learning rate
le- 5, 5e-5, 5, 1e-4. 5e 4,-4, 5e-3
RTE
learning rate
batch size
16, 32, 64, 128
lc-5, 5e-5, 1e 4, 50-
learning rate
5e-3, le-3, 2.-3.223111 le-2
Table 5: Hyperparameter ranges used in experiments with GLUE and SuperGI.UF. benchmarking datasets for RoBERTa (left) and DeBERTa (right) models. p the prompt length used for P-Tuning v1/v2, and r is the rank of weight factorization used for AoT P-Tuning (See Section For GLUE experiments, each hyperparameter more details.
evaluated with different seed values. See Section 4. for
14

---

Under review as a conference paper at ICLR 2023
CB,
MNLI, QQP
Other Tasks
COPA, WSC
Other
RTE
QNLI
WiC
MultiRC
Tasks
Epochs
200
5
10
100
500
500
10
100
Patience
20
2
2
10
20
100
4
10
Table 6: The number of maximum epochs used for each GLUE and SuperGLUE Task. Once the Dev score stopped increasing for "patience" steps, training was halted. See Section 4.1 for more details.
(a)
(b)
(c)
(d)
NutiMliee CAP
(h)
(k)
BuRIhoare D4N
(m)
(n)
(0)
(p)
Figure 5: Per-task GLUE Benchmarking Dataset results for a different number of trained parameters of P-Tuning v2 and AOT P-Tuning with RoBERTa-Base (a-h) and RoBERTa-Laare (i-p). We also provide results of plain fine-tuning for reference. See Section 4.2 for more details.
15

---

Under review as a conference paper at ICLR 2023
(d)
g
(h)
(k)
Fancs raay
Pamrs reginess
Fumm
(m)
(p)
Figure 6: Expected Validation Performance (Dogge et al. 2019 of trained models with GLUE Benchmarking Datasets for RoBERTa- Base (a-h) and RoBERTa Large (i-p). See Section 4.2 for more details.
16

---

Under review as a conference paper at ICLR 2023
Tabak
Tabuld
(b)
(c)
(d)
Tataak
Tanadk
(h)
2.0
(m)
(n)
Figure 7: Per-task SuperGLUE Benchmarking Dataset results for different number of trained parameters P-Tuning v2 and AoT P-Tuning with RoBERTa-Large (a-g) and RoBERTa-Large (h-n). We also provide results of plain fine-tuning for reference. See Section 4.2 for more details.
17

---

Under review as a conference paper at ICLR 2023
(b)
(d)
(h)
Mark
m
(n)
Figure 8: Expected Validation Performance Dodge et al. 2019) of trained models with SuperGLUD Benchmarking Datasets for RoBERTa- Base (a-g) and RoBERTa-Land (h-n). See Section 4.2 for more details.
18

---

Under review as a conference paper at ICLR 2023
Formerrr
(b)
AdjCtlie Larpe
Varrabe FolewA
(d)
(I)
INGLEDAL SHOT
(h)
(m)
Hining
Changg
Mayin
(p)
4
Figure 9: Speed measurement for baseline methods with sequence length E [16, 64) for different backbone models. See Section 4.3 for details.
19

---

Under review as a conference paper at ICLR 2023
1#
Tokens x with largest norm Pa
likes, a, is, loves, was, to, as, wants, pony, eded, himself, Man, were, and, I, has, are, Frank, hates, As, A, A, like, It, crop, Frank, After, joins, As, Eric, Likes, It, just, would, onna, him, To, behaving, after, because, behaves, Is, We, Like
narrower, doorway, backdoor, window, lousy, shortest, nicer, checkpoint, knob, thinner, narrowing, oub, quieter, BAD, i, VID, rectangle, tighter, crappy, intruder, tongues, fing, rimination, blocker, and, raiding, detector, unmarked, sharper, knife, coolest, thicker, hoops, DOWN, lightsaber, asshole, millisec, KEY, sharp, token, slashing, Defenders, jug, Donna, slider, wedge, dding, kb
her, Her, herself, him, above, she, out, hers, him, HER, She, she, care, HER, above, Her, bold, CARE, cared, over, harder, louder, Above,
10 smarter, sooner, her, cares, better, Out, vind, stronger, She, taller, tougher, Him, ahead, HIM, Susan, happier, up, Harry, aloud, higher, Above, SHE, could, apart, barking, inem
there, dry, for, There, Her, her, sword, the, arse, wy. dry, duc, The, it, took, cr, Rig. og, There, landing, the, wide, 22 centrally, red, grass, sw, oa, above, engine, FT, spir, cd, Coun, Ross, there, ws, guy, starter, mans, aniel, green, freely, d, wide, stall, far, artz, THERE, didn
it, me, olit, Polit, Pat, him, Private, Susan, pat, he, her, Self. Ins, Doc, Coun, Ang, Aut, Sil, ochond, me, Nob, IT, Senator, Professional, Dri, itized, Capt, Hillary, Whe, He, Kid, Registered, itious, Michelle, Political, It, Shut, Phot, BIT, Politics, Bit, Jacob, ruct, Young, HE, Tu, them, Mot, itu
32
him, they, it, they, her, them, their, his, he, it, its, hers, theirs, was, he, Susan, old, older, THEY, They, ITS, forth, Georg, Thom, Tom, erved, Carl, Anna, nob, anos, itans, to, Eric, itcher, Harry, Tim, Jen, them, Kid, Jeremy, JOHN, Jennifer, hands, Todd, put, Thomas, she, Dan, Michelle, s
37
chased, erved, house, houses, life, chasing, market, self, chase, raised, chester, hunt, castle, HOU, atics, Singer, western, ogenous, rounded, stretched, esian, essed, omorphic, horse, SER, central, ledge, hole, asio, Self, Self, iverse, oker, Judd, DF, aday, paced, ourced, erness, Barkley, scape, sey, ationally, owned, landers, ded, study, directed, OWN, produced
46
Table Tokens with the largest La norm of P entries for the WSC task. See Section C for more details.
20

---

Under review as a conference paper at ICLR 2023
1#
Tokens x with largest norm
Pa
fit, Loud, as, Air, Upon, Sets, Bound, Apart, scratched, sets, fit, Upon, hosted, Shot, Unt, Host, fitt, Sight, atri, Ocean, ceed, ashore, set, enture, underwent, planes, boats, Waves, Ali, shi, Active, Set, Atmosp, Airways, Host, chat, Endless, pelled, rew, ached, unct, fitted, Proud, flu, itable, anson, Bound, Assets, host, sets
set, Set, sets, Set, SET, Setting, setting, SET, set, padd. Setting, Sets, sets, bed, Cause, setting, the, cause, tread, itch, paddle, cause, thirsty, Khe, he, anned, this, ?, of, Cause, What, bidding. This, This, what, What, a, his, lic, The, wish, fugitive, they, Bed, Air, wake, conscience, crood, Let
5
?, ?!, ?2, ?", )2, ???, '? set,
22, ?), 1, ????? ...?, set, P' Set, Set, ed, to, ??, ???????? as, ?????, 2)., sctting, ?, ?", ?]. sets,
10
-, lt, -,,,, lie, ???, led, ur ...., punching, of, sets, um
What, set, What, out, Set, sets, on, to, what, Set, in, WHAT, Sets, Setting. WHAT, from, Setting, dropped, of, Dig, Got, ?'
22 set, Exper, Gets, Ground,...3 happened, Whatever, Your, decom, Getting, Got, overlooked, Crack, )?, He, police, !?, happens, Suc, sets, what, Detective, GOT, Whatever, SET, Getting, Flying
glued, hid, melted, sent, breaths, etz, breath, Breath, baptized, watch, putting, tongues, braces, put, hid, bleach, icating, burying, aver, lifting, Illuminati, orneys, melting, withdrawing, numb, radios, inserts, amins, avert, breathing, puts, informants, lifting, hide, conscience, recommending, withdrawn, ransom, catch, Gael, Vern, roth, ears, Put, gins, breathed, attorneys, loss, biblical
32
hid, dropped, raided, fought, ungle, Hide, destruct, smuggled, abandoned, looted, attacked, barric, slid, dodged, drop, shut, drowned, hide, destruct, buggy, battled, shutdown, Hide, Attack, hid, rawl, inaccessible, avalanche, slipped, deleted, rawling, encrypted, withdrawn, Killed, dug, dropping, hoard, weapon, swallowed, defensive, destroy, exited, destroy, fight, Fighting, lost, deny, suppress, encrypt, aggressively
37
hopped, chats, pumped, paints, backed, spun, tread, coached, reefs, privately, noodles, buddies, malls, whisper, endorsements, squeezed, pals, blush, comed, edits, rallies, gigs, recol, mocked, curs, Bare, bubbles, warmed, chat, profiles, emails, Dreams, pads, chalk, interviewed, sneakers, rocked, Gloves, hubs, docs, shaved, Rise, primaries, listened, shy, essays, whispers, leeve, girlfriends, socks
45
Table Tokens with the largest La norm of entries for the COPA task. See Section C for more details.
21

---

Under review as a conference paper at ICLR 2023
1#
Tokens x with largest norm
Pa
gression, rium, History, orer, aic, history, oration, ré, orative, amic, history, version, ural, osa, avage, ory, lia, range, History, rica, nation, root, USE, á, ination, ulation, mentation, issance, state, rum, adal, idden, jection, oly, 6, esis, orean, discovery. ria. ada, uration, entry, ord, verse, inations, ugal, itus, olics, ESSION, ativity
0
to, in, for, and, of, S, the, be, 's, by, on, or, from, or, with, ly, a, on, in, under, an, as, I, and, !, about, er, In, but, A, is, ed, a, that, ers, S, ing, now.).
5
of, and, for, morph, votes, elector, with, uild, igraph, tatt, Assignment, contribut, advant, are, hod, Voters, matically, Init, rede, olon, on, rehabilit, neum, mog, looted, req, by, Claim, the, ynchron, dule, promot, socio, portfolios, goto, vulner, vote, setup, nominate, anism, s, subscrib, iop, lihood, slot, elist, ramid, ysc
10
in, In, be, In, being, Straw, -. its, a. Majority, of, Latest, the, Jack, ine, latest, it, Lawyers, Watts, "-, Massachusetts, their, been, ure, Till, ".,, Signs, "" Seventh, ?", Taxes, Atlanta, !", electric, at, IN, ide, Current, Ladies, KP, Jersey, Students, Knights, it, Anders
22
Se, Hum, Brazil, Mur, Hur, aver, Hum, Yugoslavia, Mour, jud, a, Hawai, Pag. Kant, ibal, Malaysia, EFF, Hur, adj, mur, Islam, and, Guinea, Britain, Sadd, Def, Niger, Holland, amus, Hay, Ma, Appro, Mur, Countries, Wid, Asians, Nor, else, Calendar, Hed, Ved, Idom, english, Hind, mur, bury, Ded, hol
32
[SEP), Sk, Ble, Gre, cloud, Else, - +,,
uran, cs, Ever, 2048, Ble, Keefe, Hyp. athan, Lib, Fra, Exp, bro, Edit, Ros, Bean, Bo, Beck, Shell, sit, !., Saud, Phys, shell, OI, BLIC, -, Over, ea, orthy, Shot, pn, pas, ester, Reviewed, Spe, sell, 2024
37
Chance, Sw, chance, Nine, Shares, Chance, Scientists, Tw, Besides, Prof, chances, Sn, sw, TW, EFF, J, IJ, Besides, chance, Between, icist, GU, SW, pan, Ja, Psy, tw, Between, xon, Bj, Conj, Shares, Moh, UTH, Prediction, science, intend, Science, iov, Nine, jp. dds, NJ, Jr, Nin, etsy, Ibid, ymm, Reporting
45
Table 9: Tokens with the largest La norm of entries for the RTE task. See Section for more details.
22

---

Under review as a conference paper at ICLR 2023
1#
Tokens x with largest norm
Pa
", didn, doesn, don, ".... Didn, Doesn, didn, doesn, Don, wouldn, Wouldn, Does, ", couldn, DON, Did, hadn, But, Don, Isn, ).". DON, shouldn, Obviously, Obviously, Isn, don, hasn, )), Does, "?, J.". wasn, Did, J.".... Naturally, ...".."" Would, " But, "; Naturally, J., .). DOES
't, !,, ?, didn, not, to, +., *.
). the, .. J.. considered, doesn, in, .), a, /.. -L, you, don, shouldn, O., hasn, ;, for, thought, weren, hadn, thought, wasn, NOT, hair, aren, Said, , couldn, ....-ddeeed
5
Shant, Georg, Expect, Led, Assistant, Amph, Registered, Ear, McA, THEIR, Prev, Emb, -. Called, Gw, Alc, Until, Rhod, Introduced, that, Lat, Unt, UI, Sv. Gh, to, of, Fernand, clta, jac, unch, Ov, Sebast, apologised. JOHN, !".. L1, hid, Somewhere, Been, Recently, and, Somebody, Fram, Coh, ").. Sty, Elsewhere, Unt
10
's. 're. A. 've, the, a, her, be, A, a, have, DOES, LIKE, ?", "? 'd )2, ?.s, s, ABOUT, Like,
22 Pant, didnt, 'm, "? E, The, doesnt, Was, re, ie, Surely, 'II, Corinth, At, Across, your, their, 2,, THEY, ...?, or, Fra, HOW.) /
"? he, "?, ?,, 2, 't, " ...2,
?" :. He, 12, 2. He, I, and, ?",, ?!", ?", ?)..
32
P. he, .., ?1, she, +., ). !?".
")., ???, !., ).", we, CLOSE, " "I ".. ????,
7, 're."" 3"
of, he, B, B, in, I, and, m, -, 8, 'd, by. for, ;or,;,he b, on, you, 1. He, to, 've y, 're, ed, with, 'll. a, back, the, b, she, He, E.C
2, :.
37
't. not, NOT, the, not, Not, never, Not, 's, 're. NOT, 've, you, of, [SEP], t, nt, Never, The, in, NEVER, he, to, the, [CLS], hardly, never, neither, I, 'm, cannot, no, The, annot, it, Their, me, didnt, He, and, doesnt, Ear, a, Never, none, if. on, nobody
45
Table 10: Tokens with the largest La norm of P entries for the CB task. See Section for more details.
23


Reviews:
[{"review_id": "vcI8M3n6SD8", "reviewer": "Reviewer_ozTx", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors proposed ahead-of-time (AoT) P-Tuning, a new prompt-tuning method that adds an input-dependent offset to each layer's activation (based on the vocabulary index). The proposed method does not increase the sequence length, leading to now significant computation increase. It achieves competitive transfer learning accuracy vs. cost trade-off. ", "summary_of_the_review": "Please refer to the strength and weaknesses section for details. I would like to hear from authors' feedback for the final decision. ", "strength_and_weaknesses": "Strength:\n1. Firstly, the paper is generally well-written and easy to follow.\n2. The proposed method does not increase the sequence length, which does not lead to significant computation overhead for the tuned model.\n3. The proposed method achieves competitive tuning accuracy compared to existing P-tuning work. \n\nWeakness:\n1. Compared to P-tuning, the work is more similar to adapter-based tuning (since the offset is input-dependent; packing multiple tasks in a batch does not sound like a more important factor). However, the authors did not compare the proposed method with existing adapter-based tuning methods, which also does not significantly increase computation.\n2. The latency measurement setting (batch size 256, sequence length 128) seems unfair. The sequence length is smaller than usual, and the batch size is larger than usual, which all favors the proposed method, leading to a smaller average overhead. Can the authors also compare the latency with P-Tuning using a batch size 1, sequence length 1024, and discuss if the overhead is still smaller than P-tuning?\n3. The comparison also left out some recent state-of-the-art lite tuning methods that also do not increase sequence length like LoRA. I think it is not fair to exclude LoRA and adapter tuning in experiments, since in many cases, people would perform inference with single batch size.  ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "q5x_Jxy_X8a", "reviewer": "Reviewer_YHHD", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies the problem of parameter-efficient fine-tuning for large transformer-based models. More specifically, authors consider a subset of parameter-efficient fine-tuning methods that support multi-task inference.\nAuthors extend the popular soft prompt-tuning (or p-tuning) technique, resulting in a technique called \"Ahead-of-Time P-tuning\". Authors demonstrate that their method can match or outperform prefix tuning (p-tuning v2) on standard GLUE / SuperGLUE benchmarks on popular MLM transformers. The paper also emphasizes that Ahead-of-time p-tuning allows more efficient inference than standard prefix tuning.", "summary_of_the_review": "This is a well-written, easily reproducible paper that has a fundamental flaw: it unfairly ignores many advanced baseline methods based on - to the best of my knowledge - an incorrect assumption about their multi-task capabilities. Above, I made an argument (and a code snippet) showing that the more advanced baselines can in fact support multi-task inference in several ways. If my argument is correct, the paper is fundamentally flawed and will mislead readers into disregarding important results in parameter-efficient fine-tuning. As such, I will only recommend accepting the paper if either (1) my argument is formally proven incorrect, with minor revisions as requested or (2) my argument is accepted, in which case the paper will need a major revision / repositioning that may be difficult to review in one round.", "strength_and_weaknesses": "Preface: I strongly believe that __the current manuscript can mislead the reader__ by unjustly omitting SoTA parameter-efficient methods. I make my case below and suggest several options to correct that. I am also open to discussion if authors disagree with my assessment.\n\n__Strengths:__ \n- the paper is generally well-written and easy to follow, apart from minor typos\n- authors work on a real-world problem, optimizing for the types of models that are often used in NLP applications\n- the proposed method is evaluated in a fairly standard setup, making it easier to understand the reported statistics\n- authors make considerable effort to make their work reproducible.\n\nRegardless of my case against the paper, I deeply respect authors' intent to specify the exact versions and even providing a dockerfile that refers to pytorch/cuda in use. It would also be nice - but not critically required - to include some human-readable instructions on running the code, e.g. readme.\n\n__Weakness:__\n\nAs authors note in the paper, P-tuning v1 and v2 have been largely surpassed by methods such as LoRA. However, authors decide to exclude these methods from experiments because \"Adapters do not imply performing multi-task inference\" (see full quote below‡).\n\nTo the best of my understanding, the only way this statement _could_ be true is when simultaneously inferencing multiple tasks in one batch - since non-simultaneous multui-task inference with adapters can be done by simply swapping adapters between batches. Disregarding the fact that this is a fairly niche scenario, it is simply wrong: __both LoRA and simpler adapters (Houlsby et al. 2019) support simultaneous multi-task inference,__ and actively run it in many - though not most - applications.\n\nFor Houlsby et al (2019) adapters, if every sequence in a batch uses its own adapter weights, you can run multi-task inference using batched GEMM, (e.g. bmm in PyTorch). This way, each sequence can be processed with its own adapter. For simplicity, I provide a simple code snippet to illustrate this principle later at the end of this section†.\nLoRA adapters are parallelized the same way, except that there is no nonlinearlity, and the adapter outputs are applied to Q/K/V matrices separately.\n\nOn top of that, industrial frameworks for inferencing with adapters typically optimize this process by (1) grouping together samples for the same task and (2) writing custom kernel that runs BMM with _few_ adapters per batch, instead of one adapters per sample. However, even when this technique is not used, it is trivial to run the common parts of the transformer model in parallel for many tasks, then use a for loop to apply task-specific bits.\n\nFinally, there are other methods for parameter-efficient fine-tuning that belong to neither adapter nor p-tuning category. Notably,\n\n1. IA3 (Liu et al, 2022; [1]) can run multi-task inference out-of-the-box and claims to outperform p-tuning in low-resource settings at no extra computation cost\n2. BitFit (Zaken et al, 2021 [2]) can run multi-task inference out-of-the-box and claims to be competitive with p-tuning at no extra inference cost\n3. FISH (Sung et al, 2021 [3]) can run multi-task inference with sparse batch GEMM, similar to adapters\n\nIn its current state, the paper may mislead a non-experienced reader into thinking that if they want to run multi-task inference, they would be restricted to P-tuning variants and not allowed to use other parameter-efficient fine-tuning methods - and thus may end up with an inferior solution. I believe that this is a major concern that outweighs several positive aspects listed in the \"Strengths\" section.\n\nI suggest two ways this could be potentially alleviated:\n\n1. comparing against all the relevant baselines for parameter-efficient fine-tuning (adapters, BitFit / IA3, FISH, etc), which may yet reveal that AOT p-tuning outperforms them in some applications;\n2. stating, _throughout the paper,_ that this is a specific optimization for prompt tuning - and still showing how it compares to other parameter-efficient prompt-tuning methods, even if unfavorably;\n\nWhile I give authors the benefit of the doubt, it may be difficult to address this concern in the time-frame for author response. For performance comparisons, if authors are not willing to implement efficient multi-task inference for each method, their performance can be upper-bounded by single-task code with popular inference frameworks (e.g. by extending FasterTransformer[4]).\n\n\n‡ - full quotes, for the reference\n\n> One may note that the proposed method is more similar to Adapters Tuning (Houlsby et al., 2019) than P-Tuning. Although, Adapters do not imply performing multi-task inference, thus we refer to the proposed method as a variant of P-Tuning, rather than a special case of Adapters.\n\n> Based on this experimental design choice, we exclude experiments with Adapters (Houlsby et al., 2019; He et al., 2022), as well as with LoRA (Hu et al., 2022). While a wide range of efficient fine-tuning methods could be similar to the proposed method (Ding et al., 2022; He et al., 2022), they do not allow to perform multi-task inference, which is the motivation to use AoT P-Tuning.\n\n\n† - sample code for inferencing adapters for multiple tasks in parallel\n\n```python\nimport torch   # tested with torch 1.10.1, no cuda required to run this test\nbatch, seq_length, d_model, d_adapter = 3, 5, 1024, 8\n\n# pre-adapter activations, e.g. outputs of previous layer\ninputs = torch.randn(batch, seq_length, d_model)\n\n# adapter weights, one for each sequence in a batch\nadapters_weight1 = torch.randn(batch, d_model, d_adapter)\nadapters_weight2 = torch.randn(batch, d_adapter, d_model)\n\n# apply adapters in parallel via batched GEMM\nadapters_hidden = torch.bmm(inputs, adapters_weight1)\nadapters_nonlinearity = torch.relu(adapters_hidden)\nadapters_out = torch.bmm(adapters_nonlinearity, adapters_weight2)\n\n# verify that this is equivalent to non-simultaneous computation\nfor i in range(batch):\n    weight1, weight2 = adapters_weight1[i], adapters_weight2[i]\n    reference = torch.matmul(inputs[i], weight1).relu().matmul(weight2)\n    assert torch.allclose(adapters_out[i], reference)\n```\n\n\n[1] Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning, Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, Colin Raffel , https://arxiv.org/abs/2205.05638\n\n[2] BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models, Elad Ben Zaken, Shauli Ravfogel, Yoav Goldberg\n\n[3] Training Neural Networks with Fixed Sparse Masks, Yi-Lin Sung, Varun Nair, Colin Raffel , https://arxiv.org/abs/2111.09839\n\n[4] https://github.com/NVIDIA/FasterTransformer", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "gJ6TGCWDjP", "reviewer": "Reviewer_BA6Y", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes a new method to improve the inference efficiency of a family of parameter-efficient fine-tuning methods. The authors propose to add input-dependent biases to the transformer's weight matrices (Q, K, V) instead of concatenating prefixes which results in decreasing the dimensions of matrices needed to evaluate the transformer attention. Further, the authors propose two reparameterization tricks to make the calculation of these matrices tractable. Evaluation on GLUE and SuperGLUE benchmark datasets shows that the proposed method performs on par (sometimes better) with previously proposed parameter-efficient fine-tuning methods while being faster at inference. ", "summary_of_the_review": "In this work, the authors propose a new method for improving the inference time of parameter-efficient methods. The writing of this paper is clear and the method is well presented. Although the idea is novel and works well when compared to P-Tuning, authors miss out on some of the most important baselines. Rationale provided by the authors to skip those baselines is unjust and thus I cannot recommend accepting this paper and thus I vote to reject this paper.", "strength_and_weaknesses": "Strengths:\n\n+ The motivation for this work is well-founded.\n\nWeaknesses / Questions\n\n- Authors decide to exclude recent parameter efficient tuning methods which seem unfair. Their rationale for skipping those methods is that those methods do not work with multi-task inference. Even if that is the case, authors should still compare their method to the recent baselines.\n- Without the above evaluation, it is hard to comment on the efficacy of the proposed method.\n- According to table 2, the std for AoT P-Tuning is generally considerably higher than baseline models. This suggests that the model might not be stable.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "UL70uFQfMjP", "reviewer": "Reviewer_8wRe", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a new parameter-efficient method for fine-tuning a transformer model, called AoT P-tuning. Experiments show that it performs at least as well as P-tuning v2 while being 1.3x faster. As P-tuning, AoT P-tuning enables efficient multi-task inference.", "summary_of_the_review": "This paper is overall quite solid: it proposes a new method, it provides empirical validation, and it is well written.", "strength_and_weaknesses": "This paper introduces a new variant of P-tuning and shows its effectiveness (especially in terms of inference speed). The paper is well written and I don't see any particular weaknesses beside the set of experiments being somewhat limited (as pointed out by the other reviewers).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject