ADePT: Adaptive Decomposed Prompt Tuning for Parameter-Efficient Fine-tuning
Authors: Pengwei Tang
Xiaolin Hu
Yong Liu
Venue: ICLR 2025 Poster (2025)
Keywords: Natural Language Processing
Large Language Models
Parameter-efficient Fine-tuning
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: We propose  "Adaptive Decomposed Prompt Tuning" as an improvement over the Prompt Tuning method, and our method can map the input token embedding into better embedding space , enabling better performance for adaptaiton.

Abstract: Prompt Tuning (PT) enables the adaptation of Pre-trained Large Language Models (PLMs) to downstream tasks by optimizing a small amount of soft virtual tokens, which are prepended to the input token embeddings. Recently, Decomposed Prompt Tuning (DePT) has demonstrated superior adaptation capabilities by decomposing the soft prompt into a shorter soft prompt and a pair of low-rank matrices. The product of the pair of low-rank matrices is added to the input token embeddings to offset them. Additionally, DePT achieves faster inference compared to PT due to the shorter soft prompt. However, in this paper, we find that the position-based token embedding offsets of DePT restricts its ability to generalize across diverse model inputs, and that the shared embedding offsets across many token embeddings result in sub-optimization. To tackle these issues, we introduce \textbf{A}daptive \textbf{De}composed \textbf{P}rompt \textbf{T}uning (ADePT), which is composed of a short soft prompt and a shallow token-shared feed-forward neural network. ADePT utilizes the token-shared feed-forward neural network to learn the embedding offsets for each token, enabling adaptive embedding offsets that vary according to the model input and better optimization of token embedding offsets. This enables ADePT to achieve superior adaptation performance without requiring more inference time or additional trainable parameters compared to vanilla PT and its variants. In comprehensive experiments across 23 natural language processing tasks and 4 typical PLMs of different scales, ADePT consistently surpasses the leading parameter-efficient fine-tuning methods, and even outperforms the full fine-tuning in certain scenarios. We also provide a theoretical analysis towards ADePT. Code is available at https://github.com/HungerPWAY/ADePT.

Published as a conference paper at ICLR 2025
ADEPT: ADAPTIVE DECOMPOSED PROMPT TUNING FOR SHRAMETER-EEFICHEE
FINI-TUNING
Pengwei Tang, Xiaolin Hu, Yong Liu" Renmin University of China, Beijing, China
(tangpwei, x ao inhu, 1 liuyonggsa i e ruc. edu.ce
ABSTRACT
Prompt Tuning (PT) enables the adaptation of Pre-trained Large Language Mod- els (PLMs) to downstream tasks by optimizing a small amount of soft virtual tokens, which are prepended to the input token embeddings. Recently, Decom- posed Prompt Tuning (DePT) has demonstrated superior adaptation capabilities by decomposing the soft prompt into a shorter soft prompt and a pair of low-rank matrices. The product of the pair of low-rank matrices is added to the input token embeddings to offset them. Additionally, DePT achieves faster inference com- pared to PT due to the shorter soft prompt. However, in this paper, we find that the position rosition-based token embedding offsets of DePT restrict its ability to generalize across diverse model inputs, and that the shared embedding offsets across many token embeddings result in sub-optimization. To tackle these issues, we introduce Adaptive Decomposed Prompt Tuning (ADePT), which is composed of short soft prompt and a shallow token-shared feed-forward neural network. ADePT utilizes the token-shared ferwaforward neural network to learn the embedding off- sets for each token, enabling adaptive embedding offsets that vary according to the model input and better optimization of token embedding offsets. This en- ables ADePT to achieve superior adaptation performance without requiring more inference time or additional trainable parameters compared vanilla PT and its variants. In comprchensive experiments across 23 natural language processing tasks and 4 typical PLMs of different scales, ADePT consistently surpasses the other leading parameter-efficient fine-tuning methods, and even outperforms the full fine-tuning in certain scenarios. We also provide a theoretical analysis towards ADePT. Code is available at / htw.:/bb..coomoooeeee..eaaa TAA.//// ADePT
INTRODUCTION
Recently, Pre-trained Large Language Models (PLMs) (Raffel et al. 2020 Touvron et al. 2023) have seen rapid development, with commonly used models now on the scale of hundreds of millions and billions of parameters. Full fine-tuning (FT) of these PLMs requires substantial GPU resources, which a common challenge faced by both academia and industry. To alleviate this resource- intensive issue, Parameter-Efffcieet Fine-Tuning (PEFT) (Houlsby et al.) 2019 Liu et al. 2022]
Hu et al. 2021 Ben Zaken et al., 2022) methods have gained significant attention and have seen breakthrough progress. These PEFT methods tune only small amount of the internal parameters of a model or extra parameters, allowing PLMs to adapt effectively to target downstream tasks while maintaining performance comparable to FT.
The vanilla Prompt Tuning (PT) (Lester et al. 2021) uses a trainable soft prompt prepended to the input token embeddings (Lester al.) (021), as shown in Figure Tal The few trainable parameters make PT one of the mainstream methods for parameter-efffiien fine-tuning. The improvements to PT can be categorized into four paths: the first path involves adding soft prompts to each layer of one PLM (Li & Liang 202I): the second path involves stabilizing the optimization of soft prompt through shallow network with a residual connection (Razdaibiedina et al., 2023); the third path involves using soft prompts that had already been trained by other methods for transfer learning (Vu et al. 2022 Asai et al. 2022; Wang et al. 2023): the fourth path uses input token embedding offsets
"Corresponding Author.

---

Published as a conference paper at ICLR 2025
Low rank matriees
Input Token bimbeddingg
Br
leput Token anbeokkings
Phompi
Irput Token Enbzzoding
food-Foraand Neural N'Newoork
Prompt
Prompt
Pro-Trinna Language
Pre Trained Large Language Models
Pro-Trainod Lange Langinge Models
Prompl Tuning
Decomposed Prouupt Tuning
(c) Adaplive Decomposed Prompl Tuning
The overview of Prompt Tuning (PT), Decomposed Prompt Tuning, and Adaptive Decom- posed Prompt Tuning (ADePT). PT uses soft prompt prepended to input token embeddings. DePT uses short soft prompt and offsets the input token embeddings using a pair of low-rank matrices. ADePT uses a short soft prompt and offsets the input token embedding using token-shared shallow feed-forward neural network. ADePT can adaptively give input token embedding offsets based on input tokens, while DePT can only give position- based input token embedding offsets. Moreover, the use of a short soft prompt makes DePT and ADePT faster during inference.
Figure I:
to map the input token embedding into better embedding space (Shi & Lipani 2024). The first three approaches either require increasing trainable parameters necessitate additional transfer learning. while the fourth approach requires neither. For the fourth path, Decomposed Prompt Tuning (DePT) (Shi & Lipani 2024) pioneers the shift in attention from soft prompt to applying input token em- beddings offsets to the input token embeddings. As shown in Figure [b] DePT learns embedding offiets AE- AB [ Aeg, ... Ae,] by optimizing pair of low-rank matrices A Rexr, and B E Rx where denotes the length of input tokens, d denotes the dimension of input to- kens and rs denotes the maximum rank of matrices A and B. Given the input token embeddings E [e1, E2, es] E Re the updated token embeddings E' are:
E' = E+AA E AB
DePT also learns short soft prompt P. Rl xd with length 1,- The input token embeddings E, after DePT processing, are formulated as [Px, E + AB). In this paper, we focus on improving the fourth path by providing better offsets for the input token embeddings. Additionally, our improve- ment of the fourth path is orthogonal to the first three and can be integrated with their methods.
The main contribution of DePT is the use of learnable token embedding offsets, which map the input token embeddings into a more suitable embedding space. DePT has demonstrated promising results across various tasks and PLMs, but it still has limitations. The updated token embeddings formulated as + AE- = [e1 + Ae₁ E2 + Ae2. ... e, + Aez]. We can observe that target downstream task, the input token embeddings E vary, while the offsets A position-based and fixed after fine-tuning. For a token e, its position the task can be arbitrary. meaning that both e Ae, and + Ae; can be its updated embedding in this task, where < s, 1 S و s, + j. This means that within the same task, the same token may have multiple different token embeddings, implying that the token embeddings of DePT violate the uniqueness token embeddings. In Section we design two experiments that reveal two limitations of DePT: (1) the offsets obtained by tokens at different positions may lead to different prediction outcomes; (2) to ensure that token embeddings are unique as possible, the offsets in DePT are often much smaller than the input token embeddings, leading to sub-optimization.
3.2
Based on the above analysis, the embedding offset for each token should be the same at any position. Additionally, to enhance expressiveness, the offset corresponding to each token should be as unique as possible. Surprisingly, we find that using function of the token embedding can meet the above requirements, i.e., using the function f(e) to the offsets of input token embedding e. To avoid increasing the number of trainable parameters compared to vanilla PT. utilize shallow and narrow forward neural network to approximate the function for offset prediction. As shown we propose Adaptive Decomposed Prompt Tuning (ADePT), which uses short soft
in Figure Tc
2

---

Published as a conference paper at ICLR 2025
prompt and two-layer feed-forward neural network, where the short soft prompt is prepended to the input token embeddings and the feed-for beed-forward neural network is shared by all tokens and produces the offset of each token. The use of a short prompt makes ADePT achieve faster inference speeds than the vanilla PT and comparable inference speeds compared to DePT.
In summary, the main contributions of this paper are as follows:
We point out that the limitations of position-basse -baed token embedding offsets in DePT.To tackle these issues, propose Adaptive Decomposed Prompt Tuning (ADePT), which can produce unique token embedding offset for each token.
ADePT employs token-shared feed-forward neural networks to learn unique offset for each token, which can adaptively adjust the token embedding offsets based on model input and allow the training parameters to achieve better optimization.
Extensive evaluations across 23 NLP tasks and 4 PLMs of different scales show that ADePT surpasses leading PEFT methods, including full fine-tuning in certain scenarios. We theo- retically analyze that ADePT possesses better expressive power than PT and DePT.
2 RELATED WORKS
Parameter- Effficeent Fine-tuning: PEFT methods adapt PLMs to downstream tasks by optimizing small number of parameters, significantly reducing computational resource demands. PEFT meth-
ods can be divided into four (ypes:(1) Adapter and its variants (Houlsby et al. [2019] He et al. 2022]
Rücklé al.,
2021 Ivison & Peters 2022); (2) Low-rank Adaptation (LoRA) and its variants (Hu
et al.
2021
Luu et al. [022] Kopiczko et al. [2024]; (3) Prompt Tuning (PT) and its variants (Lester
ct al.
2021
Li & Liang [021] Vu et al. 2022] Asai et al. 2022 Wang et al. 2023 Ma et al. 2022
[
Xiao et al.
2023 Razdaibiedina et al. 2023 Shi & Lipani 2024); (4) other methods (Ben Zaken
ict al.
2022
Guo et 2021). The adapter inserts new modules into the transformer blocks (Houlsby
cct al.
2019 Hyperformer (He et al. (022) uses shared hypernetwork to generate task-conditioned
adapters, reducing the trainable parameters in multi-task scenarios. AdapterDrop (Rücklé et al. 2021] removes adapters from lower transformer layers during training and inference, which reduces the computation overhead. Hyperdecoders (Ivison & Peters 2022) generate input-specific adapters using shared decoder for multi-task scenarios. LoRA (Houlsby al., 2019) adopts the matrix product of a pair of low-rank matrices to approximate the updates of corresponding parameters. (IA) (Liu et al. 2022) rescales internal activations only using learned vectors injected into the at- tention and feedforward modules. Prompt tuning (PT) (Lester et 2002 addapts PLMs to the new downstream tasks by optimizing learnable virtual tokens. LST (Sung et al.) 2022) reduces mem- ory requirements by training small side network, achieving adaptation without pnoo-propagating through the main network. BitFit (Ben Zaken et al.] (022) only tunes the bias of PLMs, which extremely reduces the trainable parameters. Diff pruning learns sparse "diff" vector to modify small percentage of pre-trained parameters, enabling efficient adaptation (Guo et al. 202T). Our proposed ADePT is an improved variant of PT, which can also be unified with other PEFT methods.
Prompt tuning and its variants: PT (Lester et al. 202I) enables the adaptation of PLMs to down- stream tasks by learning a soft prompt appended in front of model input. Prefix-tuning (Li & Liang 2021) can be viewed as an extension of Prompt Tuning applied across the entire depth of the model. SPoT (Vu et al. 2022) utilizes trained prompts from source tasks initialization, but it requires extensive search to find the optimal initialization. ATTEMPT (Asai et al., (022) adopts an atten- tion module to compose the trained prompts of source tasks. MPT (Wang et al. 2023) learns transferable prompt by distilling from multiple task-specific prompts. XPrompt (Ma et al. 2022) empirically demonstrates the negative impact of trained prompt tokens and proposes hicrarchical structured pruning for a trained soft prompt, which can be seen as post- post-training method. DPT (Xiao et al. 2023) adopts the product of two low-rank matrices to approximate the soft prompt. Residual Prompt Tuning (Razdaibiedina et al.] (023) reparameterizes soft prompt embeddings with a shallow feed-forward network and residual connection, while ADePT uses the feed-forward net- work to learn input token embedding off'sts--a distinct objective. DePT (Shi & Lipani 2024) first discovers that the input token embedding offsets can enhance the performance of PT and uses short soft prompts to accelerate inference. ADePT produces adaptive input token embedding offsets via shallow token- shared networks, addressing the limitations of DePT mentioned in Section] I With a comparable number of trainable parameters, ADePT can outperform both vanilla PT and DePT.
3

---

Published as a conference paper at ICLR 2025
The comparisons between DePT and ADePT on RTE and BoolQ tasks. All results are based on the T5-base model. The numbers within "DePTO" indicate the amount of cyclic left shift applied to the position-based imput embedding offsets of DePT.
Table 1:
DePT (0)
DePT (50)
DePT(100)
DePT(150)
DePT(200)
ADePT
RTE BoolQ
79.1
78.4
79.9
79.1
78.4
82.0
78.4
74.7
73.1
70.9
73.9
80.2
Table 2: The Mean and Variance of input token embeddings and embedding offsets of DePT and ADePT on RTE and BoolQ tasks. All results are based on the T5-base model. The Mean and Variance are calculated from the tokens of the entire training dataset. The "mean()" and "variance()" refer to the corresponding task inside the parentheses.
Mean (RTE)
Variance (RTE)
Mean (Bnolq)
Variance (Boolq)
Input Hmbeddings
6.07
16.29
7.93
9.98
The offset (absolute value) of DePT
0.01
0.06
0.02
0.01
The offset (absolute value) of ADePT
8.31
5.45
6.09
3.70
3 METHOD
In this section, we revisit the preliminaries of PT and DePT (Section 3.1 and analyze the limitations of DePT (Section 3.2p. Then, we introduce our proposed Adaptive Decomposed Prompt Tuning
(ADePT) in Section 3.3 followed by theoretical analysis of ADePT in Section 3.4
3.1 PRELIMINARIES: PROMPT TUNING (PT) AND DECOMPOSED PROMPT TUNING (DEPT)
Prompt Tuning(PT). Let D = (X+, yi J. be the training dataset of the target downstream task T. where is the number of the training data. Given a PLMs with parameters O each input X, first mapped to token embeddings E, Rx by tokenizer and embedding layer. where denotes the maximum length of input tokens and d denotes the dimension of the input token embeddings. The objective of PT is learn soft prompt P E Rixd to enable the adaptation of PLM to the target downstream task. Here, denotes the length the soft prompt. The soft prompt is prepended to the input token embeddings E. The loss PT for the target downstream task is formulated as:
CPT Σ log P(y: [P. O)
where LpT is the loss function only optimized with regard to the soft prompt P.
Decomposed Prompt Tuning (DePT). DePT adapts PLMs to the new target downstream task via short soft prompt P., 12,, Xd and a pair of low-rank matrices, i.e., A R and B IR xd where 1,, denotes the length of short soft prompt and < min(s, d) denotes the maximum rank of matrices A and B. Similar to PT. the short soft prompt of DePT prepended to the frozen input token embeddings. The product of low-rank matrices of DePT is used as the offsets of the input word embeddings. The loss of DePT the target downstream task formulated as:
Rd
CDATT - Log P(y
[Ps, E, + AB]0)
where the loss function LDPT optimized only with respect to the short soft prompt P., and the pair of low-rank matrices A and B.
3.2 THE LIMITATIONS DEPT
In this section, we explore several key factors that limit the performance of DePT.
Let AB = AE, the updated input token embeddings DePT is formulated as [e + dei, e2 + Aeg, C, De.]. For downstream task, the input token embeddings E vary. However, the
4

---

Published as a conference paper at ICLR 2025
pair of low-rank matrices are fixed after adaptation, meaning that the input token embedding off- sets AE are resition-basedd Assume there are input token embeddings [a, b, c] (here, we omit the right padding), the updated token embeddings Assume we have a meaningless sequence that does not affect the prediction performance, denoted as [t,, t2]. Prepending the meaningless sequence to input token embeddings [a, b, c). the updated input token embeddings are [t] + Δei, to + Aes, a + Ae3, b + Aes c Des]. We can ob- serve that offsets of [a, b, cl are different. The offiets of the former [ Aes, Aes], while the offsets of the latter is [Aes, Des: Aes]. Just adding meaningless sequence [t,, ty] that does not affect the prediction results, the token embedding offsets for [a, b, c| become different. These embedding offsets make the embeddings of each token non-unique in one task, which may be detrimental to the model performance. To simulate this scenario, we design an ideal experiment where we cyclically left shift the column vectors of A E and then test the model per- formance, as shown in Table I Let AE be the cyclic left shift of AE by j positions, defined as E1, .yyl. For example,
formulated as [a + Aei, b + Ae2, + Aez].
RTE, the performance of DePT is decreased by 0.7 points after cyclically left-shifting the position- based embedding offsets by 50 positions, and increased by 0.8 points after shifting by 100 positions. On BoolQ, the performance of DePT is worse than the original after cyclically left-shifting the position- embedding off- sets. Table [ shows that position-based embedding offsets in DePT can cause unstable prediction performance across different positions.
AE' [ Ae2+s.
Table reports the mean and variance of elements in input token embedding e and the elements in embedding offset Ae from DePT across two entire training datasets, i.e., RTE and BoolQ tasks. All results are calculated by their absolute values. We can observe that the mean and variance of elements in Ae are only few percent of those of elements in e. For example, on the RTE task, the mean absolute value of elements Ae only 0.01, while the mean absolute value of elements in e is 6.07. This implies that DePT makes only minor changes to the input token embedding space, which may result in its inability to map the input token embeddings to the appropriate embedding space. This is because, for token e, its offset can be any position Ae,, where < <is I The requirement that tokens within the same task should be unique causes the offsets of DePT to become extremely small, leading to the sub-optimization of DePT. The elements in the embedding offsets of ADePT have a much larger range of values than that of DePT. The optimal embedding space may lie outside the range of DePT, whereas ADePT may be able to access this embedding space.
3.3 OUR METHOD: ADAPTIVE DECOMPOSED PROMPT TUNING (ADEPT)
The limitations of DePT lie in its position- based token embedding offsets. To address this issue, the input token embedding offsets should be tailored for model input, and the corresponding embedding for each token should be unique after being offset. We find that making the input token embedding offsets AE a function of the input token embedding can meet such requirements. For an input token embedding e, we want to get function f() which can produce the offset of this input token embedding, namely, f(e). For the input token embeddings E = [ee, e2, e2,..... .2.....,, ea]. the updated input token embedding can be formulated as E' E + AE [er + For example, the input token embeddings [a. b, c) can be updated as (a f f(a), b f(b). c+ f(c)]- Prepending the meaningless sequence [t,, t2] to input token embeddings (a, b, c). the updated input token embeddings are [4 + (ta)), l2 + (ta), a + ((a), b+ f(b). e [(e)]. We can observe that the offsets for (a, b, c) are the same in this two scenario, ie., (f))) f(e).. Therefore, if such a function f exists, we can achieve input token embedding offsets that are tailored for model input, and the embedding for each token is unique within a task.
.e + f(e))
To avoid increasing the number of trainable parameters, we use a shallow and narrow feed-forward neural network to approximate the function f. Thus, we propose Adaptive Decomposed Prompt Tuning (ADePT), which can offset the token embeddings adaptively based on the model input. We implement the shallow token-shared feed-forward neural network by a two-layer multi-layer perceptron (MIP).. It consists of down-projection matrix Wdown E IRdxr and a up-projection matrix WT up E R and a down- projectioon bias b, E R" and a up-projection bias by E Rdd. Here, r is the bottleneck size of the MLP. The updated input token embeddings by the shallow token-shared feed-forward neural network are formulated as:
E E, ReRRUEEEWWW..... W down + bi) W + b2-
(3)
5

---

Published as a conference paper at ICLR 2025
To ensure that the number of trainable parameters does not exceed that of the vanilla PT, we use short soft prompt Pa E R2 xd, similar to DePT. The loss of ADePT is formulated as:
LADePT = log
Σ log P(y: [Px. ; :0),
(4)
where the loss function CADePT is optimized only with respect the short soft prompt Ps, and the parameters of the feed-forward neural network Wdown Wup up- b1 and b2.
3.4 THEORETICAL ANALYSIS
In this section, inspired by Petrov al. 2024), we provide theoretical analysis towards ADePT.
The multi-head self-attention layer serves as crucial component in each transformer layer. We ana- lyze how PT and ADePT affect the first transformer layer. To simplify the analysis, let us consider single head self-attention 7 in the first layer, which is parameterized by W W W E Rddd, Given input sequence embeddings E (ei, e2..... e,) = with each a query vector e; passing through the single-head self-attention H. in the first layer is formulated as:
Rd the output of
Attention (e: W EWW EW?) Softmax ((e:W (EW?) ") EW?, (5)
where the scaling constant vdz is ignored for notation convenience.
For the vanilla PT with the soft prompt P = [p]. P...... Pil E Rixd, the output of query vector e; passing through the single-head self-attention H in the first layer is formulated as: concat PP E E concat| E.)
PT Attention ,
A
Σ
Ask
Oi
(6)
bios
scale
We (pxW
exp
with A,k
Ek exp e; W (ptW))²) + Σj exp (e:W (e, (e)
where A. is the attention score assigned to the prefix vector py: for e, Thus, in the first transformer layer, PT cannot affect the relative attention patterns across the content and only scales the attention patterns down while adding constant bias to the original output o, (Petrov al. 2024)..
For our proposed ADePT with the soft prompt neural network f, the output of query vector e; passing through the single-head self-attention H in the first layer is formulated as:
[Pi, .2..... p.] E Rixd and ferw-forward
o ADePT
Attention ((e, + (e.)) W. concat[P. E WE) concat E WE)
AuPeW
R=
(1 Σ Ai Softmax ((e:+ W (EE (B) (()) (E f (E)) w
exp + ((:) W (
(p&
with
Au
B
B
Σ exp ((le+
w (pa
k=
Σ exp ((le) ((e)+(((s)) W ((e) [(e))) 1=1
+
Hence, in the first transformer layer, ADePT can change the original relative attention patterns and add bias dependent the input, which makes ADePT have expressive power than PT.

---

Published as a conference paper at ICLR 2025
4 EXPERIMENTS AND RESULTS
4.1 EXPERIMENTAL SETUP
Tasks and Models. We conduct extensive experiments to validate our proposed ADePT. We con- sider four benchmarks and 4 other datasets: (1) GLUE (Wang et al.
2018) benchmark, which in-
cludes MNLI (Williams et al. 2018). QQF"1 QNLI (Rajpurkar et al. 2016), SST-2 (Socher et al.
2013), STS-B (Cer et | 2017 MRPC (Dolan & Brockett 2005), RTE (Giampiccolo et al. (007) and CoLA (Warstadt et al. 2019)::
which in- cludes MultiRC (Khashabi et al. 2018), BoolQ (Clark et al. 22019) WiC (Pilehvar & Camacho-
(2) SuperGLUE benchmark (Wang et
Collados 2019) WSC (Levesque et al. 2012), CB (De Marneffe et al. (019 and ReCoRD (Zhang
ct al.
2018: (3) MRQA 2019 Shared Task (Fisch et al. 2019), which includes Natural Ques-
tions Kwiatkowski et al. 2019). HotpotQA (Yang et 2018). SearchQA (Dunn et al.] 2017) and
NewsQA (Trischler et al. 2017):
7): (4) MBPP benchmark (Austin et al. (021), which is a code gener- ation task; (5) other datasets, which includes WinoGrande (Sakaguchi et al. 1021). Yclp-2 (Zhang ctt al. 2015). SciTail Khot et al. 2018) and PAWS- S-iiki (Zhang et (011)) Following (Asai et al,,
2022 Wang et al., 2023 Shi & Lipani| 2
evaluate our proposed ADePT on all datasets for the T5-base model (220M) (Raffel et al. 2020), except for MBPP and ReCoRD. For the T5- 3B model (Raffel et al. 2020 we focus on large and challenging datasets (i.e., MNLI, ReCoRD, Natural Questions, HotpotQA, SearchQA, and NewsQA) to differentiate the performance of vari- ous PEFT methods. For the decoder-only PLMs (i.e., CodeGen-350M (Nijkamp et al.. 2023) and Llama3-8B (Dubey et al., 2024),, we evaluate our proposed method ADe on MBPP benchmark.
Baselines. To evaluate our proposed ADePT, we compare it with five types of fine-tuning methods: (1) full fine-tuming (FT), which optimizes all the model parameters; (2) the vanilla PT (Lester et al.) 2021 where target prompt vectors are initialized with randomly sampled top vocabularies; (3)
the variants of PT using additional transfer or multi-task learning, including SPoT (Vu et al.) 2022),
ATTEMPT (Asai al., [202)), and MPT (Wang et al. (2033); (4) the variants of PT using input token embedding offsets. i.e., DePT (Shi & Lipani] 2024); (5) ofate-of-the-art PEFT methods including
Adapter(Houlsby et al., 2019), Dropter Ruckle et al. (02T), BitFit (Ben Zaken et al. 2022).
HyperFomer (Karimi Mahabadi al., (021). HyperDecoder (Ivison & Peters 2022), P-tuning (Liu) ct al. 2021). LoRA (Hu et al. 20211) LST (Sung et al. 2022). and their multi-task learning variants.
Implementation Details. Following Shi & Lipani (2024), we use 100 learnable virtual tokens as the soft prompt of PT. For our proposed ADePT, we adjust the hyperparameters to maintain an equivalent number of trainable parameters PT. For instance, in the T5-base model, the token embedding dimension is 768, so the number of trainable parameters is / x = 100 x 768 = 76800. Following Shi & Lipani (2024), we search the length of soft prompt from 20, 40, 60, and 80. For ADePT, if using 60 virtual tokens for soft prompt, the d, is got by solving the unequal equation 60 x 768 + 2 d, x 768 + d, + 768- < 76800. Thus, the d, S 19.49 and dl is set to 19 because the d is the integer. According to this calculation method, the corresponding d, values for soft prompt lengths of 20, 40, 60 and 80 are 39, 29, 19, and 9, respectively. For fair comparison of the T5-base model, we directly quote performance metrics from published papers (Mahabadi et al., 2021 Karimi Mahabadi et al. 2021; Asai et al. 2022 Wang et al. 2023 Sung et al. 2022; Shi & Lipani 2024). For T5-3B model, we consistently use 60 virtual tokens and bottleneck size 19.
Due to the lack of experimental results, for a fair comparison with the T5-3B model, we reproduce the experiments of the vanilla PT and DePT. For decoder-only PLMs, following Jain et al. (2024),
we use 10 virtual tokens for PT. virtual tokens and rank т. = for DePT, 7 virtual tokens and bottleneck size samples) based on T5 model, we follow the learning strategy of SSh & Lipani] (2024): we search the learning rate for the soft prompt from 3e 1, 4e 5e 1, and for the feed-forward neural network from le - 4, le - 5. For large datasets (> 70, 000 training samples) based on T5 model, we use learning rate 3 for the soft prompt and le for the feed-forward neural networks.
for ADePT, and rank 16 for LoRA. For small datasets (<. 70, 000 training
For the MBPP benchmark. following Jain et al. (2024). we use learning rates of le 3 for the prompting-ste -tnyle uning method, le 4 for LoRA.
https://www/ quora. wtwporaa.coo//gooooraattat.
7

---

Published as a conference paper at ICLR 2025
Table 3: The experimental results on GLUE and SuperGLUE benchmarks, with the associated size of trainable parameters. All results are based on the T5-base model. We report Pearson correlation for STS B, F1 for MultiRC (Multi), and accuracy for other tasks test metrics.
GLUE
SuperGLUE
Method
#Para
MNLI QQP QNLI SST-2 STS-B MRPC RTE CoLA Meun Multi Bool WIC WSC CB
Mean
Single- Bangle-Task Task Learning
Full Finetuning'
220M
86.8
91.6
93.0
94.6
89.7
90.2
71.9
61.8
84.9
72.8
81.1
70.2
59.6
85.7
73.9
Adapter
1.9M
86.5
90.2
93.2
93.8
90.7
85.3
71.9
64.0
84.5
75.9
82.5
67.1
67.3
85.7
75.7
Dropt
1.1M
86.3
90.2
93.2
93.6
91.4
86.3
71.2
62.7
84.4
72.9
82.3
68.3
67.3
85.7
75.3
BitFir'
280K
85.3
90.1
93.0
94.2
90.9
86.8
67.6
58.2
83.3
74.5
79.6
70.0
59.6
78.6
72.5
LORA²
86.3
89.0
93.2
94.3
90.1
75.5
63.3
85.3
68.3
67.3
92.9
LST
3.8M
85.6
88.8
93.3
94.1
90.4
71.9
58.1
84.1
PT
76.8K
83.4
90.2
93.1
91.9
90.2
90.1
78.8
60.7
84.8
65.7
63.7
50.8
51.9
67.9
60.0
DePT
76.8K
85.0
90.4
93.2
94.2
90.8
90.7
79.1
63.8
85.9
74.3
79.3
68.7
67.3
92.9
76.5
ADePT (ours)
76.1K
85.7
90.4
93.2
94.0
90.9
91.2
82.0
65.5
86.6
74.6
80.2
68.7
67.3
96.4
77.4
Additional Transfer Learning or Multi- Task Learning
Full Fine-tuning (m)
28M
85.7
91.1
92.0
92.5
88.8
90.2
75.4
54.9
83.8
74.4
81.1
70.0
71.2
85.7
76.1
Adapter (m)
1.8M
86.3
90.5
93.2
93.0
89.9
90.2
70.3
61.5
84.4
72.6
82.3
66.5
67.3
89.3
75.6
HyperFormer (m)
638K
85.7
90.0
93.0
94.0
89.7
87.2
75.4
63.7
84.8
72.9
82.5
69.0
67.3
85.7
75.4
HyperDecoder (m)
1.8M
86.0
90.5
93.4
94.0
87.7
71.7
55,9
83.7
70.4
67.1
61.5
72.0
SPoT (t)
76.8K
85.4
90.1
93.0
93.4
90.0
79.7
69.8
57.1
82.3
74.0
77.2
67.0
50.0
46.4
62.9
ATTEMPT (t)
232K
84.3
90.3
93.0
93.2
89.7
85.7
73.4
57.4
83.4
74.4
78.8
66.8
53.8
78.6
70.5
MPT (1)
77.6K
85.9
90.3
93.1
93.8
90.4
89.1
79.4
62.4
85.6
74.8
79.6
69.0
67.3
79.8
74.1
ATTEMPT (m)
96K
83.8
90.0
93.1
93.7
90.8
86.1
79.9
64.3
85.2
74.4
78.5
66.5
69.2
82.1
74.1
MPT (m)
10.5K
84.3
90.0
93.0
93.3
90.4
89.2
82.7
63.5
85.8
74.8
79.2
70.2
67.3
89.3
76.
sourced from (Asai et al.
2022
sourced from (Sung et al. 2022]
sourced from (Wang et al.
2023
sourced from (Shi
& Lipam 2024 (m) represents additional multi-tusk truining. (1) represents additional transter learning.
Table 4: The experimental results on MRQA 2019 Shared Task and other datasets with the associated
size of trainable parameters. All results are based on the T5-base model. We report the F1 for MRQA tasks and accuracy for other datasets as test metrics. The results are averaged over three runs and the subscripts denote standard deviation. All baseline results are quoted from (Shi & Lipani 2024).
MRQA
Others SciTail
Method
#Para
NQ
HP
SQA
News
Mean
WG
Yelp
PAWS
Mean
Full Fine Tuning
220M
75.1
77.5
81.1
65.2
74.7
61.9
96.7
95.8
94.1
87.1
Adapter
1.9
74.2
77.6
81.4
65.6
74.7
59.2
96.9
94.5
94.3
86.2
BitFit
280K
70.7
75.5
77.7
64.1
72.0
57.2
94.7
94.7
92.0
84.7
LORA
3.8M
72.4
62.3
72.5
56.9
66.0
58.2
97.1
94.7
94.0
86.0
PT
76.8K
67.9
72.9
75.7
61.1
69.4
49.6
95.1
87.9
55.8
72.1
SPoT
76.8K
68.2
74.8
75.3
58.2
69.1
50.4
95.4
91.2
91.1
82.0
ATTEMPT
232K
70.4
75.2
77.3
62.8
71.4
57.6
96.7
93.1
92.1
84.9
MPT
77.6K
72.0o.1
75.80.1
77.26.1
63.7o.1
72.2
56.5o.9
96.An.
95.5n.1
93.5o.
85.5
DePT
76.8K
73.2g.1
76.80.s
77.60.2
64.4p.1
73.0
59.0g.2
96.88 96.8o.1
95.6g.2
93.7T..
86.3
ADePT (ours)
76.
73.9p.o
77.10.1
78.
64.7o.1
73.6
59.10.9
96.8g.o
95.9g.3
93.77..
86.4
4.2 RESULTS BASED ON T5-BASE MODEL
#1 Performance on GLUE and SuperGLUE benchmarks.
As demonstrated in Table 3 our proposed ADePT surpasses leading PEFT methods. including Adapter, LoRA, BitFit, and ISST, on the GLUE and SuperGLUE benchmarks, while utilizing the least trainable parameters. ADePT outperforms the vanilla PT while using comparable trainable pa- rameters and less inference time. ADePT also outperforms the variants of PT using additional trans- fering learning, including SPoT, ATTEMPT and MPT while not requiring the complicated training and storage of soft prompts for source tasks. Remarkably, ADePT outperforms DePT, demonstrat- ing that the adaptive input token embedding offsets by oohrneshared feed-forward neural networks better than the position-based input token embedding offsets. Moreover, ADePT can out- perform the full finetuning method and the PEFT methods using additional multi-task learning.
#2 Performance MRQA 2019 Shared Task and other four datasets.
8

---

Published as a conference paper at ICLR 2025
Table 5: The experimental results on six large and challenging tasks with the associated size of trainable parameters, All results are based on the T5-3B model. We use F1 for Natural Questions, HotpotQA, SearchQA, NewsQA, and ReCoRD, and accuracy for MNLI test metrics.
Method
#Para
NQ
HP
SQA
News
MNLI
ReCoRD)
Mean
LoRA
25.8M
80.6
82.6
87.1
69.5
91.3
72.8
80.7
PT
102.4K
77.5
80.8
84.5
67.7
90.7
72.9
79.0
DePT
101 4K
77.2
80.7
83.8
66,4
89.7
72.8
78.4
ADePT (ours)
101 4KK
77.7
80.9
84.7
67.8
90.9
73.0
79.2
The experimental results on six large and challenging tasks with the associated size of trainable parameters. All results based the T5-3B model. We EM (Exact Match) for Natural Questions, HotpotQA, SearchQA, NewsQA, and ReCoRD as test metrics.
Table 6:
Method
#Para
NQ
SQA
News
ReCoRD
Mean
LoRA
25.8M
69.7
67.6
82.5
55.1
59.2
66.8
PT
102.4K
65.4
65.4
79.2
51.6
59.2
64.2
DePT
101.4K
65.0
65.4
78.3
48.9
59.1
63.3
ADePT (ours)
101.4K
65.8
65.5
79.4
51.5
59.3
64.3
Table 7: Performance comparison on MBPP benchmark. CodeGen-350M and Llama3-8B models.
We report average pass@1 scores on
Code Generation
Model
Method
#Para
MBPP
LoRA
1.3M
20.32
PT
10.2K
16.12
DePT
10.4K
16.83
CodeGen-350M
ADePT(ours)
10.2K
17.86
LoRA
9.4M
49.08
PT
41.0K
18.27
DePT
42.7K
42.50
Llama3-8B
ADePT (ours)
41.0k
43.22
Table presents the performance of different PEFT methods in the MRQA dataset and four other tasks. Despite having fewer parameters (76.1K) and faster inference (shorter soft prompt), ADePT shows a significant improvement of 6.1% on MRQA and 19.8% on the four other datasets over the vanilla PT. Also, ADePT surpasses the variants PT using additional transfer learning, including SPoT, ATTEMPT and MPT on MRQA and the other four tasks. Furthermore, ADePT can consis- tently outperform DePT on MRQA and achieve comparable performance compared to DePT on the other four tasks. Compared Adapter, ADePT can achieve comparable performance when only using 4.0% trainable parameters.
4.3 RESULTS BASED ON T5-3B MODEL
In this section, we evaluate our proposed ADePT the T5-3B model on six large and challenging tasks, including MNLI from the GLUE benchmark, ReCoRD from the SuperGLUE benchmark, and the MRQA 2019 Shared Task. Tables 5 and 6 present the experimental results of PT, DePT, and ADePT on six large and challenging tasks. DePT underperforms the vanilla PT across all tasks. embedding offsets of DePT are harmful to the T5-3B model; (2) DePT is sensitive to hyperparameters, and the hyperparameters selected based on GLUE and SuperGLUE benchmarks are not conducive to the optimization of DePT. Both of these reasons indicate that the token embedding offsets based on position and the sharing of token em- bedding offsets among multiple tokens cause sub-optimization of PLMs, especially in billion-scale PLMs. We can observe that ADePT almost achieves the optimal results across all tasks, indicat- ing that the use of the forwwrd neural networks to learn adaptive embedding offset tailored for each token can still map the input embedding into better embedding space on billion-scale PLMs. Although our method does not perform well as the LoRA on the T5-3B model, method is
There may be two reasons for this: (1) the position-

---

Published as a conference paper at ICLR 2025
Table 8: Test results of longer soft prompt lengths using the T5-base model on the GLUE benchmark.
Method
#Para
Average Glue Performance
Inference samples per second (SST2)
(m=200)
153.6K
85.2
$7.4
DePT (m=120, 60)
153.6K
86.0
77.2
ADePT (m=120, r=39) (ours)
152.9K
86.5
72.7
the best among PT-style methods, and compared to LoRA, it can use significantly fewer parameters while flexibly switching parameters adapt different downstream tasks.
4.4 RESULTS BASED ON DECODER-ONLY PLMs
We evaluate our proposed ADePT on decoder-only PLMs (i.e., CodeGen-350M model (Nijkamp
al. 2023) and Llama3-8B model (Dubey al. 2024) through instruction tuning (Ouyang al., 2022 We use MBPP benchmark, which is a Python program generation task (Austin et al., 2021] Following Jain et al. (024), we use a 50-50 split for training and test. We report average pass&l scores to evaluate the performance, best among PT-style methods, demonstrating its effectiveness. In the Llama3-8B model, the vanilla PT performs much worse than DePT and ADePT. This shows that the inability of PT to change the relative attention patterns limits its adaptation ability. whereas DePT and ADePT perform better because they change the relative attention patterns. Also, in CodeGen-3500 and Llama3-8B, the use of adaptive token embedding offsets helps ADePT perform better than DePT. Although ADePT performs slightly worse than LoRA on the MBPP benchmark. ADePT requires far fewer parameters than LoRA. More importantly, LoRA needs to merge weights, which typically limits it to only a single downstream task. contrast, ADePT can flexibly switch between tasks and adapt multiple downstream tasks simultaneously, which is unique advantage of PT--tyle methods.
shown in Table 7
We can observe that ADePT performs
4.5 FURTHER ANALYSIS
Performance using longer soft prompt. Table compares the performance of vanilla PT, DePT, and ADePT under a comparable number of trainable parameters (corresponding to vanilla PT with length of 200). The results show that ADePT outperforms both vanilla PT and DePT, demon- strating the effectiveness of its adaptive embedding offsets. In terms of inference speed, ADePT achieves speeds that exceed those of PT and match those of DePT. Although the additional shal- low token-shares feed-forwart neural network introduces some latency, the impact is minimal. It worth noting that ADePT requires computing embedding offsets for each token in real-time for every model input, whereas DePT relies fixed token embedding offsets. This real-time computa- tion may introduce additional latency; however, if the embedding offsets obtained from the ADePT method precomputed and added to the corresponding tokens in advance, this latency can be eliminated. Furthermore, since the addition operation E + AB in DePT cannot be avoided, the theoretical upper-speed limit of ADePT is expected to be faster than that of DePT, highlighting its potential to achieve both superior performance and faster inference speeds.
CONCLUSION
We propose a new parameter-efficient fine-tuning method, i.e., Adaptive Decomposed Prompt Tun- ing (ADePT), which consists of a short soft prompt and shallow token-shared forward neural network. The feed-forward neural network can learn unique offset for each input token and map the input token embeddings into a better embedding space in a position-independenn manner. Ex- tensive experiments demonstrate that ADePT outperforms leading PEFT methods, including full fine-tuning, in certain scenarios. Further analysis demonstrates that ADePT exhibits a faster infer- ence speed compared to the vanilla PT with a comparable number of parameters, while simultane- ously achieving superior performance. We provide a thcoretical analysis demonstrating that ADePT exhibits greater expressive power compared to PT and DePT.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDDMEENTS
This research was supported by National Natural Science Foundation of China (No.62476277), National Key Research and Development Program of China 00.2224YYFE00322000 CCF- ALIMAMA TECH Kangaroo Fund (NO.CCF-ALIMAMA OF 2024008), and Huawei-Renmin Uni- versity joint program on Information Retrieval. We also acknowledge the support provided by the fund for building worldclass universities (disciplines) of Renmin University of China and by the funds from Beijing Key Laboratory of Big Data Management and Analysis Methods, Gaoling School of Artificial Intelligence, Renmin University of China, from Engineering Research Center of Next- Generation Intelligent Search and Recommendation, Ministry of Education, from Intelligent Social Governncce Interdisciplinary Platform, Major Innovation & Planning Interdisciplinary Platform for the "DoubleFirst Class" Initiative, Renmin University of China, from Public Policy and Decision- making Research Lab of Renmin University of China, and from Public Computing Cloud, Renmin University of China.
REFERENCES
Akari Asai, Mohammadreza Salchi, Matthew Peters, and Hannanch Hajishirzi. ATTEMPT: Parameter-efficient multi-task tuning via attentional mixtures of soft prompts. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 6655-6673 Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. URL /tttp:://aclantholooy...
2022 emnlp main. 446
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, al. Program synthesis with large language models. arXiv preprint arXiv:2 108.0733,, 2021.
Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. BitFit: Simple parameter-efficient fine- tuning for transformer-based masked language-models. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 1-9, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. 10.18663/w1/2222.acllle acl20022..cllhoooo 1.
wrtp:/gy..rgg/20/0//// acl short.1
Daniel Cer, Mona Diab, Eneko Agirre, lñigo Lopez-Gazpio, and Lucia Specia. SemEval-2011 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 1thh International Workshop on Semantic Evaluation (SemEval-2007)) pp. 1-14, Vancouver, Canada, August 2017. Association for Computational Linguisties. doi: 10.
URL https: / ac antho org./llg...g. 2000
Christopher Clark, Kenton Lee, Ming-Wer Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 2924-2936, Minneapolis, Minnesota, June 2019, Association for Computational Linguistics. doi: 10.18653/ 11/N19-1300. URL htttps:///caatttllllooott// 9tw.1/11o...r//111130000
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers solve math word problems, 2021. URL https://arxiv.
org/ Bkg/abs/2110: 14168
Marie-Catherine De Marneffe, Mandy Simons, and Judith Tonhauser. The commitmentbank: In- vestigating projection in naturally occurring discourse. In proceedings of Sinn und Bedeutung, volume 23, pp. 107-124, 2019, URL tttppss//sematticsscccooo T332GT2M/Mrrneffee
William B. Dolan and Chris Brockett. Automatically constructing corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005), 2005. URL https:://cclanthg http://cccoaattoooooooooo/////-/ ogg.

---

Published as a conference paper at ICLR 2025
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, et al. The Ilama 3 herd of models. arXiv preprint arXiv:240.221783, 2024.
Matthew Dunn, Levent Sagun, Mike Higgins, V Ugur Guney, Volkan Cirik, and Kyunghyun Cho. Searchqa: A new q&a dataset augmented with context from search engine. arXiv preprint NoXiv:/704.05177 2017. URLhttps: /ttp:s///xxvvv..a wrw. 1704 05179
Adam Fisch, Alon Talmor, Robin Jia, Minjoon Seo, Eunsol Choi, and Danqi Chen. MRQA 2019 shared task: Evaluating generalization in reading comprehension. In Proceedings of the 2nd Workshop on Machine Reading for Question Answering, pp. 1-13, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: (0.18653/v1/D194
URL https:
santhology..rge. 5-8001
Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. The third PASCAL recognizing textual entailment challenge. In Proceedings of the ACL-PASCAL Workshop on Textual Entail- ment and Paraphrasing, pp. 1-9, Prague, June 2007. Association for Computational Linguistics. org/ 700//07-1000 1401
URL
Demi Guo, Alexander Rush, and Yoon Kim. Parameter-efficient transfer learning with diff prun- ing. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguis- tics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 4896, Online, August 2021. Association for Computational Linguis- URL
tics. doi:
org/2021.
acl acl-long.
Yun He, Steven Zheng, Yi Tay, Jai Gupta, Yu Du, Vamsi Aribandi, The Zhao, YaGuang Li, Zhao Chen, Donald Metzler, et al. Hyperprompt: Prompt-based task-conditioning of trans- formers. In International Conference on Machine Learning, pp. 8678-8690, 2022. URL
mlr wttp:mmwww...w///.m.em
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learn- ing for nlp. In International Conference on Machine Learning, pp. 2790-2799, 2019. URL http: / pttpp//eceeng....o//
www.eeiiiinmmm......m.m.m www. /veeess/vv,, htthhhulsbby 9a/ htw../sbyl///aholleyyyyyyy http:/hollsbyyll. .pdf
Edward Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang. Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In International Confer- ence on Learning Representations, 2021. URL httpp::/ net/ / forum?id= nžeVKeeFYrs.
Hamish Ivison and Matthew Peters. Hyperdecoders: Instance-specific decoders for multi-task NLP. In Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 1715-1730, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguisties. URL
/attps://aclanthin wwwaaannnnoooooooooooooooooooooe htt.:2222....niinn.eeee Findings-emn 124
Abhinay Jain, Swarat Chaudhuri, Thomas Reps, and Chris Jermaine. Prompt tuning strikes back: Customizing foundation models with low-rank prompt adaptation. In The Thirty-eighth Anmual Conference on Neural Information Processing Systems, 2024. Utthhttss:// //tpn:/e///peereiee... net/ Sitrrummid-Syyhhiii1 1vCv VVV
Rabeeh Karimi Mahabadi, Sebastian Ruder, Mostafa Dehghani, and James Henderson. Parameter- efficient multi-task fine-tuning for transformers via shared hypernetorrks. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 565-576, On- line. August 2021. Association for Computational Linguistics. doi: 10. 20.18653/w1/2021..o acc.ooo.................. 47. URL htll:ttss///caaathoooooooo wtw.://yy..o.///2..aall---------
Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. Looking be- yond the surface: A challenee set for reading comprehension over multiple sentences. In Proceed- ings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long Papers), pp. 252-2022222 2622, New Orleans,
12

---

Published as a conference paper at ICLR 2025
Louisiana, June 2018. Association for Computational Linguisties. doi: 10.18653/vI/N18. 1023.
URL
Tushar Khot, Ashish Sabharwal, and Peter Clark. Scitail: A textual entailment dataset from sci- ence question answering. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1), Apr. 2018. doi: 10. 10.1609/aaii.v32/ 12022. URL httl.httss://eo //ojs:::////.//// aaai www.aaaaa...aaaa.ee .org/nndex. phpp
Dawid Jan Kopiczko, Tijmen Blankevoort, and Yuki M Asano. VeRA: Vector-based random matrix adaptation. In The Tweifth International Conference on Learning Representations, 2024. URL
https://openrews
net wwrumoorriioooooo -iw.eaiddddNNNEN Ni
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming- Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le. and Slav Petrov. Natural questions: A benchmark for question answering research. Transactions of the URL
Association for Computational Linguistics, 7:452-466, 2019, doi: https::/aclanthoo.oo wwwp:/hwwwooooooooooooooo w01p://-or/1001100001100 109-1
Brian Lester, Rami A1-Rfou, and Noah Constant. The power of scale for parameter-efffiientt prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Lan- guage Processing, pp. 3045-3059, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. 22023////22...mlpmmaiil httpps:/aaee antho ogy. wttp///gl1100 emnlp-ma ma n. 243
URL
Hector Levesque, Ernest Davis, and Leora Morgenstern. The winograd schema challenge. In Thir- feenth International Conference the Principles Knowledge Representation and Reasoning. 2012. URL wwt.t:tp/////m
// //cdn
/cc/////c///////// 4492 4492 21843-188--1 1-BB. pBf
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the I Itth International Joint Conference on Natural Language Processing (Volume I: Long Papers), pp. 4582 4597, Online, August 2021. Association for Computational Linguistics. doi: 18.58/33/ v1/202 11/2021.acl-oo laal--looog......
201- 353
Haokun Liu, Derek Tam, Mohammed Mugeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin A Raffel. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Ad- vances in Neural Information Processing Systems, volume 35, pp. 1950-1965. Curran Associates, Inc., 2022. URL /ttt://s///poooeenggg..nnuueu
ct/ppc/:ppppppeeee Files/paper)
Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. Gpt un- derstands, too. arXiv:21033 1010031..3..... 10385, 2021.
10385
Fang Ma. Chen Zhang, Lei Ren, Jingang Wang, Qifan Wang, Wei Wu, Xiaojun Quan, and Dawei Song. XPrompt: Exploring the extreme of prompt tuming. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 10933-1104,, Abu Dhabi, United Arab Emirates, December 2022. 16653//1/2022.emmlpppo
Association for Computational Linguistics. doi: wrw.//22g.22002 eww.022.emmnmm..mmmm wawnnpmmme in. 758
URL
Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder. Compacter: Efficient low- rank hypercomplex adapter layers. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL https:
//openzeview. wwwpe//ooooo.ddoeeeeee.eeeeeeeee
Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. Codegen: An open large language model for code with multi-turn program synthesis. In The Eleventh International Conference on Learning Representations, 2023. URL htttps://openreet
net http://wwww....eeoooooooooeeaaaa
13

---

Published as a conference paper at ICLR 2025
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Gray, John Schulman, Jacob Hilton, Fraser Kel- ton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neu- ral Information Processing Systems, 2022. URL /eLthtts:://// http::///pppeeeee..eooo net forott/foommiao TG8KACxEON
/
Aleksandar Petrov, Philip Torr, and Adel Bibi. When do prompting and prefix-tuning work? theory of capabilities and limitations. In The Twelfth International Conference on Learning Representa- fiony, 2024. URL htttt:://ppeneewe/ee net/ wwrmm/oormmmmmo id=::m:ddddwwwooo
Mohammad Taher Pilehvar and Jose Camacho-Collane
WiC: the word-in-context dataset for eval- uating context-sensitive meaning representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume 1 (Long and Short Papers). Association for Computational Linguistics, owg/N10...//1100 9-1288
June 2019. URL
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with unified text- to-text transformer. J. Mach. Learn. Res., 21(1), jan 2020. ISSN 1532-4435. URL https:
7/dl. aall.acm.oo org./oi/////ooooooooooo http://di//bbb////a
abs /tbs/11... 5555 3455716 3455856
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference Empirical Methods in Natural Language Processing, pp. 2383-2392 Austin, Texas, November 2016. Association for Computational Linguistics. doi: 10. 10.18653/v
URL attps://aclanthenn
erg/D16-1280
Anastasiia Razdaibiedina, Yuning Mao, Madian Khabsa, Mike Lewis, Rui Hou, Jimmy Ba, and Amjad Almahairi. Residual prompt tuning: improving prompt tuning with residual reparame- terization. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguistics: ACL 2023, pp. 6740-6757, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10. 101118/11112223 10dp:5//111112..mmnnnnnnl-11///0 URL ltttps://e httpss///claathhloooyy. 2023.
wtt.:2....fnndii. windininnnnannee acl 421
Andreas Rücklé, Gregor Geigle, Max Glockner, Tilman Beck, Jonas Pfeiffer, Nils Reimers, and Iryna Gurevych. AropterDrop: On the efficiency of adapters in transformers. In Proceedings of the 202/ Conference Empirical Methods in Natural Language Processing. Association for Computational Linguistics, November 2021. URL httph: httpa////cccaaatooooooorr/////// emm ma in. 626
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An ad- versarial winograd schema challenge at scale. Commun. ACM. 64(9):99 106, aug 2021. ISSN 0001-0782. doi: 10.1 10.1444/347481. URL https:://oi. /ttp://....001111111333333733333 11155/4444143
Zhengxiang Shi and Aldo Lipani. DePT: Decomposed prompt tuning for parameter-efficient fine- tuning. In The Twelfth International Conference Learning Representations, 2024. URL
https:// //tp::////ppenevieww net (ttee//ooummndddeeeeessse
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, sentiment
and Christopher Potts. Recursive deep models for semantic compositionality treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 1631-1642, Seattle, Washington, USA, October 2013. Association for Computa- tional Linguistics. URL httpsts:: htttpss:/aclaatthlllog...ooo wtwpo/gg.....o..ooo.//o/ D
Yi-Lin Sung, Jaemin Cho, and Mohit Bansal. LST: Ladder side-tuning for parameter and memory efficient transfer learning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL https://openrewe wtt.:eeeoeoooffitattee.m wtn.niidddiinianieemmm sPnna
14

---

Published as a conference paper at ICLR 2025
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foun- dation and fine-tuned chat models. arXiv preprint arXiv:2307.09228, 2023. URL https:
//arxiv.org//a abs/ 09788
Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. NewsQA: A machine comprehension dataset. In Proceedings of the 2nd Workshop on Representation Learning for NLP. pp. 191-200, Vancouver, Canada, Au- gust 2017. Association for Computational Linguistics. doi: 10. 18653//1//17-26223. URL ht. fittps://ack antho
Tu Vu, Brian Lester, Noah Constant, Rami Al-Rfou', and Daniel Cer. SPoT: Better frozen model adaptation through soft prompt transfer. In Proceedings of the 60th Annual Meeting of the Asso- ciation for Computational Linguistics (Volume 1: Long Papers), pp. 5039-5059 Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. (0.188655///2222.2a44mo.......
URL
https://acce antho www.thhlogy...ooooom ooy.. orggyoorg/2222. long................o...oo. 346
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceed- ings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pp. 353-355, Brussels, Belgium, November 2018. Association for Computational Lin- guistics. doi: 10.18653/v1/W 10663//11W18-5446.
URL. [t..tttpss.. http:htts:::/e/t //acl antho logy. org. W1 844466
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. Superglue: A stickier benchmark for pereral-purpose language understanding systems. In arxiv, 2019. htllhtttp://rxxww.orr///t
905. 00537
Zhen Wang. Rameswar Panda, Lconid Karlinsky, Rogerio Feris, Huan Sun, and Yoon Kim. Mul- titask prompt tuning enables parameter-efficient transfer learning. In The Eleventh International Conference on Learning Representations, 2023. URL httpttts:://peneevieeee net/ forum?
id=Nk2pDtuhTq
Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625-641, 2019. doi: 10.1162/
tacl URL http://s//accaahhoooooooo - wwtg/19-104//
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sen- tence understanding through inference. In Proceedings of the 2018 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long Papers), pp. 1112-1122, New Orleans, Louisiana, June 2018 Association for Computational Linguistics. URL https: / /ataa/aaannttoooooo/o org/ N1 8 1101
Yao Xiao, Lu Xu, Jiaxi Li, Wei Lu, and Xiaoli Li. Decomposed prompt tuning via low-rank reparam- eterization. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp.
Singapore, December 2023. Association for Computational Linguistics. doi: 10. 10.18633/112200 1/2023. wwnn:iiinnnnnennnn.e.... https: http://cclatttt ooy.. org/ 2023. f.r.f...nnii. ndings-emn 1p. 890
URL
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio. William Cohen, Ruslan Salakhutdi- nov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi- hop question answering. In Proceedings of the 2018 Conference on EMNLP, Brussels, Bel- gium, October-November 2018. Association for Computational Linguistics. URL. https: //aclanthology.co .rw.:yy.orrr// org.orrgg./..... 111p/1181111999
Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme. Record: Bridging the gap between human and machine commonsense reading comprehension. 1810 12885
arXiv preprint arXiv: 1810.12885 2018. URL
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In Proceedings of the 28th International Conference on Neural Infor- mation Processing Systems Volume 1, NIPS' 15, pp. 649-657, Cambridge, MA, USA, 2015. MIT Press. http:p/s//pppccce..gg......rreee wttpp//wwwws....pp//////////.e..
15

---

Published as a conference paper at ICLR 2025
Yuan Zhang, Jason Baldridge, and Luheng He. PAWS: Paraphrase adversaries from word scram- bling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume I (Long and Short Pa- pers), pp. -998-,00, Minneapolis, Minnesota, June 2019. Association for Computational Lin- guisties. doi: 10. 10.18653/1//199-1133
htthttss:: a antho anthollogy.orr//N19-111
16

---

Published as a conference paper at ICLR 2025
APPENDIX OVERVIEW
The appendix is structured as follows:
Appendix provides a theoretical analysis towards DePT. We combine the theoretical analysis from Section 3.4 and the Appendix A to theoretically explain why ADePT performs better than
DePT.
Appendix
B
provides additional experiments to further analyze proposed ADePT.
Appendix
provides a more detailed implementation of our experiments.
Appendix
D
provides detailed description of datasets.
Appendix
B
provides detailed hyperparameters of our experiments.
A
How DECOMPOSED PROMPT TUNING AFFECT THE FIRST MULTI-HEAD SELF-ATTENTION LAYER?
In this section, we analyze how the DePT affects the first multi-head self-attention layer.
Given the soft prompt P [pt, P......... .P] E Rid and the offset embeddings AE AB E Raxd the output of a query vector æ; passing through the single-head self-attention formulated
as,
DePT_
Attention ((e: + Ae;) W concat P, E + AE concat P, E + AE
Aips W + 1 Σ
((ee+ + Ae,) ) W ") AEW?
Ask
Softmax
k=1
A,.)Softmar (((e; De,) W (E+ EW?
+ (1
exp ((ee ((e)+ Wiy (p)
with
A
==
Σ exp ((e;
+ W (paW)) ))
k=1
exp ((e;+ W ((e; + de;) wX)') )
Ø=1
(8)
where Au is the attention score gives to the prefix vector Ps. for given query vector e,, We can observe that, in the first transformer layer, DePT can change the relative attention patterns. However, compared to ADePT, the attention patterns change along the change of position. Also, DePT cannot add bias dependent on model input in the first transformer layer.
17

---

Published as a conference paper at ICLR 2025
B ADDITIONAL EXPERIMENTS
B. ABLATION STUDY
Table 9: The comparison of training time based on T5-3B model. "h" means hours.
Method
#Para
NQ
HP
LoRA
25.8M
9.73 h
9.63 h
153.6K
12,60
12.53
DePT
153.6K
12.60
12.53
ADePT (ours)
152.9K
12.67
12.58
Table 10: the experimental results of fine-tuning both the prompt and the embedding matrix, as well as our proposed ADePT based on the T5-base model for RTE.
Method
#Para
RTE
Finetuning prompt and embedding matrix ADePT (ours)
24.8M
76.3
76. IK
82.0
Table 11: the experimental results of ADePT/DePT when used without the token offsets but only learned soft prompt. The results are based on the T5-base model and the RTE task.
Method
With token offsets
Without token offsets
DePT
79.1
78.4
ADePT (ours)
82.0
58.3
Table 12: The results ADePT with different bottleneck sizes using the T5-base model on the RTE.
The size of the Bottleneck
5
10
20
30
ADePT (ours)
78.4
82.0
82.0
79.9
Table 13: The results of ADePT with soft prompt lengths using the T5-bass model on the RTE.
The length of soft prompt 20
40
60
80
ADcPT (ours)
72.7
77.7
79.1
82.0
80.6
79.1
Table 14: The comparison of ADePT with only feed-forwadd neural network and the original ADePT based on T5-base model for the RTE.
Method
#Para
RTE
Only Feed-forward Neural Network ADePT (ours)
76.1K
73.4
76.
82.0
Table 9 indicates that PT, DePT, and ADePT need similar training time. The PT-family method needs longer training time than LoRA due to the longer input sequence.
Table 10 shows the experimental results of fine-tuning both the prompt and the embedding matrix, as well as our proposed ADePT. We use the same length soft prompt, and we search learning rates for prompt matrix from {3e-1, 4e-1, 5e-1} and embedding matrix from {lc-3, Ie-4, 1e-5). We observe that fine-tuning both the prompt and the embedding matrix underperforms our proposed ADePT. Additionally, this method requires fine-tuning large number of parameters, which may lead to overfitting.
Table II show the experimental results of ADePT/DePT when used without the token offsets but only learned soft prompt. We can observe that the token offsets of ADePT play a much more important role than DePT.
18

---

Published as a conference paper at ICLR 2025
presents how the size of the bottleneck affects the performance of the RTE task. We can observe that an overly small or overly large bottleneck size will cause decline in performance. When it is too small, it can lead to under-fittin;; when it is too large, it can lead over-fitting.
Table 12
Table 13
presents how the length of the prompt affects the performance on the RTE task. We can observe that performance drops significantly when the prompt length is less than 40. Performance optimal when the prompt length is 50 or 60, but it decreases when the prompt length is too large, possibly due overfitting.
Table 14 shows that the performance on the RTE task when all parameters are relocated to the learnable projection (prompt length 0, bottleneck size 49, trainable parameters 76.1k) lk) is 73.4,
indicating the soft prompt is necessary.
B.2 FEW-SHOT LEARNING
Table 15: Few-shot learning results, obtained from three runs, with k: (4, 16, 32} training samples on the BooQ, CB and SciTail datasets. Baseline results are directly quoted from Shi & Lipani (0024)
Task
I-shot
Full Finetuning
(LA))
ATP
MPT
DePT 76.8K
ADePT (ours)
62.7s
To.
BoolQ
16
56.5
51.4
61.9
50.5
50.2
62.0
50.0
63.3
66.94
69.91.
32
58.4
54.5
61.7
61.2
58.3
67.2
65.3
68.9
67.2s.
70.0
4
57.7
51.1
53.5
71.4
60.7
65.5
67.9
73.6
75.0s.1
52.1a4
CB
16
77.0
74.8
63.5
64.3
76.3
71.4
71.4
78.6
78 78.6ua
36.723
32
82.1g
39.5g1
78.16
76.9.
SciTiil
16
80.0
83.2
60.8
71.9
86.5
74.4
79.5
78.5
82.12a
32
81.9
85.0
60.2
71.9
85.8
80.4
80.2
86.3
85.4g1
82.62a
Table 15 shows the few-shot learning results with k= f4, 16, 32) training samples on BoolQ, CB and Tail datasets. We pre-train both the soft prompt and the feed-forward neural network on source tasks and select the best checkpoint to initialize the parameters. We can observe that ADePT performs best on the BoolQ dataset, performs well on the SciTail dataset, and performs the worst on the CB dataset. This might indicate that ADePT unsuitable for few-shot learning, which reasonable since learning the embedding offsets for each token using feed-forward neural network requires considerable training samples.
B.3 UNSUCCESSFUL ATTEMPTS
We attempted perform instruction tuning on a larger scale of the GSM8K dataset (Cobbe al. 2021) using Llama3 8B, with 7,473 examples in the training set and 1,319 examples in the test set. The goal of this instruction tuning was fine-tune the model to better solve mathematical reasoning problems in the GSM8K dataset. However, unfortunately, we were not successful in our training attempts, including both ADePT and DePT. This might be because the simple low- rank matrix multiplication and shallow neural networks are unable to fit the input token embedding offsets required by large datasets when working with large models that have a very large vocabulary. Addressing this issue will be a direction for our future research.
19

---

Published as a conference paper at ICLR 2025
C ADDITIONAL IMPLEMENTATION DETAILS
We implement our experiments by using Pytorch 2 Huggingface Transformers and Huggingface PEFT We evaluate our proposed ADePT in four PLMs. i.e. T5-base mode T5-3B mode CodeGen-350M and Llama3-8B Following Asai et al. 2022); Wang et al. 2023 Shi & Lipani
2024), we train the T5 model using the original checkpoint rather than the LM-adapted 1.1 version Lester et al., (022). For the T5-3B model, due to the limitations of computational resources, we select the several most convincing datasets to evaluate our proposed ADePT. We think that convine- ing datasets should have large training dataset and large test dataset, and be challenging for the T5-base model. We use the criteria of more than 70,000 training samples, an aceuracy/Fi score of less than 90% on the T5-base model, and more than 4,000 test samples to select the datasets to test our proposed ADePT on the T5-3B model. Shi & Lipani (2024) found that training PT for additional steps typically leads performance improvements, and we follow this setting to train proposed ADePT. We measure the latency of ADePT by running a feed-forward neural network for each token in real time.
For the small datasets (< 70, 000 training samples), following Shi & Lipani 2024), we search the learning rates for the soft prompt from (3e 1, 4e 1, 5e }} and for the feed-forward neural network from [1e-4, 1e 5}. We also search for the prompt length from {20, 40, 60, 80) with cor- responding bottleneck sizes of {39, 29, 19, 9} to ensure the number of trainable parameters remains below 76.8K. For the large datasets (> 70, 000 training samples), we set the prompt length as 60, the bottleneck size as 19, the prompt learning rate as 3e 1, and the forw-fowward neural network learning rate as le 4. For decoder-only PLMs, following Jain et al.] (2024), we use 10 virtual tokens for PT, 7 virtual tokens and rank r. = for DePT, virtual tokens and bottleneck size r for ADePT, and rank 16 for LoRA. For the MBPP benchmark, following Jaiin et al. (2024), we use
learning rates of 1e- 3 for the prompting-style tuning method and 1e 4 for LoRA.
Following Shi & Lipani (024), for the T5-base model and the small datasets, we train the model for 30,000 steps; for the T5-base model and the large datasets, we train the model for 300,000 steps. For the T5-3B model, we train the model for 30,000 steps. In each trial of the t5-base model and the T5-3B model, we evaluate the performance every 1,000 steps and select the best checkpoint based on the optimal performance on the evaluation set. For the MBPP benchmarks, following Jain et al. 2024), we train the model for 10 epochs. We train the model with a batch size of 32, except for the MBPP benchmark, where we batch size of 4. We typically use a maximum sequence length of 256, except for SuperGLUE-MultiRe where the maximum sequence length is 348, and MRQA, where is 512.
For the few-shot learning, following the prior works (Asa et al. 2022 Wang et al. 2023 Shi & Lipani 2024), we first pre-train five source tasks (i.e., MNLI, QQP, SST-2, SQUAD, and ReCoRD), and then select the best checkpoint to use as the initialization for shot training.
https://pptoochhoorg/
https: htferr//sthh..ooo/hngggnnteeatts
httpss//oogpinfeee.cc/oooo--17/3 mono
https//mogginffeeeccc//ooo/aal/d 3-
20

---

Published as a conference paper at ICLR 2025
D DATASETS
The datasets assessed in this study are described as follows. The term "Source Length" refers the average length of source sentences in the training set, whereas "Target Length" indicates the average length of target sentences the training set. The term "Train" refers to the number of samples in the training set, whereas "Valid" and "Test" indicate the number of samples in the validation set and test set, respectively. The term "Type" refers to the task type of the dataset.
Table 16:
GLUE Benchmark
Dataset
Source Length
Target Length
Train
Valid
Test
Type
MNLI
31.8
1.0
392,702
9,832
9,815
NLI
QQP QNLI
24.1
1.0
362,846
1,000
40,431
Paraphrase
38.4
1.0
103,743
1,000
5,463
NLI
SST-2
10.4
1.0
66,349
1,000
872
Sentiment
STS-B
21.9
1.0
5,749
750
750
Sent. Similarity
MRPC
45,9
1.0
3,668
204
204
Paraphrase
RTE
54.4
1.0
2,490
138
139
NLI
CoLA
8.7
1.0
8,551
521
522
Acceptability
SuperGI.UE Benchmark
Dataset
Source
Target
Train
Valid
Test
Type
MultiRC
286.1
1.0
27,243
2.424
2,424
Question Answering
BoolQ
108.3
1.0
9,427
1,635
1,635
Question Answering
WiC
18.4
1.0
5,428
319
319
Word Sense Disambiguation
WSC
28.1
1.0
554
52
52
Common Sense Reasoning
CB
1.0
250
28
ReCoRD
210.7
137.484
1,370
15,176
Common Sense Reasoning
MRQA 2019 Shared Task
Dataset
Source
Target
Train
Valid
Test
Type
NaturalQuestions
242.7
4.5
103,071
1,000
12836
Question Answering
HotpotQA
225.7
71,928
1,000
5,901
Question Answering
SearchQA
942.8
2.0
116,384
1,000
16,980
Question Answering
NewsQA
615.5
73,160
1,000
4,212
Question Answering
Other Datasets
Dataset
Source
Target
Train
Valid
Test
Type
WinoGrande
23.8
1.0
39,398
1,000
1,267
Common Sense Reasoning
YelpPolarity
134.0
1.0
100,000
1,000
38,000
Sentiment
SciTail
30.8
1.0
23,596
652
652
NLI
PAWS
44.7
1.0
4,9401
8,000
8,000
Sent. Similarity
21

---

Published as a conference paper at ICLR 2025
E HYPERPARAMETERS
Table 17: Hyperparameters of small datasets for ADePT on T5-base model.
Hyperparameter
Assignment
number of steps
30,000 steps (evaluate every 1,000 steps)
batch size
32
maximum learning rate (01)
3e-1, 4e-1, 5e-1
maximum learning rate (ax2)
lc-4, 1c-5
length the soft prompt (m)
20, 40, 60,
maximum sequence length
256
learning rate optimizer
Adam
Adam epsilon
lc-6
Adam beta weights
0.9, 0.98
learning rate scheduler
Warmup linear
Weight decay
0.01
Warmup steps
500
Table 18: Hyperparameters of large datasets for ADePT on T5-base model.
Hyperparameter
Assignment
number of steps
300,000 steps (evaluate every 1,000 steps)
batch size
16
gradient accumulation steps
2
maximum learning rate (ax))
maximum learning rate (ax))
le-4
length of the soft prompt (m)
60
maximum sequence length
512
learning rate optimizer
AdamW
Adam epsilon
le-6
Adam beta weights
0.9,
learning rate scheduler
Warmup lincar
Weight decay
0.01
Warmup steps
500
22

---

Published as a conference paper at ICLR 2025
Table 19: Hyperparameters for ADePT on T5-3B model.
Hyperparameter
Assignment
number of steps
30,000 steps (evaluate every 1,000 steps)
batch size
16
gradient accumulation steps
2
maximum learning rate (a11)
3e-1
maximum learning rate (a:2)
lc-4
length the soft prompt (m)
60
maximum sequence length
512
learning rate optimizer
AdamW
Adam epsilon
le-6
Adam beta weights
0.9, 0.98
learning rate scheduler
Warmup lincar
Weight decay
0.01
Warmup steps
500
23


Reviews:
[{"review_id": "6cjW5woaiR", "reviewer": "Reviewer_3RMb", "summary": "The paper finds that the fixed token embedding offset in DePT limits its generalization capability across different model inputs, leading to suboptimal performance. To address these issues, the authors introduce Adaptive Decomposed Prompt Tuning (ADePT), which consists of a short soft prompt and a shallow token-shared feedforward neural network. ADePT uses the token-shared feedforward neural network to learn embedding offsets for each token. This enables ADePT to achieve superior adaptability without requiring more inference time or additional trainable parameters than standard PT and its variants.\n\nOverall, this paper provides a thorough and clear analysis. It offers a simple yet effective solution to the offset issues present in standard PT. However, I still have the following questions: \n1. How robust is this method, and does it possess general applicability? \n2. Is this method still effective for long text problems? \n3. Is ADePT effective for other large parameter language models, such as the LLaMa series? I hope the authors can provide answers to these questions.", "questions": "1. How robust is this method, does it have general applicability, and will incorporating AdePT affect the model's generalizability?\n2. Is this method still effective for long text problems? \n3. Is ADePT effective for other large parameter language models, such as the LLaMa series? I hope the authors can provide answers to these questions.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper introduces the Adaptive Decomposition Prompt Tuning (ADePT) method, which innovatively addresses the generalization limitations caused by fixed token embedding offsets in traditional DePT methods. ADePT achieves excellent adaptability without increasing inference time or requiring additional parameters. The analysis is thorough, and the method is both simple and effective, offering new directions for future research. Additionally, the experiments are conducted rigorously, and the writing is clear and concise.", "weaknesses": "I believe this method may lack generality, especially when applied to large language models and long-text tasks. The main reason for questioning this is whether feedforward neural networks possess sufficient semantic understanding capabilities. Additionally, there is room for further optimization in the figure.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yNnXqMePyK", "reviewer": "Reviewer_znHg", "summary": "The paper introduces Adaptive Decomposed Prompt Tuning (ADePT), a novel approach in parameter-efficient fine-tuning (PeFT) that significantly enhances the adaptability of pre-trained large language models (PLMs) to various downstream tasks. ADePT improves Decomposed Prompt Tuning (DePT) by addressing its limitations: DePT's fixed token embedding offsets often underperform due to their inability to dynamically adjust to different model inputs. By integrating a token-shared feed-forward neural network (FFNN), ADePT dynamically adjusts embedding offsets, tailored to each specific input token. This adaptive mechanism allows ADePT to maintain the inference speed advantages of DePT while achieving state-of-the-art (SOTA) performance in adaptation. Extensively tested across 22 natural language processing (NLP) tasks and two PLMs of differing scales, ADePT not only surpasses other PeFT methods like Adapters, LoRA, and standard Prompt Tuning (PT) but also exceeds full model fine-tuning benchmarks in certain scenarios.", "questions": "1. What are the criteria of selecting the datasets and baselines for 3B-scale models?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper provides a comprehensive overview of Parameter Efficient Finetuning (PeFT), effectively situating ADePT within the broader research landscape and highlighting its contributions.\n\n2. ADePT is intuitive and well-motivated. The arguments and experiments in Section 3.2 convincingly demonstrate the limitations of DePT being a low-rank absolute positional embedding, paving the way for ADePT, which instead uses token-wise MLP for calculating embedding offsets.\n\n3. The experiments conducted with T5-220M are thorough, incorporating all relevant benchmarks and key baseline comparisons. The results are presented with nice detail and clarity.\n\n4. At the 220M scale, ADePT shows remarkable efficacy, particularly in data-scarce scenarios such as RTE and CoLA tasks, where it not only competes but also surpasses full finetuning, showing its superior performance.", "weaknesses": "1. The robustness of the experiments with the 3B model does not match the standards set by the 220M scale evaluations. Notably, the selection of fewer benchmark tasks without clear justification, as well as the omission of significant baselines such as Adapters and LoRA, weakens the overall experimental credibility for the 3B model.\n\n2. The performance improvements of ADePT over PT for the T5-3B model is only 0.1 or 0.2 pts for each task in Table 5. This tiny margin on a selected set of tasks may suggest that DePT-style methods cannot scale to larger models.\n\n3. A key appeal of PeFT methods is their cost-efficiency in finetuning; however, the paper lacks comparative analysis of the training costs across different PeFT methods, specifically within the PT family where training speed is more comparable (PT, DePT, and ADePT). Such omission leaves a gap in understanding their real-world benefits.\n\n4. The description of DePT and ADePT in the \"Introduction\" section lacks clarity. Phrases such as \"the token embeddings of DePT violate the uniqueness of token embeddings\" are presented without sufficient context, making them hard to understand without referring to the formulas in Section 3.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SLwH9RlzLc", "reviewer": "Reviewer_cKVM", "summary": "This paper proposed to extend Prompt Tuning and DePT, by replacing the position-based updates that DePT applies to the true token embeddings with content-based updates by passing the token embedding through a small (bottlenecked, down-project + up-project) MLP and adding that to the original token embedding.\n\nThey test its effectiveness on many datasets, against many baselines, and use multiple frozen models. They find that their method performs best in many cases.", "questions": "Did you compare to fine-tuning the prompt and the embedding matrix? The would be a small loss in generality (tokens not seen during training would not get updated) but it would be a much simpler implementation. I would be curious to see how this version performed on CB too, as a possible explanation of ADePT's poor performance could be that the NN offsets don't work well on unattested words.\n\nWhen measuring the latency of different methods, where the token offsets for ADePT precomputed and folded into the embedding table or was the NN run for each token?\n\nDid you try using the LM-adapted model from Lester et al 2021 instead of the span-corrupted T5 models used here? It is also unclear if you used the original T5 models or the T5 1.1 models, IIRC, some of the datasets used were seen during pre-training of the original T5 models.\n\nAs the DePT offsets only differ between each position, it seems reasonable that they have low mean and variance. In contrast, ADePT is per-token and content based so it seems reasonable that it would have a much higher variance. I'm not sure mean and variance is the correct metric to gauge how much these offsets are actually doing (or if they are sub-optimized). Something like a norm of the offsets might make more sense? Or it could have been measured directly in an ablation where the soft prompt learned in ADePT/DePT is used without the token offsets.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "The paper introduces the other work in the space really well and does a good job contextualizing itself among that work.\n\nThe pilot experiments highlighting the weakness of DePT are a good motivation for their work.\n\nThe paper compares to a lot of different baselines, including PEFT methods beyond just prompt tuning.\n\nThe paper evaluates the method on a lot of different datasets, increasing the trust you can put into setting good results if you used it on your task.\n\nThe paper uses multiple different pre-trained models as the frozen models the PEFTs are applied to. These are also at two different scales. It is good to see the results still hold.", "weaknesses": "The weakness of DePT is outlined in the paper as its \"fixed token embedding offsets\". This point would be much clearer if it was re-framed as the DePT offsets are \"position-based\" while the ADePT offsets are \"content/token-based\". Both are \"fixed token embedding offsets\" (ADePT output is fixed once the input token is know, it isn't contextual). This framing would make a lot of their examples about the issues much clearer. For example the section about the [t1, t2] being added causing a shift if which offsets are applied where much clearer. It also makes their point about the DePT offsets not doing much because they have to handle all tokens more obvious! It also could make this example much clearer where it could cast [t1,t2] as a \"system prompt\" added after the fact that messes up the learned position embeddings from DePT (and also hightlight how DePT may not play nicely with prompt engineering which ADePT probably would).\n\nThey state that \"embedding for each token should be unique after being offset\" as a critique of DePT, but thinking of DePT as position-wise offsets it doesn't seem like a problem, especially given positional embeddings work.\n\nThe prose can be tighten up quite a bit. There are lots of parts that repeat themselves multiple times, for example the position-wise implementation of DePT is over-explained multiple times. Similarly, much of the algebraic manipulations of the parameter counts could be omitted.\n\nMany of the increases in performance are rather small, although the simplicity of this method makes that more acceptable.\n\nIt is unclear is they use the optimization stabilization of Razdaibiedina et al 2023 they mention in the introduction when using PT.\n\nMuch of the baseline performance numbers are from other works, opening the possibility of a mis-match in setting. For example, the numbers for SPoT in Table 3 are surprising as they are lower than the Prompt Tuning numbers, but they are taken from different papers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0GVfwJbYYS", "reviewer": "Reviewer_Z8Tq", "summary": "This paper is about prompt tuning, which improves on the DePT method and propose a method called ADePT. It replaces fixed token embedding offsets with a shallow feed-forward neural network that can dynamically generate offsets for the input token embeddings, thereby providing better adaptability to different input tokens.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The introduction of a token-shared feed-forward neural network as a solution is well-motivated, by effectively identifying critical limitations in the DePT method of the static token embedding offsets.\n2. The experiments are thorough, the performance are reported in many datasets, including NLU and NLG.", "weaknesses": "1. Lack of analysis of the proposed method. For example, the method includes the bottleneck hyperparameter that controls the total #params that does not exceed vanilla PT. However, the size of bottleneck as well as the corresponding prompt length on the performance is not clear. It is also interesting to show if relocated all the #params on the learnable projection, where prompt length=0.\n2. In Table 8, comparing with the improvement DePT brings to PT in both accuracy and latency, the proposed ADePT seems trade latency to accuracy. Moreover, the complexity introduced in the learnable projection hurts the few-shot performance compared with DePT.\n3. The method is only tested on two T5 language models, including the base and 3B variants. It should also be tested on the decoder models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)