Decomposed Prompt Tuning via Low-Rank Reparameterization
Authors: Yao Xiao
Lu Xu
Jiaxi Li
Wei Lu
Xiaoli Li
Venue: EMNLP 2023 Findings (2023)
Keywords: prompt tuning
parameter-efficient tuning
TL;DR: We propose a low rank version of prompt tuning.

Abstract: While prompt tuning approaches have achieved competitive performance with high efficiency, we observe that they invariably employ the same initialization process, wherein the soft prompt is either randomly initialized or derived from an existing embedding vocabulary.
In contrast to these conventional methods, this study aims to investigate an alternative way to derive soft prompt. Our empirical studies show that the soft prompt typically exhibits a low ``intrinsic rank'' characteristic.  With such observations, we propose decomposed prompt tuning, a novel approach that utilizes low-rank matrices to initialize the soft prompt. Through the low-rank reparameterization, our method significantly reduces the number of trainable parameters while maintaining effectiveness. Experimental results on the SuperGLUE benchmark in both high-resource and low-resource scenarios demonstrate the effectiveness of the proposed method.

Decomposed Prompt Tuning via Low-Rank Reparameterization
Yao Xiao and Lu Xu' and Jiaxi Li' and Wei Lu and Xiaoli Li²
'Singapore University of Technology and Design Institute for Infocomm Research, A#Star, Singapore (xiao_yao, xu_lu, [taxi_li)@mymall sutd. edu. sg luwei@sutd. edu. sg, xlli@i2r a-star edu. sg
Abstract
While prompt tuning approaches have achieved competitive performance with high efficiency, we observe that they invariably employ the same initialization process, wherein the soft prompl either randomly initialized or derived from an existing embedding vocabulary. In con- trast these conventional methods, this study aims to investigate an alternative way to derive soft prompt. Our empirical studics show that the soft prompt typically exhibits low "in- trinsic rank" characteristic. With such observa- tions, we propose decomposed prompt tuning, novel approach that utilizes low-rank matri- ces to initialize the soft prompt. Through the low-rank reparameterization, our method sig- nificantly reduces the number of trainable pa- rameters while maintaining effectiveness. Ex- perimental results the SuperGI.U bench- mark in both high-resouree and low-ress low-resource scenarios demonstrate the effectiveness the proposed method."
SuperGLUE Average Score
75
70-
65
Ours
60
T5-Small
TS-Base
TS-Large
Figure 1: Average performance over all the datasets of SuperGI with T5 models. The number of train- able parameters used are 11.2K, 102K, and 925K with T5-Large for DPT, vanilla prompt tuning (PT) (Lester et al., 2021), and Residual PT (Res PT) (Razdaibiedina et al., 2023), respectively. More details are included in Section 5.1.
1 Introduction
of these methods fine-tune only a subset of the model's parameters or additionally introduced pa- rameters while freezing the majority of parameters of a pre-trained model. These approaches require saving only the trainable parameters for different tasks, striking balance between performance and efficiency.
Pre-trained language models (Peters al., 2018; Radford et al., 2019; Devlin et al., 2019; Liu et al., 2020; Raffel et al., 2020) have achieved remarkable performance on various natural language under- standing and generation tasks. The pretrain-then finetune paradigm has been adopted as a common approach to deal with downstream tasks. How- ever, such paradigm is often considered ineffi- cient, especially in the era of large language mod- els (LLMs), as it requires tuning a large number of model parameters and saving separate model for each task.
The success of PEFT methods is also aligned with the previous findings that pre-trained models possess a low "intrinsic rank" (Li et al., 2018; Agha- janyan et al., 2020). Aghajanyan et al. (2020) em- pirically showed that employing a low-dimensional reparameterizatioo is equally effective as full- model fine-tuning. Hu et al. (2022) further demon- strated that weight updates during model training also exhibit low "intrinsic rank". By only tuning the proposed low-rank adaptation module, their ap- proach achieves high efficiency while maintaining competitive performance.
Recently, parameter-efficien fine-tuning (PEFT) approaches (Houlsby et al., 2019; mahabadi et al., 2021; Liu et al.. 2022b,a; Vu et al., 2022; Asai et al., 2022; Wang et al., 2022, 2023) have been proposed to address this challenge. The main idea
'Our code available at /VaC
com/
Another line of research work focuses on p

---

tuning (Li and Liang, 2021; Liu al.. 2021; Qin and Eisner, 2021; Lester al., 2021). Specifically, and Liang (2021) proposed prefix tuning by prepending a sequence of virtual tokens to each transformer layer, and it updates the representa- tions of these virtual tokens while keeping the pre- trained model frozen. Prompt tuning (Lester et al.. 2021) further simplified prefix tuning by updating only sequence of continuous prompt tokens in the embedding layer.
and low-resource scenarios demonstrate the effectiveness of our proposed approach.
Compared with the vanilla prompt tuning approach, our method requires significantly fewer trainable parameters while achieving strong performance (i.c., 11.2K vs. 102K with T5-Large (Raffel et al., 2020)).
Emergence of "Intrinsic Rank"
Owing to its simplicity, subsequent studies (Ma et al., 2022; Razdaibiedina et al.. 2023) contin- uously improve the vanilla prompt tuning ap- proach. Despite the substantial achievements of these prompt tuning methods, observe that they invariably employ the same initialization process, wherein the soft prompt is either randomly initial- ized or derived from the existing embedding vocab- ulary. Different from the previous approaches, this paper aims to explore an alternative approach to deriving soft prompt.
To investigate the rank of soft prompt. we design an analysis based on the vanilla prompt tuning ap- proach on the sub-tasks within the SuperGLUE benchmark. Specifically, we re-design the soft prompt, which facilitates a probe into its rank.
2.1 Prompt Tuning
We employ the vanilla prompt tuning (Lester et al., 2021) approach to conduct our analysis, as illus- trated on the left side of Figure 2. By considering the classification task a conditional generation task, prompt tuning models the probability as:
Our motivation stems from the observations that weight updates in the transformer layer during train- ing have low "intrinsic rank", as highlighted by Hu et al. (2022). This leads us to inquire whether soft prompt also have a similar low "intrinsic rank" pattern. To this end, we conduct studies examining the "intrinsic rank" of soft prompt, and we include the details of the analysis in Section 2. Based on the studics, we find that the soft prompt indeed tends to exhibit low "intrinsic rank" behavior. Armed with this insight, we propose decomposed prompt lun- ing (DPT), novel approach that employs low-rank matrices to initialize the soft prompt. Specifically, we decompose the original soft prompt into the product of two compact matrices, and we include the detailed description in Section 3. With this low-rank reparameterization, DPT significantly re- duces the number of trainable parameters while achieving strong performance. A comparison with previous approaches, depicted in Figure 1, verifies the efficacy of our method.
Pr Pro:o~(Y[PP;))
where the input is the concatenation of the prepended prompt P of length e and the original input X of length n. After the embedding layer, the representations for the prompt and the original input are Pemms E Reex and Xemb E Rxxn, respec- tively, and e is the embedding dimension of the pre-tran pre-trained model. Y denotes a sequence of output tokens, which also serves the class label for the input". 0 indicates the frozen parameters of the pre- trained model, while 0p represents the trainable parameters corresponding to the prompt P.
2.2 Low-Rank Behavior of Soft Prompt
As highlighted earlier, the soft prompt Pemb E Rexe comprises the only tunable parameters dur- ing training. Typically, such a weight matrix in a neural network has a full rank both before and after model training. Therefore, to investigate whether the prompt matrix Pomb has low "intrinsic rank", need to reparameterize the soft prompt.
Our contribution can be summarized as follows:
We present an empirical study to show that the soft prompt exhibits low "intrinsic rank" characteristic.
Consider an example matrix M E Rmxn, the singular value decomposition of the matrix M is in the form of M UEV, where U E Rmxm V E Ru and Σ E Rmxn diagonal matrix. Note that the diagonal entries of Σ represent the singular values of M, and the number of non-zero
Motivated by such findings, we propose our method to initialize the soft prompt with low- rank matrices. Experimental results on the SuperGLUE benchmark in both high-resource
2y can be a single token or sequence of tokens.

---

T5
c-1
R-1
Soft Prompt
Input Embedding
the rank of Σ Decomposed Soft Prompt
Input Embedding
Trainable
Frozen
Dot Product
Concat
Figure 2: Left: vanilla prompt tuning, Right: our proposed DPT. The soft prompt Pems E Rexe (left) can decomposed into matrices, A E Rexb and B R By setting b & min(c,e), the number of trainable parameters of our DPT (eb + be) is much smaller than that of vanilla prompt tuning (ec).
singular values is equal to the rank of M. With the above decomposition, can first reparameterize the soft prompt Pamb E Rexx as:
P Pemb= UEV
(2)
Here, the matrices U E xRe and V E Re are randomly initialized and set trainable parame- ters in this analysis. Σ RRex is initialized with positive values along the diagonal, while the off- diagonal elements are set as We further impose constraints on such that only the diagonal en- tries are trainable while the remaining are frozen. Intuitively, if the soft prompt Pemb has a low "in- trinsic rank",, then some of the diagonal entries of E will converge to 0 during training. We provide explanation of why approximation for the rank of Pamb in Ap- pendix A.1.
Epach
Figure 3: Number of positive and negative diagonal entries in Σ.
number negative diagonal entries increases as training progresses. We interpret these observa- tions as indicative that the soft prompt has a low "intrinsic rank". Note that the soft prompt length is set as the commonly used value of 100 in this amalysis, this accounts for the initial count of 100 positive diagonal entries. We also observe similar behavior on other datasets with models of differ- ent sizes, with additional details provided in Ap- pendix A.2.
However, due to the dense nature of the neural network, the diagonal entries of Σ can hardly be updated to exact O. To resolve this challenge, we apply a rectified linear unit (ReLU) to the diagonal matrix Σ:
Pemb V
(3)
Method
Through such reparameterization of the soft prompt Pemb, we can count the number of posi- tive entries in the rectified diagonal matrix Σ post- training to approximately investigate the rank the matrix Pemb- We can infer that if the soft prompt has a low "intrinsic rank", more diagonal entries of E E Rexe will be updated to negative values.
Motivated by the observation of the soft prompt's low "intrinsic rank" behavior, as discussed in the previous section, propose a parameterization of the soft prompt that explicitly constrains it be of low rank. In this section, we provide the details of our approach, as illustrated on the right side Figure 2.
We conduct our empirical experiments with the T5-Base model on the CB dataset (De Marneffe al., 2019) of SuperGLUU benchmark, and Fig- ure 3 shows the results. We observe that the num- ber positive diagonal entries decreases while the
3.1 Decomposition of Soft Prompt
As described in Section 2.1, the input of the vanilla prompt tuning approach is the concatenation of the

---

prompt P and the input text X. The overall input the pre-trained model is then as follows:
4 Experimental Setup
4.1
Datasets
{pr, (2:,P2.-.. P2. ................................ Po, 31, . }
(4)
To assess the effectiveness of our proposed DPT method, we conduct experiments on eight datasets from the SuperGLUE benchmark (Wang et al., 2019) under both high-resouree and low-resource conditions. We follow previous work (Lester et al., 2021; Vu al., 2022) to report the performance of our model on the validation sets due the re- stricted access to the test sets of SuperGLUE. We provide detailed descriptions, evaluation metrics, and statistics of the SuperGLUE benchmark in Ap- pendix A.3.
where there are è soft prompt tokens and n text tokens. The focus of our approach is a more effi- cient representation of the soft prompt Pemb. which usually has dimension e x c.
Instead of using random initialization for the soft prompt, we decompose it into the product of two matrices:
Pemb AB
(5)
where A E Rexb and B E Rbxe, as illustrated on the right side of Figure 2. One of the main ad- vantages of this representation is that it offers us the ability modulate the rank of the soft prompt Pemb by controlling the size of the intermediary dimension b, which we term as the "bottleneck". Specifically, by setting b to a relatively small value, the resulting soft prompt Pemb inherently possesses a low rank. Moreover, this decomposition approach leads to a reduction in the number of trainable pa- rameters by compressing the bottleneck. Specifi- cally, there ec trainable parameters in the vanilla prompt tuning approach. With our proposed DPT approach, the number of trainable parameters is eb + be. When setting < min(c, e), the total number of trainable parameters can be significantly reduced. Note that e is usually set to 100 and is 1024 for L5-Lerge (Raffel et al., 2020), and we set b as 10 in our main experiments. A more detailed analysis of bottleneck b can be found in Section 6.1.
4.2 Settings
We use the encoder-decoder T5 model (Raffel et al., 2020) as the backbone for our experiments. We em- ploy three variants of the T5 model: T5-Small, T5- Base, and T5-Large, which comprise 60M, 220M, and 770M parameters respectively. We implement our approach with the HuggingFace Transformers library (Wolf et al., 2020). Each dataset within the SuperGLUE benchmark is converted into text-to- text format for compatibility with the T5 model. The model trained for 100 epochs with an initial learning rate of 0.3, and Adam (Loshchilov and Hutter, 2019) is employed as the optimizer. The length of the soft prompt is fixed at 100. The embedding dimension is configured as per the model variant, being set to 512, 768, and 1024 for T5-Small, T5-Base, and T5-Large respectively. Importantly, we set the bottleneck size 10 achieve balance of performance and parameter efficiency. This bottleneck plays an instrumental role in inducing low-rank constraints on the em- bedded soft prompt matrix Pemb- We initialize the matrices A and B of our prompt parameters with Gaussian distribution.
3.2 Training and Inference
We adopt similar training procedures as discussed in Section 2.1. Our training objective is as follows:
0r (Y[[P;
(6)
4.3 Baselines
where P indicates the prompt with length to- kens, 0 denotes the frozen parameters of the pre- trained model, and 0p is the trainable parameter associated with the prompt P. In our approach, the representation of the soft prompt Pomb, is de- rived as the product of two matrices (Eq. 5), A and B. which are both initialized randomly and serve the tunable parameters of our model. Op cor- responds to the parameters contained in matrices and B. After the training is completed, we can store the product of A and B for inference.
To maintain consistency in evaluation metrics and preprocessing procedures across all datasets, we reproduce most of the scores to ensure fair com- parison. More details about implementation can be found in Appendix A.4.
Fine-tuning We compare DPT with the conven- tional fine-tuning approach of the T5 model (Raffel et al., 2020), where separate copies of the model must be tuned and stored for different datasets. Though fine-tuning is not a sarameter-efficient ap-

---

# Trainable
WSC
WiC
BoolQ
CB
COPA
MultiRC
ReCoRD
RTE
Model
Avg.
Params.
Acc.
Acc.
Acc.
Acc.
Acc.
F1
FJ
Acc.
T5-Smal1
Fine-Tuming
60M
67.94
68, 18
77.06
89.28
59.00
66.98
55.64
72.56
69.58
Prompt Tuning (Lester et sl., 2021)
51K
63.14
59.29
66.78
74.99
58.33
64.89
52.75
59.92
62.51
Residual PT (Razdaibiedina et sl., 2023)
462K
63.14
60.96
73.35
72.02
56.66
65.12
53.08
67.02
63.91
Ours
6K
63.78
64.31
71.74
79.16
58.00
64,89
53.27
70.51
65.70
TS-Base
Tine-Tuming"
220M
81.70
69.30
82.30
91.70
60.00
76.90
80,90
84.50
78.41
Prompt Tuning (Lester et sl., 2021)
77K
64.74
59.97
68.18
70.23
56.33
72.69
71.84
56.43
64.30
Residual PT (Razdaibiedina et al., 2023)
693K
67.94
63.31
80,00
77.37
$6,66
72.11
72.21
81.70
71.41
Ours
9K
67.30
68.49
78.64
78.56
$6,66
71.22
72.53
81.94
71.88
TS-Large
Fine-Tuming"
770M
88.50
73.50
88.30
94.30
87.00
85,40
89,20
90.60
87.10
Prompt Tuning (Lester et al., 2021) Residual PT (Razdaibiedina et al., 2023)
102K
76.52
70.00
84.09
74.40
62.00
76.18
84.51
88,95
77.08
925K
70.50
72.25
85.04
73.21
62.66
76.46
84,36
88.92
76.67
Ours
11K
79.16
71.99
84.76
88.09
62.33
76.72
84,46
90.25
79.72
Table 1: Results on the SuperGLUE validation set. All scores are the average over 3 runs with different random seeds. The last column (Avg.) indicates average score across datasets the SuperGI.UIE benchmark. models with symbol denote that the results retrieved from Aribandi et al. (2022), and the rest reproduced by following the original implementations. Standard deviation of three runs is in Appendix A.5.
proach, it is usually treated upper bound of performance.
residual prompt tuning (Razdaibiedina et al., 2023) across datasets from the SuperGLUE benchmark. Besides the model performance, we also include the comparison of the total number trainable parameters. Remarkably, our model consistently outperforms the aforementioned approaches across T5 models of different sizes in terms of the average scores over the 8 datasets. Specifically, our model outperforms the vanilla prompt tuning by 3.19, 7.58, and 2.64 points in terms of the average score (Avg.) with T5-Small, T5-Base, and T5-Large models, respectively. In most cases, our model surpasses the vanilla prompt tuming approach by a large margin. Moreover, DPT is highly efficient terms of parameter usage, requiring approximately one-ninth of the number of trainable parameters compared to vanilla prompt tuning (i.e., 11.2K vs. 102K for T5-Large). When compared with the cently proposed residual prompt tuning, our model also shows better or comparable performance. Cru- cially, DPT achieves this while requiring signif- icantly fewer trainable parameters. Specifically, residual prompt tuning consumes nearly 84 times more parameters than our method (i.e., 1.2K vs. 925K for T5-Large). Note that we set the length of the soft prompt for residual prompt tuming to 100 which empirically yields better performance 3, We also provide the experimental results of residual prompt tuning when setting the length of the soft prompt to in Appendix A.6. The experimental
Prompt Tuning Prompt tuning (PT) (Lester et al., 2021) prepends a soft prompt to the input embedding. It adapts to downstream tasks by exclu- sively updating the parameters of the soft prompt, keeping the language model frozen.
Residual Prompt Tuning Residual prompt tun- ing (Residual PT) (Razdaibiedina et al., 2023) is a recently proposed variant of PT. It enhances the original prompt embeddings through an additional layer of shallow network instead of passing them di- rectly into the frozen transformer layer. A residual connection is also employed boost the perfor- mance and convergence rate.
Our proposed model is not directly comparable with the multi-stage XPrompt (Ma et al., 2022), which iteratively updates the soft prompt embed- ding. Adapter-based methods (Houlsby et al., 2019; Rücklé et al., 2021; Guo et al., 2021) are also not in- cluded for comparison in this paper they require efforts to modify the transformer layers. Further- more, they often include considerably larger num- ber of parameters than prompt tuning approaches as mentioned Razdaibiedina et al. (2023). More detailed comparisons can be found in Section 7.
5 Results
5.1
Main Results
Table presents a comparison of our DPT with the vanilla prompt tuning (Lester et al., 2021) and
'There are 100-token and 10-token residual prompt tuning variants in the original paper (Razdaibiedina et al., 2023).

---

10
Awg
16
Sampla
16
Sample
16
Samalo
16
Sample:
1g
Samsles
(a)
(b)
(d)
(e)
Figure Comparison DPT with vanilla prompt tuning (PT) and Residual PT (Res PT) in the few-shot setting.
WC
ATE
COPA
80
100
95
70
90
65
75
880
70
Ouas Bas
70
Dut
Rea
65
60
10
12
14
8
14
10
12
Bottleneck
Bottleneck
Bottleneck
Bottienick
(b)
(c)
(d)
Figure 5: Comparisons of DPT with and Res PT with different sizes of the bottleneck b. Each point represents the average of three with different random seeds.
results demonstrate the effectiveness of our pro- posed low-rank soft prompt in both performance and parameter efficiency.
Analysis
Sensitivity of Bottleneck Size
As mentioned Section 3.1, the rank of our soft prompt can adjusted through the bottleneck In this section, we investigate how the size of the bottleneck impacts the performance of DPT. We conduct experiments using Ta-Large on the WiC. CB, RTE, and COPA datasets, with bottleneck b E (4, 6, 8, 10, 12, 14). The results are shown in Figure 5. We experiences performance fluctuations with differ- sizes of the bottleneck b, DPT outperforms the vanilla prompt tuning and residual prompt tuning in quite a few cases. Intuitively, the size of the bot- tleneck plays a critical role in the expressiveness of our model. We observe a performance drop when the size of the bottleneck is getting smaller. We find similar behavior when T5-Bast is used, and the results can be found in Appendix A.7.
5.2 Few-shot Performance
We further evaluate method in few-shot set- ting on all datasets in the SuperGLUE benchmark. Specifically, we sample total of 8, 16, and 32 train- ing instances without balancing the label distribu- tion to mimic a realistic scenario. To ensure a fair comparison, the sampled data for our method, vanilla prompt tuning, and residual prompt tuning are kept identical for each run. We employ the T5- Large model as the backbone for this evaluation as it achieves the strongest performance in the vanilla prompt tuning. The results presented are averaged across three separate runs with different random seeds. We compare our approach against the pre- vious methods in terms of the average score over all datasets in the SuperGLUE benchmark, and the results are shown in Figure 4a. We observe that our method consistently outperforms both vanilla prompt tuning and residual prompt tuning in our experiments. Additionally, We also include ex- perimental results on specific datasets from Super- GLUE in Figure 4b, 4c, 4d, and 4e. These results further demonstrate the effectiveness of our method over the baselines in the shot setting.
that though approach
6.2 Effect of Prompt Length
To investigate the effect of prompt length on the performance of our proposed method, we conduct an analysis on four of the SuperGLUE tasks', and the results are shown in Figure 6. The prompt
"These four datascts are randomly selected for analysis.

---

WIC
RTE
COPA
70
65-
57-
Ourt
Durt
Cure
Gutt
Roo
Rho
Ras
Longth
70
Promitt Longth
230
Prompt Length
Prompt Lon gth
250
(a)
(b)
(c)
(d)
Figure 6: Comparisons of our method with and Res PT with different prompt lengths e {200 100, 200 Each point is the average of three runs with different random seeds.
length e set as 20, 100, and 200. We use the T5- Large model as the backbone, while the bottleneck variable is held constant at 10. As mentioned in Section 3, the number of trainable parameters in the vanilla PT is ec, whereas for our DPT method, it is eb + bc. The benefit of our approach in terms of parameter--effciencyy is more pronounced as the prompt length increases". We observe that our DPT consistently outperforms the vanilla prompt tuning approach across various prompt lengths. More- over, DPT is generally found to be comparable or superior to residual prompt tuning. Corroborat- ing the observations made by Lester et al. (2021), a long soft prompt does not necessarily improve the model performance. While prompt length of 100 has been empirically validated as ef- fective hyperparameter in previous work (Lester et al., 2021; Razdaibiedina et al., 2023), our analy- sis reveals that the performance gains can still be achieved with prompt length of 200, as illustrated in Figure 6a. Furthermore, our DPT is less sen- sitive to the change of prompt length on CB and RTE, shown in Figure 6b and Figure 6c. We observe similar behavior when switching T5- Base as the backbone, and details are included in Appendix A.8.
WIC
70-
160
30
Prompe Longth
Penngh
(a)
(h)
WIC
10
Arompt Longon
Lenghh
(c)
(d)
Figure Further comparisons with short prompt lengths and small bottleneck. 7a and 7b are based T5-Large while 76 and 7d are based on T5-Base.
efficiency under such conditions, we compress the bottleneck to an extremely low value of 2. The experiments conducted on the CB and WiC tasks using the T5-Large and T5-Base model are illus- trated Figure 7. Notably, even with the bottle- neck compressed to such a small value, our method is still able to surpass baselines. This demonstrates that our method is able to effectively generalize to extremely short prompts with very constrained bottleneck.
6.3 Constrained Bottleneck under Short Prompt
A critical aspect we sought to address with DPT is the potential to sustain both parameter efficiency and competitive performance, even with a short prompt. When the prompt length e is reduced to values such as 6 or 10, DPT may not exhibit param- eter efficiency if the bottleneck b remains to be 10. To explore the possibility of achieving parameter
6.4 Bottleneck vs. Number of Trainable Parameters
To further study the effect bottleneck size with- out considering parameter efficiency, we conduct experiments with the bottleneck of varying mag-
'When b and are fixed, the ratio ebide decreases us the prompt length increases. This implies that our method ex- hibits higher parameter efficiency with longer prompt lengths.

---

85
90
72
85
64
65
60
53
50
10800
Bottleneck
Bottleneck
Bottleneck
(a)
(b)
(c)
We enlarge the bottleneck to even 10000 to study the performance of DPT. r-axis is the bottleneck size. y-axis is the performance. 8a and 8b are from LarLe while & is from T5-Base.
Figure 8:
nitudes, specifically b E {10, 1000, 10000), while maintaining a fixed prompt length of 100 tokens. The experimental results are on the RTE dataset with the T5-Large and T5-Base model, and we record the average, minimal, and maximum scores across three runs. Figure 8 shows the comparison, and it is noteworthy that the number of trainable parameters involved are 11.2K, 1.IM, and 11.2M respectively for varying bottleneck sizes under T5-Large. When configuring the bottleneck b to a large value, the soft prompt ceases to be a low-rank matrix and the number of trainable parameters is more than that of a vanilla prompt tuming approach. More importantly,
of the adapter were updated (Pfeiffer et al., 2021; Sung et al., 2022; Chen et al., 2023; Zeng et al., 2023). To enhance this, Karimi Mahabadi et al. (2021) incorporated hypernetworks, which gener- ate weights for the main network. thus enabling shared information capture across tasks. Following that, Hu et al. (2022) proposed LoRA to approxi- mate the update the neural weight. LoRA was based on the assumption that the change in weights during model adaptation has a low "intrinsic rank". Different from the adapter, LoRA did not intro- duce additional latency. Apart from being PEFT method, the adapter has also been employed in broader applications in natural language process- ing (Pfeiffer et al., 2021).
observe that increasing the number of trainable parameters does not necessar- ily lead enhancement in performance. Fur- thermore, overparameterrvzatoo could deteriorate the stability of the performance.
Our work diverges from LoRA (Hu et al., 2022) and the broader adapter framework. Rather than learning updates to the parameters, our approach directly learns the parameters, under the hypoth- esis that the soft prompt inherently exhibits low "intrinsic rank". Furthermore, LoRA focuses on Query- and Value-specific parameters within the transformer architecture, whereas our approach adheres to the prompt tuning paradigm, where trainable parameters are exclusively inserted in the embedding layer. Despite these fundamental differences, it is noteworthy that both our approach and LoRA share underlying principle of lever- aging low-rank structures.
7 Related Work
Parameter-efffeicnt fine-tuning (PEFT) meth- ods (He et al., 2022a; Ben Zaken et al., 2022; He et al., 2022b; Mao et al., 2022; He et al.. 2022c) have emerged as popular approach to fine-tune language model which can largely re- duce the number of trainable parameters while maintaining competitive performance. There two primary paradigms for current PEFT methods: adapter-based and prompt-based approaches.
Adapter-based Methods The concept of the adapter originally proposed by Houlsby al. (2019). They inserted down projection layer fol- lowed by an up projection layer each sub-module the transformer sequentially. When adapting a model to downstream tasks, only the parameters
Prompt-based Methods Prompt-based methods can be categorized into hard prompt and soft prompt approaches. Hard prompts involve adding fixed sequence of discrete tokens for the model to condition on for generation (Jiang et al., 2020; Shin et al., 2020; Zhao et al., 2021; Liu et al., 2023).

---

However, they are sensitive slight changes and require complex designs, like verbalizer selection, To address this, trainable soft prompt methods were introduced. Li and Liang (2021) introduced prefix tuning, which adds virtual tokens in each layer of the encoder stack. As a simplification of prefix tuning. Prompt Tuning (Lester et al., 2021) only adds soft prompt to the embedding layer. Further advancements include incorporating soft prompts across all transformer layers enhance perfor- mance (Liu et al., 2022b). Additionally, transfer learning-bast based methods (Wang et al., 2023; Vu et al., 2022) have been explored for better prompt initialization through pre-training. For instance. Wang et al. (2023) performed pre-training to learn a soft prompt on collection of source tasks and subsequently employed the acquired prompt em- beddings as initialization for target tasks. Transfer learning- based techniques offer the potential for better initialization and more effective prompt tun- ing. In contrast to prior work, this paper focuses on developing a more efficient parameterization of the soft prompt.
models. We leave the explorations with encoder- only models and other large-scale pre-trained mod- els for future work.
Acknowledgements
We would like to thank the anonymous reviewers, our meta-reviewer, and senior area chairs for their constructive comments and support on this work. This rexearch/project is supported by Ministry of Education, Singapore, under Tier 3 Programme (The Award No.: MOET320200004), and Min- istry of Education, Singapore, under its Academic Research Fund (AcRF) Tier 2 Programme (MOE AcRF Tier 2 Award No: MOE-T2EP20122-00 1). Any opinions, findings and conclusions or recom- mendations expressed in this material those of the authors and do not reflect the views of the Ministry of Education, Singapore.
References
Armen Aghajanyan, Luke Zettlemoyer, and Sonal Gupta. 2020. Intrinsic dimensionality explains the effectiveness language model fine-tuning. In Pro- ceedings of ACL.
8 Conclusion
In this work, we uncover the low "intrinsic rank" behavior inherent in the soft prompt through em- pirical examination. Motivated by these findings, we introduce Decomposed Prompt Tuning (DPT), a novel approach that reparameterizes the soft prompt using two compact matrices. By adjust- ing the bottleneck, DPT enables the effective ma- nipulation of the soft prompt matrix, ensuring it maintains low rank. Notably, DPT attains strong performance across the SuperGLUE benchmark in high- resourees and low-resource scenarios while substantially reducing the number of trainable pa- rameters.
Vamsi Aribandi. Yi Tay, Tal Schuster, Jinfeng Rao, Huaixiu Steven Zheng. Sanket Vaibhav Mehta, Hon- glei Zhuang, Vinh Q. Tram, Dara Bahri, Jianmo Ni, Jai Gupta, Kai Hui, Sebastian Ruder, and Donald Met- zler. 2022. Ext5: Towards extreme multi- task scaling for transfer learning. In Proceedings of ICLR.
Akari Asai, Mohammadreza Salehi, Matthew Pe- ters, and Hannaneh Hajishirzi. 2022. ATTEMPT: Parameter-efffcient multi-task tuning via attentional mixtures soft prompts. Proceedings of EMNLP.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hin- ton. 2016. Layer normalization. In Proceedings of NewrIPS.
Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. 2022. BitFit: Simple parameter-efficient fine-tuning for transformer-basee masked language-models. In Proceedings of ACL.
Limitations
Despite our proposed approach being simple but ef- fective, our method still possesses a few limitations. We mainly evaluate our model on the natural under- standing tasks, i.e., SuperGLUE. We do not evalu- ate our proposed method on the natural language generation tasks. Furthermore, inherits few drawbacks from the vanilla prompt tuning, such slow convergence. These limita- tions serve as future directions for further improve- ment. Another limitation is that we only evaluate our method with the encoder- decoder backbone
Tianrun Chen, Lanyun Zhu, Chaotao Deng. Run- long Cao, Yan Wang, Shangzhan Zhang, Zejian Li, Lingyun Sun, Ying Zang, and Papa Mao. 2023. Sam- adapter: Adapting segment anything in underper- formed scenes. Pruceedings of ICCV.
method also
Marie-Catherine De Marneffe, Mandy Simons, and dith Tonhauser. 2019. The commitmenthank: Investi- gating projection in naturally occurring discourse. In proceedings of Sinn und Bedeutung.
Jacob Devlin, Ming-Wo Wei Chang. Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of

---

deep bidirectional transformers for language under- standing. In Proceedings of NAACL.
Xiao Liu, Kaixuan Ji. Yicheng Fu, Weng Tam, Zhengx- iao Du, Zhilin Yang, and Jie Tang. 2022b. P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks. In Proceedings of ACL.
Demi Guo, Alexander Rush, and Yoon Kim. 2021. Parameter-efficiiet transfer learning with diff prun- ing. In Proceedings of ACL.
Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujic Qian, Zhilin Yang, and Jie Tang. 2021. Gpt understands, too, 103V:2203..1388.
Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg- Kirkpatrick, and Graham Neubig. 2022a. Towards a unified view of parameter-efficient transfer learning. In Proceedings of ICLR.
Yinhan Liu, Myle Ou, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Ro bert)a: A robustly optimized (bert) pretraining approach.
Shwai He, Liang Ding, Daize Dong, Jeremy Zhang, and Dacheng Tao. 2022b. SparseAdapter: An easy approach for improving the parameter-efficiency of adapters. In Proceedings of EMNLP.
Nya Loshchilov and Frank Hutter. 2019, Decoupled weight decay regularization. In Proceedings of ICLR.
Yun He, Steven Zheng, Yi Tay, Jai Gupta, Yu Du, Vamsi Aribandi, Zhe Zhao, Yaguang Li. Zhao Chen, Donald Metzler, Heng-Tze Cheng, and Ed H. Chi. 2022c. HyperPrompt: Prompt-based aas--oonditioning of transformers. In Proceedings of ICLR.
Fang Ma, Chen Zhang, Lei Ren, Jingang Wang, Qifan Wang, Wei Wu, Xiaojun Quan, and Dawei Song. 2022. XPrompt: Exploring the extreme of prompt tuning. In Proceedings of EMNLP.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019, efficit transfer learning for NLP. In Proceedings of ICML.
Rabeeh Karimi mahabadi, James Henderson, and Sebas- tian Ruder. 2021. Compacter: Efficient low-rank hy- percomplex adapter layers. In Proceedings of ICLR.
Edward Hu, yelong shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In Proceedings of ICLR.
Yuning Mao, Lambert Mathias, Rui Hou, Amjad Alma- hairi, Hao Ma, Jiawei Han, Scott Yih, and Madian Khabsa, 2022. UniPELT: unified framework for parfieter-efficient language model tuning. In Pro- ceedings of ACL.
Zhenghao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. 2020. How Can We Know What Language Models Know? Transactions of the Association for Computational Linguistics.
Matthew E. Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lec, and Luke Zettlemoyer. 2018. Deep contextualized word repre- sentations. In Proceedings of ACL.
Rabeeh Karimi Mahabadi, Sebastian Ruder, Mostafa Dehghani, and James Henderson. 2021. Parameter- efficient multi-task fime-tuning for transformers via shared hypemetworks. In Proceedings of ACL.
Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. 2021. AdapterFusion: Non-destructive task composition for transfer learning. In Proceedings of EACL.
Brian Lester, Rami A1-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficien prompt tuning. In Proceedings of EMNLP.
Guanghui Qin and Jason Eisner. 2021. Learning how to ask: Querying LMs with mixtures of soft prompts. In Proceedings of NAACL.
Chunyuan Li, Heerad Farkhoor, Rosanne Liu, and Jason Yosinski. 2018 Measuring the intrinsic dimension of objective landscapes. In Proceedings of ICLR.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAl blog.
Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of ACL.
Haokun Liu, Derek Tam, Mohammed Mugeeth, Jay Mo- hta. Tenghao Huang, Mohit Bansal, and Colin A Raf- fel. 2022a. Few-shot parameter-efficient fine-tuning better and cheaper than in-context learning. In Proceedings of NeurIPS.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with unified text-to-text trans- former. Journal of Machine Learning Research.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023. Pre- train, prompt, and predict: A systematic survey of prompting methods natural language processing. ACM Comput. Surv.
Anastasia Razdaibiedina, Yuning Mao, Rui Hou, Ma- dian Khabsa, Mike Lewis, Jimmy Ba, and Amjad Almahairi. 2023. Residual prompt tuning: Improv- ing prompt tuning with residual reparameterization. In Proceedings of ACL.

---

Andreas Rückléé Gregor Geigle, Max Glockner, Tilman Beck, Jonas Pfeiffer. Nils Reimers, and Iryna Gurevych. 2021. AdapterDrop: On the efficiency of adapters in transformers. In Proceedings of EMNLP.
Proof. Each column of AB is a combination of the columns of A, which implies that R(AB) - R(A). Hence, dimB(((A)) S dim(R(A)), or equiva- lently, rank(AB) S rank(A) Each row of AB is a combination of the rows of B - rowspace (AB) S rowspace (B), but the dimension of rows- pace dimension of column space rank, so that rank(AB) S rank(B).
Taylor Shin, Yasaman Razeghi, Robert I.. Logan IV, Eric Wallace, and Sameer Singh. 2020. AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts. In Proceedings of EMNLP.
According to theorem A.1, if UEV, it can be easily obtained that:
Yi-Lin Sung, Jaemin Cho, and Mohit Bansal. 2022. VI-adapter: parameter-efficien transfer learning for vision-and- languugagg tasks. In Proceedings of CVPR.
rank(P) <
(Znk))
Tu Vu, Brian Lester, Noah Constant, Rami Al-Rfou', and Daniel Cer. 2022. SPoT: Better frozen model adaptation through soft prompt transfer. In Proceed- ings of ACL.
rank(E) is an upper bound of rank(P), be- cause U and V are always full rank during training process. Thus, we can use rank(E) as an approxi- mation for rank
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Aman- preet Singh. Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019. Superglue: A stickier benchmark for general-purpose language understand- ing systems. In Proceedings of NeurIPS.
A.2 Record of Rank Change
We conduct additional experiments with different models and datasets to probe the rank change of soft prompt. We find consistent observations that the soft prompt tends to exhibit low "intrinsic rank" behavior. As we can see from Figure 9, the number of positive entries is decreasing while the number of negative entries is increasing. In the second row, we also record the rank change of the soft prompt. Specifically, we calculate the rank of the matrix obtained by multiplying U, E, and V.
Yaqing Wang, Sahaj Agarwal, Subhabrata Mukherjee, Xiaodong Liu, Jing Gao. Ahmed Hassan Awadal- lah, and Jianfeng Gao. 2022. AdaMix: Mixture-of- adaptations for parameter-efficient model tuning. In Proceedings of EMNLP.
Zhen Wang, Rameswar Panda, Leonid Karlinsky, Roge- ri Feris, Huan Sun, and Yoon Kim. 2023. Multitask prompt tuning enables parameter-efffiient transfer learning. Proceedings of ICLR.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer. Patrick von Platen, Clara Ma, Yacine Jemite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In Pro- ceedings of EMNLP.
A.3 Dataset Details
We record the statistical details of the SuperGLUE benchmark and the metrics we use, and Table 2 shows the statistics. Note that the original CB dataset has 554 training samples, we follow (Raffel et al., 2020) to keep only 259 samples with a "True" label. We transform the task to text-to-text format. Specifically, we highlight the ambiguous pronoun in the input text and ask the model to predict the noun that it refers to (Raffel et al., 2020).
Guangtao Zeng, Peiyuan Zhang, and Wei Lu. 2023. One network, many masks: Towards more parameter- efficient transfer learning. In Proceedings of ACL.
Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improv- ing few-shot performance of language models. In Proceedings of ICML.
A44 Baseline Implementation Details The length of the soft prompt is set as 100 for vanilla prompt tuning, which is an optimal setting according to the paper (Lester et al., 2021). The soft prompt can be initialized by Gaussian distri- bution or copying from the vocabulary embedding. Here, we adopt the second initialization strategy because of its superiority.
A Appendix
A.1 Rank Explanation
Theorem A.1. Let A and B be matrices such that
the product AB is well defined. Then
There are 10-token and 100-token residual prompt tuning variants (Razdaibicdina et al., 2023). We implement the 100-token one because it has
rank(AB) & mainrrnn(AA) rark(B)

---

100
100
Nm
Num
20
0
Epoch
Epoch
Epoch
(a) T5-Small, RTE
(b) T5-Base, RTE
(c) TS-Small, CB
100
100
100
80
80
60
60-
60-
Runk
Rank
40
20
40
60
80
10c
20
40
101
20
80
100
Epoch
Spoch
Eppach
(d) T5-Small, RTE
(e) TS-Base. RTE
(1) T5-Small, CB
Figure 9: The first row is record of positive and negative diagonal entries. The second row is the rank record.
A.6 Residual PT 10 tokens
Datasct
Train
Dev
Task
Metric
In most parts of this paper, we focus on soft prompt of 100 tokens. (Razdaibiedina et al., 2023) present 10-token and 100-token variant of residual prompt tuning in their main results. We reproduce the 100- token residual prompt tuning in the main result section. Here, we also reproduce the 10-token one in Table 4 The results are consistent with the original paper that the performance of 10-token version is inferior to that of 100-token version.
BoolQ
9,427
3,270
QA
Acc.
CB
250
NLI
Acc.
CОPA
400
100
QA
Acc.
MultiRC
27,243
4,848
QA
FI
ReCoRD
100,730
10,000
QA
FI
WIC
5,428
638
WSD
Acc.
WSC 259
104
Coref.
Acc.
RTE 2.490
277
NLI
Acc.
Table The details of 8 SuperGI.UF tasks used our experiments.
A.7 Various Bottleneck Size
We try different bottleneck b E {4,6, 8, 10, 12, 14) and fix prompt length as 100. The result of T5-Base is similar to that of T5-Large. DPT prompt tuning in most cases, which demonstrates the robustness of DPT. In Figure 10, we show the result on T5-Base.
surpass
better performance. We also follow the paper to set the bottleneck as 400, above which leads to no im- provement according to the paper. We follow them to adopt ReLU as activation function and perform layer normalization (Ba et al., 2016).
A.8 Various Prompt Length
We try prompt length 20, 100 and 200 and fix the bottleneck as 10. The result of T5-Base is similar to that of T5-Large. The performance of DPT is consistently better than PT under various prompt lengths shown in Figure 11.
A.5 Standard Deviation of Scores
We report the the standard deviation of three runs for our method, prompt tuning and residual prompt tuning. The results are shown Table 3.

---

Model
WSC
WIC
BoolQ
CB
CОРA
MultiRC
ReCoRD
RTE
T5-Smal1
PT
1.10
2.82
1,28
3.57
1.15
0.90
0.30
3.12
Res PT
1.10
5.7
1.64
4.12
0.57
0.51
0.22
3.68
Ours
0.55
0.65
0.85
4.12
4.35
0,90
0.15
1.98
T5-Base
PT
1.10
6.30
0.01
1.02
1.15
0.24
0.09
0.20
Res 0.55
3.68
10.11
10.46
2.08
0.20
0.13
1.50
Ours
0.96
1.33
0.79
6,43
2.08
0.46
0.10
1.30
T5-Large
PT 3.08
1.04
0.82
1.03
0.00
0.18
0.09
0.20
Res PT
0.56
0.13
1.79
1.52
0.04
0.17
0.75
Ours
1.46
0.63
0.27
2.06
2.51
0.39
0.15
0.36
Table 3: We report standard deviation (%) of three runs for PT, Res PT and our method.
# Trainable
WSC
WiC Acc.
BoolQ
CB
COPA
MultiRC
ReCoRD
RTE Acc.
Model
Avg.
Params.
Acc.
Acc.
Acc.
Acc.
F1
T5-Small
Residual PT
416K
64.73
58.09
69.70
73.21
58.66
63.21
52.82
62.21
62.83
T5-Base 74.99
Residual PT
624K
68.26
66.87
79.11
57.66
71.20
72.41
68.58
69.88
T5-
Residual PT
832K
69.86
69.74
84.18
70.82
59.33
74.77
84.31
88.20
75.15
Table 4: Results on the SUp validation set. All scores are the average over 3 runs with different random seeds. The last column (Avg.) indicates the average across all datasets in the SuperGLUE benchmark.
75
80
57
00
075
56
70
50
45
10
12
6
12
14
10
14
54
10 Bottlened reck
12
Bottleneck
Bottleneck
Bottleneck
(a)
(h)
(c)
(d)
Figure 10: Comparisons of DPT with PT and Res PT with different sizes of the bottleneck b. Each point represents the average of three runs with different random seeds.
70
3-8
86
20C
Prompt Length
Prompt Length
Promot Length
Promit Length
(a)
(b)
(c)
(d)
Figure 11: The comparison between method and prompt tuning of different length in {20, 100, 200). point in the figure average of three runs with random seeds. The backhone is TS-Base.
Each


Reviews:
[{"review_id": "bjVrtoVmdc", "reviewer": "Reviewer_1oRu", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Reviewer 1oRu:\n\nThank you for your insightful review. We appreciate your valuable feedback and will address the concerns raised.\n\nRegarding weakness 1 about our experimental settings:\n\nWe follow the previous parameter-efficient finetuning work [1] [2] [3] to evaluate the model’s performance on the SuperGLUE benchmark. Note that SuperGLUE contains eight datasets that cover natural language inference, conference resolution, question answering, and word sense disambiguation. Furthermore, to verify the generalization of our model, we also include the experimental results with different models, such as T5-small, T5-base, and T5-large. Therefore, we believe our proposed approach could generalize to more tasks.\n\n[1] The Power of Scale for Parameter-Efficient Prompt Tuning\n[2] SPoT: Better Frozen Model Adaptation through Soft Prompt Transfer\n[3] Residual Prompt Tuning: Improving Prompt Tuning with Residual Reparameterization\n\nRegarding weakness 2 about the computational cost and memory:\n\nWe follow previous PEFT work to mainly focus on the comparison of number of the trainable parameters and performance. The amount of computation cost and the number of trainable parameters is generally positively correlated. Similarly, more trainable parameters also lead to more memory consumption, because the parameters and gradients are required to be stored when training. Therefore, the computational cost and memory requirement is in descending order for Res PT, PT, and our method. For instance, the stable memory requirement is 25108MB, 17086 MB, 10756 MB, and 10754 MB, for Full Finetuning, Res PT, PT, and our method respectively. The effect of pursuing the extremely limited number of trainable parameters is not conspicuous in memory reduction. Even if we decrease the number of trainable parameters to 1, it still requires about 10754 MB of memory. \n\n\nRegarding the weakness of more related work:\n\nWe mainly focus on the variants of prompt tuning in this paper. We compare our method with original prompt tuning and the latest variant of prompt tuning (Res PT). As mentioned in Section 7 Related Work, we didn’t include adapter-based approaches [1] [2], which consume many more parameters, considering the up-projection and down-projection layers are required in every block of the transformer. We did not include work like [3][4] which are also prompt variants. This is because they emphasize the transfer ability of soft prompt. The advantage of our method is that it achieves promising performance with a small amount of trainable parameters.\n\n[1] LoRA: Low-Rank Adaptation of Large Language Models \n[2] Parameter-Efficient Transfer Learning for NLP\n[3] Multitask Prompt Tuning Enables Parameter-Efficient Transfer Learning\n[4] ATTEMPT: Parameter-Efficient Multi-task Tuning via Attentional Mixtures of Soft Prompts\n\nRegarding the weakness of interpretability of the derived soft prompts:\n\nWe consider the soft prompt as a continuous version of the hard prompt. Both are trying to elicit knowledge from language models for downstream tasks. However, the soft prompt is less interpretable than the hard prompt since the embedding space is continuous and much more complex.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The strengths of this paper are:\n\n1. The proposed approach is simple but effective, and it significantly reduces the number of trainable parameters while maintaining effectiveness.\n2. The low-rank reparameterization technique is a novel and effective way to reduce the number of trainable parameters.\n3. The experimental results on the SuperGLUE benchmark in both high-resource and low-resource scenarios demonstrate the effectiveness of the proposed method.\n4. The proposed method outperforms existing methods in terms of both effectiveness and efficiency.\n5. The paper is well-written, and the experimental results are presented clearly and comprehensively.", "reasons_to_reject": "According to the conference policies, weaknesses should not include factors such as the paper's length, the number of citations, or the authors' affiliations. Based on this, the potential weaknesses of this paper are:\n\n1. The proposed method is evaluated only on the SuperGLUE benchmark, and it is unclear how well it would perform on other datasets or tasks.\n2. The paper does not provide a detailed analysis of the computational cost and memory requirements of the proposed method, which could limit its practical applicability.\n3. The paper does not provide a thorough comparison with other related methods, which could limit the understanding of the proposed method's strengths and weaknesses.\n4. The paper does not provide a detailed analysis of the interpretability of the derived soft prompts, which could limit the understanding of the model's decision-making process.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes a novel approach called Decomposed Prompt Tuning via Low-Rank Reparameterization to derive soft prompts that significantly reduces the number of trainable parameters while maintaining effectiveness. The paper addresses the problem of high computational cost and memory requirements associated with prompt tuning, which is a popular technique for fine-tuning large pre-trained language models for downstream tasks. The main contributions of the paper are:\n\n1. A novel approach to derive soft prompts that reduces the number of trainable parameters while maintaining effectiveness.\n2. A low-rank reparameterization technique that helps reduce the number of trainable parameters.\n3. Experimental results on the SuperGLUE benchmark in both high-resource and low-resource scenarios, demonstrating the effectiveness of the proposed method.\n4. Comparison with existing methods, showing that the proposed method outperforms them in terms of both effectiveness and efficiency.\n\nThe paper falls under the category of approaches for data- and compute efficiency, and it provides a solution to the problem of high computational cost and memory requirements associated with prompt tuning. The paper's contributions are significant and relevant to the NLP community, and the proposed method has the potential to improve the efficiency and effectiveness of prompt tuning for fine-tuning large pre-trained language models for downstream tasks.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "IeFwpaiq0s", "reviewer": "Reviewer_JCRp", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Reviewer JCRp:\n\nThank you for your thoughtful comments. We appreciate your valuable feedback and will address the questions and concerns raised.\n\nRegarding the weakness of training efficiency:\n\nTo have a fair comparison, we follow the original prompt tuning [1] to set the number of training epochs as 100 for all the reported methods in our paper.  Even though the Res PT [2] used 20 epochs for their experiments,  we find that training for 100 epochs leads to further performance improvement. We attach average scores of 3 runs of Res PT on CB, RTE, and WiC datasets with T5-large below, training for 20 epochs and 100 epochs respectively. \n\n| Epoch            | 20 | 100 |\n|-------------------|-------------|----------------------|\n| CB                | 57.13       | 73.21                | \n| RTE               | 59.80       | 88.92                | \n| WiC               | 69.37       | 72.25                | \n\nNote that our method consumes fewer trainable parameters (i.e., 1/84 of  Res PT for T5-Large ) while achieving better performance. \n\n[1] The Power of Scale for Parameter-Efficient Prompt Tuning\n[2] Residual Prompt Tuning: Improving Prompt Tuning with Residual Reparameterization", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The experiment results are promising according to the paper, by finetuning far less trainable parameters and achieving better performance. Meanwhile, the similar performance can not be gained through simply reducing the soft prompt length. The analysis of the proposed method is sufficient.", "reasons_to_reject": "Although the decrease of the number of tuning parameters seems promising, I have noticed that the setup of the experiments needs 100 epochs, while in Residual Prompt tuning [2], the training epoch is 20. This is in accordance with the conclusion from [1], that although soft prompt tuning is efficient in memory, it is not efficient in training time, usually suffering from longer convergence. I think in practice, the trainable parameters are whether 10K or 100K, make no significant difference since the memory is not bottleneck, but increasing the training time to 5x may be unaffordable. \n\n\n\n[1]: Lialin V, Deshpande V, Rumshisky A. Scaling down to scale up: A guide to parameter-efficient fine-tuning[J]. arXiv preprint arXiv:2303.15647, 2023.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper focus on improving prompt tuning approaches for large language models, particularly in the context of high efficiency and effectiveness, by decomposing the soft prompt into low-rank matrices, which reduces the number of trainable parameters.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Xx5WdERcje", "reviewer": "Reviewer_4fcj", "summary": "", "questions": "Question A: Could you provide further empirical or theoretical analyses to explain why the decomposed prompt works effectively?\nQuestion B: The optimization of prompt tuning is typically challenging. Could you elaborate on how your approach addresses this issue?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Reviewer 4fcj:\n\nThank you for your thoughtful review. We appreciate your recognition of our contributions and will address your concerns and questions. \n\nQuestion A: Could you provide further empirical or theoretical analyses to explain why the decomposed prompt works effectively? \n\nIn Section 2, we conduct an empirical analysis to investigate the rank of soft prompt. Through re-designing the soft prompt, our experiments facilitate a probe into its rank. We further provide a mathematical explanation for using the rank of Relu(Σ) as an approximation for the rank of P_emb.\n\nAgain, here is a short explanation. First, we reparameterize soft prompt P_emb by the form: P_emb = U∙Relu(Σ)∙V. U and V are trainable parameters that are randomly initialized. We initialize the trainable diagonal entries of Σ with positive values while fixing other entries as 0.\nThrough such a design, the soft prompt can change its rank more freely.\n\nDuring the training process, U and V are always full rank, which means Rank(Relu(Σ))  is the upper bound of Rank(P_emb).  Then, we use Rank(Relu(Σ)) as an approximation of Rank(P_emb). We provide proof for this in Appendix A.1. Intuitively if soft prompt exhibits a low “intrinsic rank”, it will update more and more diagonal entries of Σ to be negative (After Relu activation, it will be zero).\nWe plot the number of positive and negative diagonal entries of  Σ  with the training proceeding in Figure 3. Furthermore, we also directly compute the rank of P_emb after every epoch, and its pattern is identical to the curve of the number of positive diagonal entries in Figure 3. We provide more figures in Appendix A.2 of the paper.\n\nOverall, the rank of P_emb and the number of positive diagonal entries of Σ are decreasing with training proceeding. We think those theoretical analyses and empirical records can serve as evidence to support our assumption that soft prompt exhibits a low “intrinsic rank”, which motivates us to reparameterize the soft prompt with two low-rank matrices explicitly. This also helps verify the effectiveness of our method.\n\nQuestion B: The optimization of prompt tuning is typically challenging. Could you elaborate on how your approach addresses this issue?\n\nAnswer: The main contribution of our paper is that we propose a parameter-efficient training approach with a much smaller number of trainable parameters (i.e., 1/9 of prompt tuning, 1/84 of residual prompt tuning for T5-large), while achieving better performance. The discussion of optimization issues is not the main focus of this paper. However, we do try different learning rate decay strategies like warmup and annealing for our experiments, but they are not as effective as expected. We find training for longer time like 100 epochs as prompt tuning [1] did is a good way to obtain stable performance.\n\n[1] The Power of Scale for Parameter-Efficient Prompt Tuning\n\nReason to reject: While the paper presents an intriguing method and boasts compelling experimental results, the authors fail to clearly explain the relationship between the rank and the soft prompt. The experimental results and analysis do not appear sufficiently robust to draw firm conclusions.\n\nAnswer: As we have restated in the reply to answer A, we empirically show that soft prompt has a low “intrinsic rank”, which motivates us to reparameterize the soft prompt with low-rank matrices. We find that a bottleneck around 10 helps us achieve a balance between parameter efficiency and performance (in the main experiment, we use bottleneck 10).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper's strength lies in its innovative approach to constructing the soft prompt through low-rank reparameterization. This concept contributes to a deeper understanding of parameter-efficient fine-tuning in prompt tuning. Additionally, the paper provides substantial experimental evidence to support the efficacy of the proposed method.", "reasons_to_reject": "While the paper presents an intriguing method and boasts compelling experimental results, the authors fail to clearly explain the relationship between the rank and the soft prompt. The experimental results and analysis do not appear sufficiently robust to draw firm conclusions.", "questions_to_authors": "Question A: Could you provide further empirical or theoretical analyses to explain why the decomposed prompt works effectively?\nQuestion B: The optimization of prompt tuning is typically challenging. Could you elaborate on how your approach addresses this issue?", "paper_topic_and_main_contributions": "The paper introduces an innovative method that employs low-rank matrices to formulate the soft prompt, which can enhance the efficiency of prompt tuning. This approach not only improves the performance but also reduces the training parameters.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "The authors have appropriately and comprehensively included relevant references in the bibliography.", "Typos_Grammar_Style_and_Presentation_Improvements": "The paper is well-written with no discernible typographical or grammatical errors. The style and presentation of the research are clear and easy to follow."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a novel method, "Decomposed Prompt Tuning via Low-Rank Reparameterization," aiming to enhance prompt tuning efficiency for large pre-trained language models. By decomposing the soft prompt into low-rank matrices, the proposed approach reduces trainable parameters in the prompt matrix. The approach is experimentally validated using the SuperGLUE benchmark across various-sized T5 models.

Generally, the reviewers responded positively to the parameter reduction achieved by the method while maintaining similar or better performance. Additionally, the use of low-rank reparameterization seems well motivated in this domain.

There were concerns raised about the lack of comparison to other adapter-based approaches like LoRA. In the rebuttal, the authors argue that these approaches are less parameter efficient than prompt tuning. However, this argument feels a bit weak to me for multiple reasons. First, the authors compare against Fine-Tuning, which requires training the entire model (many more parameters than a LoRA). Second, and more importantly in my mind, the motivation to compare different parameter efficient fine-tuning methods is to see the trade-off across the additional parameters, computational cost, and performance of each method.

For instance, while LoRAs may require more parameters, all of these methods likely contribute to a fraction of the storage and computation requirements of the larger LLM they are augmenting. This makes the reduction in memory (even a 10x reduction) harder to quantify without additional results to better contextualize it. The authors may include some additional empirical observations (and comparison to other existing methods) along these lines to better show how this work is positioned in this space.
Decision: Accept-Findings