DePT: Decomposed Prompt Tuning for Parameter-Efficient Fine-tuning
Authors: Zhengxiang Shi
Aldo Lipani
Venue: ICLR 2024 poster (2024)
Keywords: Natural Language Processing
Large Language Models
Parameter-efficient Fine-tuning
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: We introduce "Decomposed Prompt Tuning" as an improvement over the Prompt Tuning method. Our approach substantially reduces computational demands, both in terms of time and memory, while delivering better performance.

Abstract: Prompt tuning (PT), where a small amount of trainable soft (continuous) prompt vectors is affixed to the input of language models (LM), has shown promising results across various tasks and models for parameter-efficient fine-tuning (PEFT). PT stands out from other PEFT approaches because it maintains competitive performance with fewer trainable parameters and does not drastically scale up its parameters as the model size expands. However, PT introduces additional soft prompt tokens, leading to longer input sequences, which significantly impacts training and inference time and memory usage due to the Transformer's quadratic complexity. Particularly concerning for Large Language Models (LLMs) that face heavy daily querying. To address this issue, we propose Decomposed Prompt Tuning (DePT), which decomposes the soft prompt into a shorter soft prompt and a pair of low-rank matrices that are then optimised with two different learning rates. This allows DePT to achieve better performance while saving substantial memory and time costs compared to vanilla PT and its variants, without changing trainable parameter sizes. Through extensive experiments on 23 natural language processing (NLP) and vision-language (VL) tasks, we demonstrate that DePT outperforms state-of-the-art PEFT approaches, including the full fine-tuning baseline, in some scenarios. Additionally, we empirically show that DEPT grows more efficient as the model size increases. Our further study reveals that DePT integrates seamlessly with parameter-efficient transfer learning in the few-shot learning setting and highlights its adaptability to various model architectures and sizes.

Published as a conference paper at ICLR 2024
DEPT: DECOMPOSED PROMPT PARAMETER-BFFICIER FINE-TUNING
TUNING
FOR
Zhengxiang Shi, Aldo Lipani
University College London, United Kingdom {zhengxiang. shi. 19, aldo. 1ipanipppni.... (eaapannnuucl... ac. uk
https: com/ Zhengxlaanghh//Der
ABSTRACT
Prompt tuning (PT), where a small amount of trainable soft (continuous) prompt vectors is affixed to the model input, has shown promising results across vari- ous tasks and model architecture parameter-efficien fine-tuning (PEFT), PT stands out from other PEFT approaches because it maintains competitive perfor- mance with fewer trainable parameters and does not drastically scale up param- eters as the model size expands. However, PT introduces extra soft prompt tokens, leading to longer input sequences, which significantly impacts training/inference time and memory usage due to the Transformer's 's quadratic complexity. Particu- larly concerning for Large Language Models (LLMs) that face heavy daily query- ing. To address this issue, propose Decomposed Prompt Tuning (DEPT), which decomposes the soft prompt into shorter soft prompt and pair of low- rank matrices that are then optimised with two different learning rates. This al- lows DEPT to achieve better performance while saving substantial memory and time costs compared vanilla PT and its variants, without changing trainable pa- rameter sizes. Through extensive experiments on 23 natural language processing (NLP) and lision-language (VL) tasks, we demonstrate that DEPT outperforms state-of-the-art PEFT approaches, including the full fine-tuning baseline, in some scenarios. Additionally, we empirically show that DEPT grows more efficient as the model size increases. Our further study reveals that DEPT integrates seam- lessly with parameter-efficies transfer learning in the few-shot learning setting and highlights its adaptability various model architectures and sizes.
INTRODUCTION
Imput
Text Prompt
Solt
Promot
Erbedding
Fine-tuning (FT) language models (LMs) (Raffel et al., 2020; Touvron et al., 2023) downstream tasks of- fers large performance improvements various natural language pro- cessing (NLP) tasks, but it requires updating and storing full parameters of the LMs (see Figure la), which is especially expensive when LMs con- tain hundreds of millions or even bil- lions of parameters. Prompt engineering it is typically hard to design and has ure lc). Consequently, parameter-efficien fine-tuning (PEFT) approaches (Liu et al., 2022) have attracted growing interest, aiming to learn only small number of parameters per task while main- taining performance levels comparable to full fine-tuning.
Pre-trained Model A
Pre-trained Model Frozen
Pre- tra-trained Model Frazon
Trainsblee
(a) Fine Tuning
Prompt Tuning
(c) Prompt Engineering
Figure 1:
The overview of Fine Tuning (FT), Prompt Tun-
ing (PT), and Prompting Engineering. PT increases the length of the input sequence, leading much greater com- putational demands during train and inference phrases.
(Brown et al., 2020) does not update any parameters while reg-operformance variance (Wang et al., 2023a) (see Fig-
Prompt Tuning (PT) (Lester et al., 2021) has emerged as promising PEFT approach, which ap- pends trainable continuous prompt vectors to the input (see Figure 1b). PT stands out from other PEFT approaches it maintains competitive performance with fewer trainable parameters and does not drastically scale up its trainable parameters as the model size expands. Recent works suggest that the majority of the LM's knowledge is acquired during its pretraining phase (Zhou et al., 2023), and

---

Published as a conference paper at ICLR 2024
that in-context learning (ICL) with just a few carefully designed stylistic examples and carefully designed system prompt can achieve impressive aligmment results (Lin et al., 2023). Considering scenarios where tasks have already been somewhat understood by LMs and the key challenge is just to properly prompt the LMs, PT emerges as a potentially better option to other PEFT approaches.
While PT has shown promising results across various tasks and models, it has major limitations: (1) PT often suffers from slow convergence and is sensitive to the initialization (Lester et al., 2021; Vu al., 2022; Wang et al., 2023b); and (2) PT extends the total length of the input sequence, consequently exacerbating the computation demand (i.e. train/inference time and memory cost), due to the quadratic complexity of the Transformer (Vaswani et al., 2017). This further accentuated given the slow convergence issue. Recent studies (Su et al., 2022: Vu et al., 2022; Li et al., 2022) have proposed the variants of the vanilla PT to tackle the first issue by initially pre-training soft Transfer Learning
prompts on variety of source tasks, which is known (PETL), as depicted in Figure 2a. Some studies (Asai al., 2022; Wang et al., 2023b) also improve the performance of the PT by jointly training learned prompts from these source tasks on multiple target tasks (referred to as Multi-task Learning). However, the issue of increased computational load due to the extension of sequence length remains largely unaddressed. While PETL approaches can reduce the training steps for model convergence, each optimization step remains computationally expensive in terms of time and memory. Most importantly, it does not enhance the efficiency during the inference phase, which is particularly crucial in the era of Large Language Models (LLMs), considering that the trained models may be queried millions of times per day.
Parameter-F.fiicee
In this work, we propose Decomposed Prompt Tuning (DEPT), which decomposes train- able soft prompt into a shorter soft prompt and a couple of low-rank matrices, where the multiplication of low-rank matrices is then added element-wise to frozen word em- beddings, shown in Figure 2b ($2.2). This shorter soft prompt and the updated word embedding matrix are then optimised using two different learning rates a crucial step for model convergence ($3.4). The intuition of this design is to enable representation updates within the frozen word embedding, thereby increasing the adaptability of input representations that were previously unavailable. Experimental results on 23 natural language processing (NLP) and vision-language (VL) tasks demonstrate DEPT outper- forms the state-of-the-art PEFT approaches, including the full fine-tuning baseline in certain scenarios ($3.2). Our study empirically shows that DEPT largely improves the training efficiency across various model architectures and sizes, saving more than 20% (using T5-BASE) in both training time and memory costs compared to the vanilla PT. Importantly. DEPT becomes increasingly efficient as the model size grows, making it particularly advantageous and suitable for LLMs ($3.3). Furthermore, our additional analysis in the few-shot learning setting reveals the DEPT's compatibility with PETL approaches ($3.4).
Trandfor Looring
Multi-Ta Learning
Tesk
Source
Tock
Prompts
Intisicatio
Dorgrt Tuck 21
Pre-trained Model
Pre-trained Model
(o) Parameter-efficiiooo Transfer Learning Framework
Decompose the Soft Prompt
Target Task
Equiedient
Updatt Frozen Word Embedsingg
Pre-trained Model
Low-ain Harisce
Floan Frcul Frest Crolading
(b) Decomposed Prompt Tuning (DePT)
The overview of the PETL framework (Top) and our method DEPT (Bottom). DEPT decomposes trainable soft prompt of the vanilla PT into a shorter soft prompt and couple of low-rank matrices, where the multiplication of low-rank matrices serves to update frozen word embedding.
Figure 2:
In summary, the main contributions of this paper are as follows:
We propose DEPT method, which addresses key efficiency limitation of Prompt Tuning by decomposing its soft prompt to reduce input sequence length. DEPT largely improves the training and inference efficiency, in terms of both time and memory costs;
Our comprehensive evaluation on 23 NLP and VL tasks demonstrates that DEPT outper- forms state-of-the-art PEFT approaches, including the full fine-tuning in some scenarios.
2

---

Published as a conference paper at ICLR 2024
Additionally, our experiments show that DEPT smoothly integrates with PETL approaches and the advantage of DEPT persists in the few-shot learning setting;
We empirically show that DEPT becomes increasingly efficient the model size grows, making particularly well-suited for LLMs. Furthermore, DEPT is orthogonal to various PEFT approaches (i.e., Adapter, LoRA) and can be easily combined together.
2
METHOD
In this section, we first revisit background of Prompt Tuning (PT) in $2.1 and then introduce our proposed method, Decomposed Prompt Tuning (DEPT) in $2.2.
2.1 BACKGROUND: PROMPT TUNING (PT)
Let L A {æi, Ji denote N labelled training data for the target task T. Given backbone model parameterised by θ each input text æ; is mapped into a sequence of word embeddings W; R where 8 and d represent the maximum sequence length and the dimension of word embeddings. PT appends trainable prompt matrix P R to the frozen word embedding matrix W,, where / is a hyper-parameter for the number of virtual tokens. The soft prompt P can be initialised either randomly or by sampling word embeddings from the vocabulary. Consequently, the model's input becomes the combined matrix [P; W.] (R(++))dd The targeted loss function formulated as:
CPT
> log P [[P, :))
where the loss function is only optimised with respect to the soft prompt matrix P.
2.2 OUR APPROACH: DECOMPOSED PROMPT TUNING (DEPT)
The decomposition of the soft prompt. DEPT differs from the vanilla PT method in the aspect of inputs. As shown in Figure 2b, we decompose a trainable prompt matrix P E R from the vanilla PT into two components: (1) a shorter trainable prompt matrix P. E R.mxd and (2) pair of low- rank matrices, A Rax and B E R'xd, where typically the rank of the matrices r & min(s, d). The first component, the smaller trainable prompt matrix, is appended to the word embedding matrix in similar manner in the vanilla PT. The second component uses the multiplication of two low- rank matrices to represent the update of the word embedding through a coordinate- wise sum:
(2) where W; is frozen and does not receive gradient updates during the training, whereas A and B are trainable. Following Hu et al. (2021), we random Gaussian initialization for A and zero for B, so AW- BA zero when the training starts. The loss function is then optimised as follows:
W W; AW, W, BB Reedd
LDAPT = - Σ log P(y: [P,, ::
In our experiment, choose the values of m and satisfy the equation Ix m mx (s+++++++))) xr for maintaining the exact size of trainable parameters as in the vanilla PT. Consequently, m is always less than when > This design improves memory efficiency and reduces computational expense compared to the vanilla PT, as the shorter input sequence length (i.e., m + < + s) substantially reduces computation due the quadratic complexity of the Transformer (Vaswani al., 2017).
Two rates of learning. DEPT also differs from the vanilla PT training. We train the shorter trainable prompt matrix, P., with the learning rate an and the pair of low-rank matrices, A and B, with the learning rate a2, rather than use single learning rate as in the vanilla PT. The a is typically much larger than the a2. We will empirically validate the importance of this choice in $3.4. However, DEPT may introduces extra training costs for the hyperparameter optimization (see $5).
3 EXPERIMENTS AND RESULTS
In this section, we introduce our experimental setup (see $3.1), evaluate the performance of DEPT across 23 different NLP and VL tasks (see $3.2), and assess relative train/ifference time and mem- ory cost of DEPT (see $3.3), and explore the effectiveness DEPT in the few-shot learning setting and importance of two different learning rates for training DEPT (see $3.4).
3

---

Published as a conference paper at ICLR 2024
3.1 EXPERIMENTAL SETUP
Datasets and tasks. We evaluate our proposed method DEPT on 21 NLP tasks and 2 vision- language tasks. For NLP tasks, follow the previous works (Vu et al., 2022; Sung al., 2022b; Asai et al., 2022; Wang et al., 2023b) and use various datasets sourced from: (1) GLUE (Wang et al.. 2018) benchmark, including MNLI (Williams et al., 2018), QQP'. QNLI (Rajpurkar et al., 2016), SST-2 (Socher et al., 2013), STS-B (Cer et al., 2017), MRPC (Dolan & Brockett, 2005), RTE (Giampiccolo et al., 2007) and CoLA (Warstadt et al., 2019); (2) SuperGL.UE benchmark (Wang et al., 2019), including MultiRC (Khashabi et al., 2018), BoolQ (Clark et al., 2019), WiC (Pilehvar & Camacho-Collados, 2019), WSC (Levesque et al., 2012), and CB (De Marneffe et al., 2019); (3) MRQA 2019 Shared Task (Fisch et al., 2019), including Natural Questions (Kwiatkowski et al., 2019), HotpotQA (Yang et al., 2018), SearchQA (Dunn et al., 2017) and NewsQA (Trischler et al., 2017); (4) other datasets, including WinoGrande (Sakaguchi et al., 2021), Yelp-2 (Zhang et al., 2015), SciTail (Khot et al., 2018) and PAWS-Wiki (Zhang et al., 2019). For vision- Saiionla tasks, follow prior works (Sung al., 2022a;b) experiment with the visual question-answerii task, VQA (Goyal et al., 2017), and the image caption generation task, MSCOCÓ (Chen et al., 2015).
Baselines. We compare DEPT with a variety of baselines: (1) fine-tuning (FT), where all the model parameters are tuned during adaptation on each downstream task; (2) the vanilla PT (Lester et al., 2021), where target prompt vectors are initialized by randomly sampled top vocabularies, and its variants using additional transfer and multi-task learning, including SPoT (Vu et al., 2022), AT- TEMPT (Asai et al., 2022), and MPT (Wang et al., 2023b); (3) state-of-the-an PEFT approaches including Adapters (Houlsby et al., 2019), AdapterDroop (Rücklé et al., 2021), BitFit (Ben Zaken et al., 2022), HyperFomer (Karimi Mahabadi et al., 2021), HyperDecoder (Ivison Peters, 2022), P-tuning (Liu et al., 2021), LoRA (Hu et al., 2021), LST (Sung et al., 2022b), and their multi-task learning variants. For a fair comparison, we directly quote performance metrics from published pa- pers (Mahabadi et al., 2021; Karimi Mahabadi et al., 2021; Asai et al., 2022: Wang et al., 2023b; Sung et al., 2022b) for a fair comparison, where all these baselines using the T5-BASt as the back- bone and adhere to the train, validation and test splits used by Karimi Mahabadi et al. (2021); Mahabadi et al. (2021) for NLP tasks and by Sung et al. (2022b) for vision-language tasks.
Implementation details. In our study, we mainly experiment using the T5-BASE model with 220M parameters (Raffel et al., 2020). We consistently set the number of virtual tokens / as 100 across all tasks for the vanilla PT and adjust the hyper-parameters of DEPT accordingly to maintain the equivalent number of trainable parameters. For instance, the vanilla PT contains x d trainable parameters where the hidden size d is 768 for the T5-BASE, and DEPT can configure the number of virtual tokens m as 40 and the rank of low matrices r as 45, resulting in m x d (s + d) x trainable parameters. This yields a total of 76, 800 trainable parameters, aligning with the vanilla PT. For VL tasks, we utilise the CLIP-T5 architecture which combines CLIP (Radford et al., 2021) and T5- BASE (Raffel et al., 2020), with the CLIP frozen. We follow the prior work (Sung et al., 2022b) to concatenate the visual representation from CLIP with the text embedding from the T5-BASE, where a trainable visual projection layer is used between CLIP and T5 to align the visual representation to the same dimension as the text embedding.
We also extend our evaluation to include T5-SMALL (60M), T5-LARGE (770M), GPT2-SMALL (110M),
(345M), and GPT2-LARGE (774M) models. In the few-shot experiments, we randomly select & examples three times from the training set and report the mean and standard deviations for each k-shot experiment. Following the prior works in PETL for PT (Vu et al., 2022; Su et al., 2022; Asai et al., 2022), we use MNLI, QQP, SST-2, SQUAD (Rajpurkar et al., 2016), and ReCoRD (Zhang et al., 2018) as five source tasks. Our soft prompt and low-rank matrix pairs are initialized from the soft prompts derived from one of these selected source tasks, Please see more hyper-parameter and implementation details in Appendix SD.
3.2
MAIN RESULTS
This section shows the empirical evidence supporting the effectiveness of our proposed method DEPT across 23 NLP and VL. tasks. Table 1, 2, and 3 present our experimental results on GLUE and SuperGLUEE benchmarks, MRQA 2019 Shared Task and four other NLP datasets, as well as two VL tasks. Additionally, we visualise the model performance against the number of trainable
'https:/ httrpp://www... http///ww...a.aaa//////////// wtt///////////cte http///qqooraaaaaaa/o
4

---

Published as a conference paper at ICLR 2024
Table 1: Test results on GLUE and SuperGLUU benchmarks, with the corresponding size of train- able parameters, All of the results are based on T5-BASE models. We use Pearson correlation for STS-B, Fl for MultiRC (Multi), and accuracy for other tasks as evaluation metrics.
GLUE
SuperGLUE
Method
#Para
MNII QQP QNLI SST-2 STS-B MRPC RTE CoI.A Mean Multi Bool WIC WSC CB Mean
Single-Tase Task Learning
Fine-tuning!
220M
86.8
91.6
93.0
94.6
89.7
90.2
71.9
61.8
84.9
72.8
81.1
70.2
59.6
85.7
73.9
Adapter'
1.M
86.5
90.2
93.2
93.8
90.7
85.3
71.9
64.0
84.5
75.9
82.5
67.1
67.3
85.7
75.7
AropierDrop'
1.1M
86.3
90.2
93.2
93.6
91.4
86.3
71.2
62.7
84.4
72.9
82.3
68.3
67.3
85.7
75.3
Bitlit
280k
85,3
90.1
93.0
94,2
90.9
86.8
67.6
58.2
83,3
74.5
79.6
70.0
59.6
78.6
72.5
LORA"
3.8M
86.3
89.0
93.2
94.3
90.9
90.1
75.5
63.3
85.3
72.6
81.3
68.3
67.3
92.9
76.5
LST
3.8M
85.6
88.8
93.3
94.1
90.7
90.4
71.9
58.1
84.1
PT
76.8k
83.4
90.2
93.1
91.9
90.2
90.1
78.8
60.7
84.8
65.7
63.7
50.8
51,9
67.9
60.0
DEPT (ours)
76.8k
85.0
90.4
93.2
94.2
90.8
90.7
79.1
63.8
85.9
74.3
79.3
68.7
67.3
92.9
76.5
Mulli-task Learning
Fine-tuning (m:)
28M
85.7
91.1
92.0
92.5
88.8
90.2
75.4
54.9
83.8
74.4
81.1
70.0
71.
85.7
76.1
Adupter (m)
18M
86.3
90.5
93.2
93.0
89.9
90.2
70.3
61.5
84.4
72.6
82.3
66.5
67.3
89.3
75.6
HyperFormer (m) 1
638k
85.7
90.0
93.0
94.0
89.7
87.2
75.4 63.7
84.8
72.9
82.5
69.0
67.3
85.7
75.4
Hepoder (m)
1.8M
86.0
90.5
93.4
94.0
90.5
87.7
71.7 55.9
83.7
70.4
78.8
67.1
61.5
82.1
72.0
Single-Tash Task Training + Transfer Learning
SPoT'
76.8k
85.4
90.1
93.0
93.4
90.0
79.7
69.8
57.1
82.3
74.0
77.2
67.0
50.0
46.4
62.9
ATTEMPT!
232k
84.3
90.3
93.0
93.2
89.7
85.7
73.4
57.4
83.4
74.4
78.8
66.8
53.8
78.6
70.5
MPT
77.6k
85.9
90.3
93.1
93.8
90.4
89.1
79.4
62.4
85.6
74.8
79.6
69.0
67.3
79.8
74.1
Multi-task Learning Transfer Learning
ATTEMPT (m)
96k'
83.8 90.0
93.1
93.7
90.8
86.1
79.9
64.3
85.2
74.4
78.5
66.5
69.2 82.1 74.1
MPT (m)
10.5k'
84.3
90.0 93.0
93.3
90.4
89.2
82.7
63.5
85.8
74.8
79.2 70.2 67.3 89.3 76.1
sourced from Asai et al. (2022). sourced from Sung et al. (20226). sourced from Wang et al. (2023b). 4 we reproduce and substantially increase the performance of the vanilla PT reported the prior work (Asai et al., 2022). Thhsee values are obtained after amortizing over tasks, and the minimal number paramcters perform single task remains 232k and 77.6k for ATTEMPT and MPT. (m) represents additional multi-task training.
parameters for GLUE and SuperGLUE in Figure 6 of Appendix $A. Furthermore, we evaluate the performance of DEPT using 2LAMA-2 (Touvron et al., 2023) in Appendix $B. Experimental results reveal three key findings: (1) DEPT consistently outperforms the vanilla PT and its PETL variants; (2) DRPT achieves competitive even better performance than state-of-the-art PEFT approaches while using fewer trainable parameters; and (3) DEPT falls short in some certain tasks. Below we delve deeper with respect to various tasks.
#1 Performance on GLUE and SuperGLUE benchmarks. As shown in Table 1, our experi- mental result indicates that DEPT outperforms state-of-the-art PEFT approaches, such as Adapter, LoRA and LST on the GLUE and SuperGI.UE benchmarks, while using fewer trainable parameters. Remarkably, DEPT also outperforms the full fine-tuning baseline on both benchmarks. In addition, DEPT outperforms vanilla PT and all the variants PT that introduce additional transfer learning and multi-task learning. For example, ATTEMPT, which requires additional training for the soft prompt on the source tasks, achieves an average score of 83.4 on the GLUE benchmark and 70.5 on the SuperGLUE benchmark. Mcanwhile, DEPT outperforms ATTEMPT with scores of 85.9 and 76.5 on GLUE and SuperGI.UF, despite training fewer parameters. Similarly, DEPT surpasses MPT with 0.1 11% the GLUE benchmark and 0.4% on the SuperGI.UE benchmark, without utiliz- ing additional transfer learning or multi-task learning. These results are achieved with less inference time and reduced memory resources (refer to $3.3 for specifics), which validates the effectiveness of DEPT. As the PT often underperforms in scenarios with limited labelled data (Gu et al., 2022), we investigate the compatibility of DEPT and PETL later in the few-shot learning setting ($3.4).
#2. Performance on MRQA 2019 Shared Task and other NLP datasets. Table 2 presents the performance of various PEFT approaches, including DEPT, on the MRQA 2019 Shared Task and four other datasets. We observe that DEPT improves the average performance of the vanilla PT by substantial margin of +3.6% on MRQA and +14.2% on the other datasets. DEPT exceeds the performance of the PT variants that leverage additional transfer and multi-task learning, without introducing extra trainable parameters the vanilla PT or relying on any PETI. approaches. While
5

---

Published as a conference paper at ICLR 2024
Test results on MRQA 2019 Shared Task and other datasets using the T5-BASE model. We report the F1 for MRQA tasks and accuracy for other datasets across three seeds, with standard deviations subscripts. All baseline results are directly quoted from Wang et al. (2023b).
Table 2:
MRQA
Others
Method
iaara
NQ
HP
SQA
News
Mcan
WG
Yelp
SciTail
PAWS
Mean
Fine Tuning...
220M
75.1
77.5
81.1
65.2
74.7
61.9
96.7
95.8
94.1
87.1
Adapters
1.9M
74.2
77.6
81.4
65.6
74.7
59.2
96.9
94.5
94.3
86.2
BitFit
280K
70.7
75.5
77.7
64.1
72.0
57.2
94.7
94.7
92.0
84.7
LoRA
3.8M
72.4
62.3
72.5
56.9
66.0
58.2
97.1
94.7
94.0
86.0
PT
76.8K
67.9
72.9
75.7
61.1
69.4
49.6
95.1
87.9
55.8
72.1
SPoT
76.8K
68.2
74.8
75.3
58.2
69.1
50.4
95.4
91.2
91.1
82.0
ATTEMPT
232K
70.4
75.2
77.3
62.8
71.4
57.6
96.7
93.1
92.1
84.9
MPT
77.6K
72.0p.1
75.80.1
77.2a.1
63.7o.1
72.2
56.5c.9
96.4a.0
95.50.1
93.
85.5
DEPT (ours)
76.8K
73.20.1
76.80.3
77.60.2
64.4p.1
73.0
59.0g.a
96.8o.1
95.60.2
93.76.1
86.3
DEPT improves over the vanilla PT and its variants are promising, there remains a disparity in performance when compared to the full fine-tuning baseline. Investigating ways to incorporate DEPT with other PEFT methods, such as LoRA and Adapter, may provide a valuable direction for future research towards narrowing this performance gap.
Table 3: Test results on the VQA and MSCOCO dataset using T5-BASE model. We report average results across three seeds, with standard deviations in subscripts. All baseline results are directly quoted from Sung et al. (2022b). The best performance for each column is highlighted in blue.
#3. Performance on Vision-Language tasks. Ta- ble provides an overview of the performance of various PEFT approaches on two VL tasks, specifi- cally VQA and MS COCO Caption Generation. Re- sults show that DEPT, while updating much fewer parameters, achieves a CIDEr score of 113.7 on the MS COCO Caption Generation task, outperform- ing state-of-the-art PEFT approaches. This suggests the effectiveness of our proposed method. However, while DEPT outperforms methods such as P-tuning and BitFit on the VQA dataset, it still falls short of the full fine-tuning performance. This suggests that in certain tasks, the use of greater number of train- able parameters could be beneficial.
Updated Params (%)
VOA Karpathy test Acc. (%)
MSCOCO Karpathy test CIDEr
Method
FT
100
67.1o..
112.2a.3
Adapters
7,98
67.1o.1
111.8.1
LoRA
7.54
63.7a2
110.36.4
BidFi
0.83
55.1o2
101.26.2
P-Tuning
1.26
47.40.
96.1g.
LST
7.46
66.50.1
113.50.3
DEPT (ours)
0.74
59.80.4
113.76.a
3.3 TIME AND MEMORY EFFICIENCY
This section shows the empirical evidence supporting the efficiency of DEPT, spanning over diverse model architectures of varying scales on the GLUE benchmark. To ensure fair comparison, we consistently keep the number of trainable parameters in DEPT the same as that in the vanilla PT (1= 100). As a result, once we choose the length of the soft prompt m in DEPT, the rank of the low-rank matrices r becomes determined. In our experiments, we primarily compare DEPT with the vanilla PT using 5 different lengths of soft prompt m (i.e., 0, 20, 40, 60, 80). Figure 3 and 4 depict the average GLUE performance of DEPT, along with the associated training/inference time and memory cost compared to the vanilla PT. Below we discuss two key findings.
# 1. DEPT improves time and memory efficiency substantially. Figure 3 presents the mean performance of DEPT, associated with average training time and memory, on the GLUE bench- marks, against different lengths of soft prompt m. The average training time and memory costs are computed across 8 tasks on the GLUE benchmark and three different model sizes. Both the encoder-decoder (T5) and decoder-only (GPT-2) models are evaluated across three different model sizes. The study reveals that decomposing the soft prompt (1 = 100) into a small soft prompt and low-rank matrices delivers comparable or even better performance while substantially enhancing the efficiency of training and reducing memory utilization. Specifically, using a soft prompt length greater than 20 in DEPT with the T5 model leads to a better average performance on the GLUE benchmark to vanilla PT, while improving the efficiency of training and reducing memory utiliza- tion by approximately 25%. This improvement is more pronounced (37% on the SST-2 dataset) when test DEPT (with m 60) using the T5-3B model (see $B for details). Similar observa- tions are also found when the GPT model is used, suggesting the adaptability of DEPT for different model architectures. It is worth noting that DEPT may have notable performance drop regardless

---

Published as a conference paper at ICLR 2024
Model
GPT-2 Model
10
Relabive Train Time/MMmary Cost (862
GLUE Performance (16)
Train limm/Mamary
Performance
100
20
46
60
Length Prompt, m
Longth Soft Prompt,
T5-Small
T3-Base
T5- T5-Large GargeLarge
Memory Cost
GPT2-Sm Small
GP12 Medium
Time
Figure 3: Performance the GLUE benchmark for different soft prompt lengths m in DEPT, associated with corresponding relative train time and memory cost. The time and memory are aver- aged over different model sizes using batch size as 16. DEPT consistently uses the same number of trainable parameters as the vanilla PT (m=100).
T5-Small (60M)
TS.Base (220M)
T5-Large (770M)
1.00)
1.00
1.00
1.01
1.04
1.05
1.04
1:11
1.14
1.22
165
175
70
76
21
24
Per Second
Inference Samples Per Second
Figure 4: Average inference speed on GLUE benchmark using varying soft prompt length and rank low-rank matrices r, keeping the total number of trainable parameters constant. Small texts in blue indicate the speed relative to the vanilla PT (represented by brown) (m==100).
of using T5 or GPT-2, when the soft prompt is eliminated (m = 0) and the model solely depends on the pair of low-rank matrices.
# 2. DEPT grows more efficient the model size increases. Figure 4 represents the inference speed, measured by the average number of samples evaluated per second on the GLUE benchmark using a single RTX 3090 GPU. The inference time is computed using the Huggingface Trainer Class. We observe that the relative improvement in the number of inference samples per second over vanilla PT grows the model size increases. For example, when using the T5-SMALL model, the vanilla PT evaluates 167.3 samples per second, while DEPT (m 20) evaluates 178.3 samples per second, resulting in 6.5% boost inference speed. In contrast, when the T5-LARGE is utilized, the vanilla PT evaluates 21.0 samples per second and DEPT (m = 20) evaluates 24.8 samples per second, resulting in an 18. % increase inference speed, a substantial rise from the previous 6.5%. This indicates that DEPT is particularly beneficial and more applicable in the context of LLMs. Please refer to Appendix $B for the inference speed of DEPT and PT using T5-3B and LLAMA-2.
3.4 FURTHER ANALYSIS
Few-shot Learning. The vanilla PT often underperforms the few-shot learning tasks (Gu al., 2022) due to the first limitation discussed in $1. To evaluate the performance of DEPT in the few-shot setting. we employ the transfer learning method inspired by the recent PETL studies, as illustrated in Figure 2a. Specifically, we pre-train both the soft prompt and the low-rank pair on source tasks and select the best checkpoint before proceeding with the target task. Following prior works (Karimi Mahabadi al., 2021; Asai et al., 2022; Wang al., 2023b), effectiveness D&PT across 14 NLP tasks, with training examples where k = 4, 16, 32. Our experimental findings reveal two key observations follows: (1) DEPT integrates seamlessly with PETL approaches; and (2) DEPT attains competitive or even better performance than state-of-the- art PEFT approaches in the few-shot learning setting.
evaluate the
Table 4 compares the effectiveness of our proposed method DEPT with various PEFT approaches in few-shot experiments, including full fine-tuning (FT), Adapters (AD), vanilla PT (PT), SPoT (ST),

---

Published as a conference paper at ICLR 2024
Few-shot learning results with {4. 16, 32) on the SuperGI.UE BooQ, SuperGI.UE CB and SciTail datasets. We report average results across three seeds, with standard deviations in subscripts. Baseline results are directly quoted from Wang et al. (2023b). The best performance for each row is highlighted in blue.
Table 4:
k-shot #Para
FT
AD
PT
ST
HIF
(IA)*
ATP
MPT 77.6K
DEPT
Task
220M
1.9M
76.8K
76.8K
638K
55.3K
232K
76.8K
4
50.5
53.4
61.6
50.5
48.0
56.7
61.8
62.2
62.75.4
BoolQ
16
56.5
51.4
61.9
50.6
50.2
62.0
60.0
63.3
66.9..4
32
58.4
54.5
61.7
61.2
58.3
67.2
65.3
68.9
67.25.1
57.7
51.1
53.5
71.4
60.7
65.5
67.9
73.6
75.0s.1
CB
77.0
74,8
63.5
64.3
76.3
71.4
71,4
78.6
78. 78.6
32
80.0
74.8
67.8
64.3
81,4
75.0
78.5
82.1
82.
4
79.6
79.5
57.7
69.6
82.0
65.4
80.2
80.2
78.
SciTail
16
80.0
83.2
60.8
71.9
86.5
74.4
79.5
87.3
78.5
32
81.9
85.0
60.2
71.9
85.8
80.4
80.2
86.3
85.43.1
Table 5: Few-shot learning results with == [4, 16, 32) on GLUE and SuperGLUE benchmarks. We report average results across three seeds, with standard deviations in subscripts. Baseline results are directly quoted from Wang et al. (2023b).
k-shot
Method
GLUE
SuperGLUE
MNLI
QQP
QNII
SST-2
STS-B
MRPC
RTF
Cal.A
Avg.
Multi
BoolQ
WIC
WSC
Avg.
40.4
88.8
27.4
54.7
57.7
MPT
39.4
$2.0
86.2
56.3
89.1
68.1
62.6
34,8
67.3
62.2
62.2
52.9
61.3
73.6
63.6
DIPT
44.0,
77.4g
85.8g4
50.3g,
84.1
73.5gz
63.5gx
29.3g3
64.6
62.3g,
62.764
57.5,3
67.9g
75.0g
65.1
41.5
62.3
59.9
50.9
87.8
68.1
54.7
28.5
56.7
60.3
61.9
48.9
44.2
63.5
558
16
MPT
61.6
$4.7
90.6
63.2
89.1
70,1
64.8
32.1
69.5
64.5
63.3
49.8
67.3
78.6
61.7
DEPT
61.8gs
80.3.
91.2g
77.6,
87.3
78.1a
71.9
27.1.
71.9
60.6
66.94
59.6
57.7g
78.6
64.7
PT
37.0
62.3
56.7
50.9
87.5
68.1
54.7
23.2
55.1
59.2
61.7
52.6
67.3
67.8
61.7
32
MPT
63.6
88.5
91.0
75.9
89.7
74,5
59.7
30.8
71.7
63.3
68.9
53.9
67.3
82.1
67.1
DEPT
91.305
89.201
81,433
73.4
58.0
66.4
HyperFormer (HF), (IA)*, ATTEMPT (ATP), and MPT on BoolQ, CB, and SciTail datasets. Table presents the performance of DEPT against the vanilla PT and MPT on the GLUE and SuperGLUE benchmark. Experimental results show that vanilla PT struggles with few-shot tasks, indicating the importance of PETL for the PT in few-shot learning tasks as suggested in previous works (Vu et al., 2022; Su et al., 2022). Nevertheless, the performance of DEPT largely benefits from the PETL framework (see Figure 2a). For example, while the vanilla PT obtains accuracy of 53.5% on SuperGLUE CB dataset and 57.7% on the SciTail dataset when k=4, DEPT with PETL achieves an accuracy of 75.0% on SuperGLUE CB dataset and 78. % on the SciTail dataset, for the same k value. This result supports our first observation about the compatibility of DEPT and PETL approaches. Furthermore, DEPT with transfer learning achieves comparable performance with the variant of the PT, MPT across 14 NLP tasks. Notably, DEPT surpasses the performance of all other variants of the PT (i.e. SPoT, ATTEMPT) and other PEFT approaches, demonstrating our method's efficacy and endorsing our second observation.
Learsing
The importance of different learning rates. Figure 5 presents the experimental results from different learn- ing rate settings to train the soft prompt and the pair of low-rank matrices as follows: (I) use singular learning rate of 3e-1; (2) use singular learning rate of 5e-4; (3) apply mixed learning rates (with grid search), where the soft prompt is trained with a larger rate and the pair of low-rank matrices is trained with a lower rate. In our ex- periments, the first option obtains an average performance of 40.8 on the GLUE benchmark. The second option exhibits an average performance of 54.7, while the third option demonstrates largely improved average performance of 85.7 on the GLUE bench- mark. This indicates the importance of training DEPT with two different learning rates.
6988
70
80
GLUE
LR=Ie-3
LR=5e-4
Mixed UR
Figure 5: Test results on GLUE bench- mark using T5-BASE showing the im- portance of training DEPT with differ- ent learning rates.
4 RELATED WORKS
Parameter--fficceet Fine-tuning. In contrast to standard fine-tuning and prompt-based fine- tuning (Devlin et al., 2019; Schick & Schütze, 2021; Shi & Lipani, 2023) where full parameters
8

---

Published as a conference paper at ICLR 2024
updated, parameter-efficient fine-tuning (PEFT) approaches have demonstrated remarkable perfor- mance across a wide range of tasks (Wang et al., 2018; Shi et al., 2022; Wu et al., 2023a; Hendriksen et al., 2022; Wu et al., 2023b; Yang ct al., 2023) while updating only a limited number of parame- ters. Adapters (Houlsby et al., 2019). along with its variants, HyperFormer (Karimi Mahabadi et al., 2021) and Compacter (Mahabadi al., 2021), add new trainable modules (adapters) to each trans- former block of the TS model (Raffel et al., 2020). BitFit (Ben Zaken et al., 2022) limits updates only the bias parameters, while this method tends underperform on larger networks (Lialin et al., 2023). Prefix-tuning (Li & Liang, 2021) adds soft prompt, parameterized by a feed-forward network, to the model input. Diff pruning (Guo et al., 2021) learns a sparse update of a neural net- work's weights at the cost of more memory usage. FishMask (Sung et al., 2021) also performs sparse updates, but computationally intensive and inefficient on contemporary deep learning hardware (Lialin et al., 2023). LoRA (Hu et al., 2021; Yang et al., 2024) employs straightforward low-rank matrix decomposition to parameterise the weight update. (IA)² (Liu et al., 2022) scales activations by learned vectors for few-shot learning. LST (Sung et al., 2022b) operates small transformer network on the side of the pre-trained network, aiming to decrease the training memory. Prompt Tuning (PT) (Lester et al., 2021) appends trainable soft prompt the model input embeddings. In comparison to the above-mentioned PEFT approaches, PT uses fewer trainable parameters, which do not proliferate as the model size expands. Mao al. (2022) introduces a method that combines trefix-tuning. Adapters, and LoRA through a gating mechanism. DEPT is also applicable to this method and can be easily integrated with other PEFT approaches.
Transfer Learning for PT. Recent works aim to enhance the performance of PT through PETL. PPT (Gu et al., 2022) strives to improve the performance of PT (Lester et al., 2021) by further pre-training (Gururangan et al., 2020; Shi et al., 2023), which necessitates a set of hand-crafted, task-specific designs and considerable computational cost. Su et al. (2022) improves PT via prompt transfer across different tasks and models. SPoT (Vu et al., 2022) adopts a single prompt, chosen based on a similarity measure at the cost of a massive search. ATTEMPT (Asai et al., 2022) employs an attention mechanism over the source prompts initialize the prompt for target tasks the cost of extra parameters. MPT (Wang et al., 2023b) applies a shared soft prompt across different tasks, while its effectiveness for broad range of source tasks remains untested. We find that PETL for PT (Asai et al., 2022; Wang et al., 2023b) can efficiently accelerate training convergence, and that PETL for PT is more useful for improving the model performance in the few-shot learning setting for PT (Gu et al., 2022; Wu et al., 2022). However, when extensive labelled datasets are available, training PT or DEPT for additional steps typically leads to performance improvements.
EPILOGUE
Conclusion. In this work, we propose Decomposed Prompt Tuning (DEPT), which substantially improves the efficiency of the vanilla PT in terms of time and memory while delivering competitive or even superior performance compared to the state-of-the-an PEFT methods. Remarkably, DEPT efficiency amplifies with increasing model sizes, making it exceptionally apt for LLMs. Our fur- ther analysis shows the compatibility of DEPT with PETL approaches and highlights its versatility across diverse model architectures and scales.
Limitations and Future Work. We outline several limitations in our work: (I) the main limita- tion of DEPT is the introduction of extra hyperparameters for tuning, egg., the learning rate of the low-rank matrices. This might introduce some additional computational overhead during the hy- perparameter optimization phase of model training. In our work, we train DEPT up to 300k steps (in a data-rich setting) following (Vu et al., 2022) with careful search for optimal learning rates, which may increase training costs. However, the number of training steps might be efficiently re- duced by PETL, which plan to investigate in future work. In addition, it is important note that the model training process is a one-time event, while model inference is not. In this context, the efficiency benefits of DEPT become especially valuable; (2) the number of trainable parameters in DEPT depends on the maximum sequence length s. In this work, we have limited our evaluation to tasks with hundreds of input tokens. Future work could explore the performance of DEPT when is extremely large; and (3) our research focuses on leveraging prompting techniques for LMs, where previous studies (Bender & Koller, 2020; Brown et al., 2020; Bender al., 2021) have already addressed concerns and potential hazards linked to LMs.
9

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
The authors express their gratitude to the ICLR reviewers and area chairs for their insightful discus- sions. Zhengxiang is funded by the Research Studentship from University College London (UCL).
REFERENCES
Akari Asai, Mohammadreza Salehi, Matthew Peters, and Hannaneh Hajishirzi. ATTEMPT: multi-task tuning via attentional mixtures of soft prompts. In Proceedings of the 2022 Conference Empirical Methods in Natural Language Processing, pp. 6655-6672. Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. URL https::pp//......... aaclaalclaac www.annoooogyyyoooooooooooo wwg.oooooooooooooooooooooooooooo 2022 emn 1p-ma 446.
Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. BitFit: Simple parameter-efficiet fine- tuning for transformer-based masked language-models. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 1-9, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi:
1/25533/1/2222.a// short.
1. URL https:tttt..... / ac antaaaoohhyyyy.
Emily M. Bender and Alexander Koller. Climbing towards NLU: On meaning, form, and under- standing in the age of data. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 5185-5198, Online, July 2020. Association for Computational Linguistics. doi: 10. wwwaacllminn... 463.
URL attcps:/aclanthhloogg.. .orgg
Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 202 ACM Conference on Fairness, Accountabiliry, and Transparency, FAccT '21, pp. 623, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 4781450383997. doi: 10.1145/ 1442188.344592 URL https: //ts:://///oo/ /ww.. wtg/10.....ooooooo 1145/14442211 3445922.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert- Gretchen Krueger, Tom Hemighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neu- ral Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020. URL
www.:eeeeeddnnn...nnn.eeeeeeee.e neurips. cc.pppcccccee cc/ccc///paeeeee wacp/pppr__fii pdf.
Daniel Cer, Mona Diab, Eneko Agirre, lñigo Lopez-Gazpio, and Lucia Specia. SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 1thh International Workshop on Semantic Evaluation (SemEval-20177 pp. 1-14, Vancouver, Canada, August 2017. Association for Computational Linguisties. doi: 10. 1/57722000 URL https: http///aaaaaatllcoeeeeee wwtp://og/////0// $11-2001
Xinlei Chen, Hao Fang. Tsung- Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv: 5=XXv:1504.00325, 2015. URL /ettts://arxxe org/ abs/1504 00325.
Alexis Chevalier, Alexander Wettig, Anirudh Ajith, and Danqi Chen. Adapting language models to compress contexts. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 3829-3846, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/0023 cmnlp-main.2. 232. URL https: / aclant org/ 22gg22223
232.
Christopher Clark, Kenton Lee, Ming- Wei Chang, Tom Kwiatkowski. Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational
10

---

Published as a conference paper at ICLR 2024
Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 2924-222222 2936, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/ v1/N19- 1300. URL https: httpps/s///aaaahhooogyyoooo -tt.://anthooooo.ooooo///1110000
org/N19-1300
Marie- Carte-Catherine De Marneffe, Mandy Simons, and Judith Tonhauser. The commitmentbank: In- vestigating projection in naturally occurring discourse. In proceedings of Sinn und Bedeutung, volume 23, pp. 107-124, 2019. URL https:/ /tss:///mmmanns eeemmnticssrrchi Tg33GT2M (M/GGTMM/Maa www.fe...fee...mm pdf.
net. /Archiivv
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume J (Long and Short Papers), Minneapolis, Minnesota, 2019. Association for Computational Linguistics. doi: 10.18653/v1/N194 1423. URL https: https::/aclaatthol.... org/N199 1210NN191100 1423.
William B. Dolan and Chris Brockett. Automatically constructing corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (/WP2005), 2005. URL org/
https: /aaalaaannhoooyyyy
Matthew Dunn, Levent Sagun, Mike Higgins, V Ugur Guney, Volkan Cirik, and Kyunghyun Cho. Searchqa: A new q&a dataset augmented with context from search engine. arXiv preprint arXiv: 1704.55704 2017. URL https: /tattps://rxiie //axivv..ivv////a//aa/eaaoaiivva org/ abs 1 70 704. 05179.
Adam Fisch, Alon Talmor, Robin Jia, Minjoon Seo, Eunsol Choi, and Danqi Chen. MRQA 2019 shared task: Evaluating generalization in reading comprehension. In Proceedings of the 2nd Workshop on Machine Reading for Question Answering, pp. 1-13, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 1/11855/1/DD11-5801. URL https: / //aclanttoo/ //aclanthology.... httpllloyyorrg/gggg0g wttp/oggg//a/eeee 5801.
Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. The third recognizing textual entailment challenge. In Proceedings of the ACI-PASCAL Workshop on Textual Entail- ment and Paraphrasing, pp. 1-9, Prague, June 2007. Association for Computational Linguistics. org/ 200p//07-14111
URL httph:tp////m http:///aaaaaaahaooa
Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answer- ing. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6904 6913, 2017. URL https: httpttt:////eaacccccccc cpwnaccess..ceeeeeeeeceecc...ssc thecvf. conm/contentccpr_0000 html/ Makinn S______VV_______
.tmll.
Yuxian Gu, Xu Han, Zhiyuan Liu, and Minlie Huang. PPT: Pre-trained prompt tuning for few- shot learning. In Proceedings of the 60th Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Papers), pp. 8410 8423, Dublin, Ireland, May 2022. As- sociation for Computational Linguistics. doi: 10.18653/v1/2022.com aol.ooo..... URL https: / //aclanthology owg.//////222...... acl acl-long. 576.
Demi Guo, Alexander Rush, and Yoon Kim. Parameter-efficien transfer learning with diff prun- ing. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguis- ties and the 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pp. 4884-4896, Online, August 2021. Association for Computational Linguis- tics. doi: 18.116533v11222 1/002/1112222//2 long.338. acl-iong.37g
URL https: / aacllaaaa wwwannthllooooooo 202/1021.
Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. Don't t stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguisties, pp. 8362, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v/ 2020.acl-ma 1220.acl-main.225 740. URÍ. httpt::t///........m..t //ce:///clll http///caaaaaateaccceeeeeea wttp:/20....l-maaa....... aww.0.aclmmmmiimm 740.
Mariya Hendriksen, Maurits Bleeker, Svitlana Vakulenko, Nanne van Noord, Ernst Kuiper, and Maarten de Rijke. Extending clip for category-to-image retrieval in e-commerce. In Advances
11

---

Published as a conference paper at ICLR 2024
in Information Retrieval: 44th European Conference on IR Research, ECIR 2022, Stavanger, Norway, April 10-14, 2022, Proceedings, Part I. Berlin, Heidelberg, 2022. ISBN 978-3- 030-99735-9. doi: 0-1099778---00-977366--- 20. URL httpp:///ee..org/1..... 2_.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-effiient transfer learn- ing for nip. In International Conference on Machine Learning, pp. 2790-2799, 2019. URI. http:/ /ttp///oceeecceeeoe ngs. www. ww.........r......... wa.pprss//9/ / sby1 9a/houlll sby1 9a pdf..
Edward Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In International Confer- ence on Learning Representations, 2021. URL https:://///..... /ttps////ppeeviiie ew. net/ / forum? ?id= nžeVKeeFYf9.
Hamish Ivison and Matthew Peters. Hyperdecoders: Instance-specific decoders for multi-task NLP. In Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 1715-1730, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. URL
https: / //aaaaaanthhoooo wwwp:lloy...gg///ooooo 2022 .tri22...iiie findingsnnennnne
124.
Rabeeh Karimi Mahabadi, Sebastian Ruder, Mostafa Dehghami, and James Henderson. Parameter- efficient multi-task fine-tuning for transformers via shared hypernetookss. In Proceedings of the 59th Annual Meeting the Association for Computational Linguisties and the I 1thh International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 565-576, On- line, August 2021. Association for Computational Linguistics. doi: (0.18653/v1/2021 htt.//2221....llon.aa Lacl-long- 47. URL https: : /tt///aalaaa aataaacaannhhooooooooooao
org/ / 20200021.2200 wew.aaccl-........ aal..cclllon...... 47.
Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. Looking be- yond the surface: challenge set for reading comprehension over multiple sentences. In Proceed- ings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers). pp. 252-262, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10. 10.18653/v1/N18. 1023. URL https: / aa celanhhologyyy.. org/No8- 1023.
Tushar Khot, Ashish Sabharwal, and Peter Clark. Scitail: A textual entailment dataset from sci- ence question answering. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1), Apr. 2018. doi: 10.1609/aaii.v322 12322.1112222 URL https: www.oissaaaa.................... aaai. org/ http/nidxxx//pe/ /php/ CAAT/articl//viie
Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sabharwal. Decomposed prompting: modular approach for solving complex tasks. In The Eleventh International Conference Learning Representations, 2022.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wai Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoe Le, and Slav Petrov. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguisties, 7:452-466, 2019, doi: 10.1162/tacla.00226 URL https: hcw.a/haonattcoooc.ooooooo org/ Q19-1026.
Brian Lester, Rami A1-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Lan- guage Processing, pp. 3045-3059, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. 1265:///2221..mmpmmmaaa https: http://////caaaahhhaacccc org/ 2021 emnlp-ma ..
URL
Hector Levesque, Ernest Davis, and Leora Morgenstern. The winograd schema challenge. Thir- teenth International Conference on the Principles of Knowledge Representation and Reasoning, 2012. URL https:tpps...m.....t.. cdn. aaa .org ocs 4492/ 44 92 21843- -PB.pdf.
Junyi Li, Tianyi Tang, Jian- Yun Nie, Ji-Rong Wen, and Xin Zhao. Learning to transfer prompts for text generation. In Proceedings of the 2022 Conference of the North American Chapter of
12

---

Published as a conference paper at ICLR 2024
the Association for Computational Linguisties: Human Language Technologies, pp. Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10.18653/v1/
202..mmaal-mai.22577. URL ht http::://aaaatttttoaaaaa//
org//022 http:/222.aaaammmammooeaa nwacn-aacnmaaaaaa 257.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Anmual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference Natural Language Processing (Volume 1: Long Papers), pp. 4582-445 4597, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/ v1/2021 aacl.....3.- URL https /ttp///aaaanni /ww.aaaaananaaaaaaaaaaaaaaoooola wwtpl/oooooorr/gg/e 202l acl-long aacoo....o...222....11oooogg 353.
Vladislav Lialin, Vijeta Deshpande, and Anna Rumshisky. Scaling down to scale up: A guide to parameter-efficien fine-tuning, arXiv preprint rrXiv:230315647, 2023. URL https://
arxiv. arxiv. wttp://///a.////aaaaaa 2303. 15647.
Bill Yuchen Lin, Abhilasha Ravichander, Ximing Lu, Nouha Dziri, Melanie Sclar, Khyathi Chandu, Chandra Bhagavatula, and Yejin Choi. The unlocking spell on base Ilms: Rethinking alignment via in-context learning. arXiv preprint arXiv:2312.01552, 2023. URL https:/ http:p:://aaxxv.oe/ abs/2312 01552.
Haokun Liu, Derek Tam, Mohammed Mugeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin A Raffel. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. In S. Koyejo. Mohamed, Agarwal, Belgrave, K. Cho, and A. Oh (eds.), Ad- vances in Neural Information Processing Systems, volume 35, pp. 1950-1965. Curran Associates, Inc., 2022. URL pettass/ppooceddng..nnnrrrt wttp:eerrrepp..c.op.ppeaaeeeeeee pdf.
cc/
2022/fi ffle/file//
Xiao Liu, Yanan Zheng, Zhengxiao Du Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. Gpt un- derstands, too. arXiv:2103. 10385, 2021. URL https //tp:///avvvvvv/ org/abs 21 21.3. 10385.
Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder. Compacter: Efficient low- rank hypercomplex adapter layers. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL https:
// //openzeview
Yuning Mao, Lambert Mathias, Rui Hou, Amjad Almahairi, Hao Ma, Jiawei Han, Scott Yih, and Madian Khabsa. UniPELT: A unified framework for parameter-efficient language model tun- ing. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 1015533/11/22222l---l1o tational Linguistics. doi:
Dublin, Ireland, May 2022. Association for Compu-
URL https: /tap/aaaaanttte htcp///alaaaaog....../o..
org/2022 . lone-llonngg 433.
Nihal V Nayak, Peilin Yu, and Stephen Bach. Learning compose soft prompts for compositional zero-shot learning. In The Eleventh International Conference on Learning Representations, 2022.
Mohammad Taher Pilehvar and Jose Camacho-Collados. WiC: the word-in-conter. dataset for eval- uating context-sensitive meaning representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Pupers). Association for Computational Linguistics, June 2019. URL. https:/ / /aclllllc anthol ogy. org/N1 9-1288
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transfer- able visual models from natural language supervision. In International conference on machine learning, pp. 8748 8763. PMLR, 2021. URL httphhtp:////ceeeeee radford21a.
mlr. press/v139/
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li. and Peter J. Liu. Exploring the limits of transfer learning with unified text- to-text transformer. J. Mach. Learn Res., 21(1), jan 2020. ISSN 1532-4435. URL https: //dl. acm. www//oiioooooooooooooooooooo.o.o /aeiia/bs///a 5555/ 34555/53455111 3455555335571633558555 3455856.
13

---

Published as a conference paper at ICLR 2024
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Perey Liang. SQuAD: 100,000+ questions for machine comprehension text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2383-2392 Austin, Texas, November 2016. Association for Computational Linguistics. doi: 10.18863//1/D1--- org/D16 6-116-16644
URL peatps://aclatteee ogy.
Andreas Rücklé, Gregor Geigle, Max Glockner, Tilman Beck, Jonas Pfeiffer, Nils Reimers, and Iryna Gurevych. AdapterDrop: On the efficiency of adapters in transformers. In Proceedings of the 202/ Conference Empirical Methods in Natural Language Processing. Association for Computational Linguistics, November 2021. URL emnlp-me man.miin... 626.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An ad- versarial winograd schema challenge at scale. Commun. ACM, 64(9)999-106, aug 2021. ISSN 0001-0782. doi: 10.1145/347438L. URL https://doi.orp//
Timo Schick and Hinrich Schütze. Exploiting cloze-questions for few-shot text classification and natural language inference. Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume. Association for Computational Linguistics, April 2021. URL https:://cllatttoll.l aallanthloogy....ooo org/ 2021. www.:..e.lcl.................e.a
20.
Zhengxiang Shi and Aldo Lipani. Don't stop pretraining? make prompt-based fine-tuning powerful learner. In Thirty-sevente seventh Conference on Neural Information Processing Systems, 2023. URL https: wppnrevieveeeeeeeeeeeeeeeeeeeeee net/ forum?ide ?id=s JOACI.
Zhengxiang Shi, Yue Feng, and Aldo Lipani. Learning execute actions ask clarifica- tion questions. In Findings of the Association for Computational Lingwistics: NAACL 2022, pp. 2060-2070, Seattle, United States, July 2022. Association for Computational Linguistics. naacl.158. URL https: hclaatthooot org/ 2022.
doi: 10. Sindings-naace 158.
Zhengxiang Shi, Francesco Tonolini, Nikolaos Aletras, Emine Yilmaz, Gabriella Kazai, and Yun- long Jiao. Rethinking semi-supervised learning with language models. In Findings of the Asso- ciation for Computational Linguistics: ACL 2023, pp. 5614-5634, Toronto, Canada, July 2023. Association for Computational Linguistics. URL https: : / //a/aaaaaae //acclnnthhlogyy org/202g/2202 2023. findings-acl. 347.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 1631-16422 Seattle, Washington, USA, October 2013. Association for Computa- tional Linguistics. URL https: atttss:/aaclanthologr.. org DD1// 1170.
Yusheng Su, Xiaozii Wang, Yujia Qin, Chi-Min Chan, Yankai Lin, Huadong Wang. Kaiyue Wen, Zhiyuan Liu, Peng Li, Juanzi Li, Lei Hou, Maosong Sun, and Jie Zhou. On transferability of prompt tuning for natural language processing. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Compulational Linguistics: Human Language Technologies. Association for Computational Linguistics, July 2022. doi: 10.18653/w1/0022. org/
http:///aaaaattoooooeooo naacl htt.nn//oo.ttttttt////aaaaaaaao ott/////oooo...////222222222 naacl maaacc--aan -aammmaii....mmmm 290.
Li-Lin Sung, Varun Nair, and Colin A Raffel. Training neural networks with fixed sparse masks. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Ad- vances in Neural Information Processing Systems, volume 34, pp. 24193-24205. Curran Asso- ciates, Inc., 2021. URL https: httpp::::ppooeeee/eeeo http:///pooccgee/ee/e/ee//// nwwrips..p.... cc wttp:/ppppp/e//////e..ee htt.:/a11.1..1ccc11aaeeee.eeeed
Yi-Lin Sung, Jaemin Cho, and Mohit Bansal. Vl-adapter: Parameter-efficien transfer learning for iision-and-languue tasks. In Proceedings of the IEEE/CVF Conference Computer Viston and Pattern Recognition, pp. 5227-5237, 2022a. URL https: arxiv. org/ absg///bb211 06825.
14

---

Published as a conference paper at ICLR 2024
Yi-Lin Sung, Jaemin Cho, and Mohit Bansal. LST: Ladder side-tuning for parameter and memory efficient transfer learning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022b. URL https: /pppneeeeeeeeeeeeeeeeeee www.eeeieeeeeeeeee.eeeeeeeeeeeee net/ forum? d= sPrnaTZaP5.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajwwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foun- dation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023. URL https: //arxiv. org/ abs 200. 09288.
Adam Trischler, Tong Wang. Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. NewsQA: A machine comprehension dataset. In Proceedings of the 2nd Workshop on Representation Learning for NLP. pp. 191-200, Vancouver, Canada, Au- gust 2017. Association for Computational Linguistics. doi: 10. 10.18653/11/W17-22222 URL
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention all you need.
Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Ad- vances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https: /ttp////o/eeeeee/eeeeeeo ngs... neurips. 4354 7dee fbd053cdd (alod0331144445eeee0e c4a845aa-Paper.22
Tu Vu, Brian Lester, Noah Constant, Rami AI-Rfou', and Daniel Cer. SPoT: Better frozen model adaptation through soft prompt transfer. In Proceedings of the 60th Annual Meeting of the Asso- ciation for Computational Linguistics (Volume 1: Long Papers), pp. 5039-5059, Dublin, Ireland, URL
May 2022. Association for Computational Linguistics. doi: 10. https: / http//caanntthooooooooo org/ 2022 awl.2a..ccllae aww.acclllnnon. 34
Alex Wang. Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceed- ings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pp. 353-355, Brussels, Belgium, November 2018. Association for Computational Lin- guistics. doi: 10.18653/v 1118655/1/W18-.
URL https: http///aaaaantoto/o//ee org/ 8-16-544.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. In arxiv, 2019. URL http::///// arxiv. orgg httt:::/aaax....rrg/aab///////// 11955.00377..
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe V Le, Ed H. Chi. Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, 2023a. URI. https://cpenrevia http///ppneeeeeeeeeeeeeee net wtt.eeweee//eteeeeeeeeeeee Dorumiddd1PLLNNN0W:
Zhen Wang. Rameswar Panda, Leonid Karlinsky, Rogerio Feris, Huan Sun, and Yoon Kim. Mul- titask prompt tuning enables parameter-efffcicnt transfer learning. In The Eleventh Interna- tional Conference on Learning Representations, 2023b. URL
forum? waw.mmmiiddd22ptthtggg
Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625 641, 2019. doi: 10. 162/ tacl mncla.000200. URL https: http://aaaaattt aclaathoonnoooyooooooooooooooooe wwtp///g//////
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sen- tence understanding through inference. In Proceedings of the 2018 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp. 1112-1122, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. URL https: aaclanthoooggyy....o org/ /atp//1811111111
David Wingate, Mohammad Shoeybi, and Taylor Sorensen. Prompt compression and contrastive conditioning for controllability and toxicity reduction in language models. In Yoav Goldberg.
15

---

Published as a conference paper at ICLR 2024
Zomitsa Kozareva, and Yue Zhang (eds.), Findings of the Association for Computational Lin-
guistics: EMNLP 2022, pp. 5621-5634, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguisties. doi: (0.18653/v1/0222 findiings-emmll.411... URL
https: /aae:///lllaaa /ttt:::://cllnthhlooog/o
orrg/002..... fi -indingnnneniiinni m p.412.
Bin Wu, Zaiqiao Meng, Qiang Zhang, and Shangsong Liang. Meta-learning helps personalized product search. In Proceedings of the ACM Web Conference 2022, WWW '22, pp. 2277-2287, New York. NY, USA. 2022. Association for Computing Machinery. ISBN 9781450390965. doi:
10.1145/34854473351200 URL www.thttt//// /oo.. org/oo/g////// 11 1 30555/4854447 .31122333.
Bin Wu, Jinyuan Fang. Xiangxiang Zeng, Shangsong Liang, and Qiang Zhang. Adaptive composi- tional continual meta-learning. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.). Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 37358-37378. PMLR, 23-29 Jul 2023a. URL httph:ttp///.. p www.nnsmm...........mm.. ....sssss///ss///// v202 www html.
Bin Wu, Zaiqiao Meng, and Shangsong Liang. Dynamic bayesian contrastive predictive coding model for personalized product search. ACM Trans. Web, 17(4), oct 2023b. ISSN 1559--131.
doi: 10. 10.1145/3609225. URL https: htop:s://dd/ /doi. org/or.g.... 0011//369902
Adam X Yang. Maxime Robeyns, Edward Milsom, Ben Anson, Nandi Schoots, and Laurence Aitchison. A theory of representation learning gives deep generalisation of kemell meth- ods. In International Conference on Machine Learning, pp. 39380-39415. PMLR, 2023. URL https: https::/procees: http://poocdddeeeee ml press v202/yangggggee
html.
Adam X Yang, Maxime Robeyns, Xi Wang, and Laurence Aitchison. Bayesian low-rank adaptation for large language models. In The Twelfth International Conference on Learning Representations, 2024. URL https: /ttpr/vieeeeeeeeeeeeeeeeeee n www.ooooddnnduuliiaaaea
1m.
Zhilin Yang, Peng Qi Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdi- nov, and Christopher D. Manning. HotpoiQA: A dataset for diverse, explainable multi- hop question answering. In Proceedings of the 2018 Conference on EMNLP, Brussels, Bel- gium, October-November 2018. Association for Computational Linguistics. URL https: / //aclanthology wwtp///-1/////////////
Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme. Record: Bridging the gap between human and machine commonsense reading comprehension. http//////111111......
arXiv preprint 1810. 2018. URL https://arxiv..og///a/aaa
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In Proceedings of the 28th International Conference on Neural Infor- mation Processing Systems Volume 1, NIPS' pp. 649-657, Cambridge, MA, USA, 2015. MIT Press. URL httpss:/pproceeiiggs www.:eeeeeeiin.....eeeeeeeeeeeee cc htcp//pppe//////////eeeee 2
pdf.
Yuan Zhang, Jason Baldridge, and Luheng He. PAWS: Paraphrase adversaries from word seram- bling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Pa- pers), pp. 1298-1308, 1308, Minneapolis, Minnesota, June 2019. Association for Computational Lin- guistics. doi: 10. 10.18653/v1/N191111 1131. URL httpt: ac wwwaaanttooly..ooooooooo
91131.
Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, LILI YU, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. LIMA: Less is more for alignment. In Thirty-seventh Conference on Neural Information Process- ing Systems, 2023. URL htttpss//opeereettt net/
wwwtffoumm forum? K=BBMOMX2hee.
16

---

Published as a conference paper at ICLR 2024
APPENDIX OVERVIEW
The appendix is structured as follows:
Appendix $A provides visualization of the model performance against the number of trainable parameters on the GLUE and SuperGLUE benchmarks.
presents the additional experimental results, including using a larger size of lan- guage models (LLAMA-2 and TB-3B) and testing the impact of different lengths of soft prompts.
Appendix $B
Appendix $C
provides brief description of all datasets used in this work.
provides implementation details and hyperparameters for all comparison methods used in our experiments.
Appendix $D
Appendix $E provides further discussion regarding intuitions and related works.
A MODEL PERFORMANCE AGAINST THE PARAMETER EFFICIENCY
DARAMETER-EFRCIENCY
We visualize the experimental results in Table shown in Figure 6. The visualization shows that our proposed method DEPT outperforms other PEFT approaches and full fine-tuning baselines on the GLUE and SuperGLUE benchmark (y-axis) while updating only small number of trainable parameters (x-axis).
GLUE Benchmark
SuperGLUE Benchmark
77.5
86
75.0
Average Performance (%)
Average Performance (%6
72.5
70.0
84
67.5
83
65.0
62.5
82
60.0
10P
10
10'
10*
109
103
10
10'
109
109
number trainable parameters
The number of trainable parameters
Full Fine-tuning
LoRA
DePT (T5-Base)
HyperDecoder (m)
Ad adapter
LST
Full Fine-tuning (m)
SPoT
AdapterDrop
Adapter (m)
ATTEMPT
BitFit
DePT (GPT-2 Large)
Hyperformer (m)
MPT
Figure 6:
The average performance against the number of trainable parameters on the GLUE and SuperGLUE benchmark using the T5-BASE model.
B ADDITIONAL EXPERIMENTS
LLAMA-2. We evaluate the performance and inference speed of our proposed method DEPT using LLAMA-2-7D A-BBB and LLAMA-2-13B (Touvron et al., 2023) the SST-2 dataset. In our experiment, the soft prompt length for the vanilla PT set to = 100. For DEPT, we set the soft prompt length to 60 and select rank of 40 for the low-rank matrices. As shown in Table 6, our experimental results suggest that DEPT not only outperforms the vanilla PT terms test accuracy but also improves the speed of inference. We only limit evaluation of DEPT to the
17

---

Published as a conference paper at ICLR 2024
SST-2 dataset due the high computational expenses. We will do best to get the necessary resources to further probe the performance of DEPT, aiming to deliver a more exhaustive evaluation in future work.
Prompi Tuning
DEPT (ours)
Method
Test Acc
Inference samples per second
Test Acc
Inference samples per second
LLAMA-2-7BB
94.48
3,895 2.083
94.95
4.857 2.835
LLAMA-2-13B
95.99
96.01
Table
Test results using LLAMA-2-7B and LLAMA-2-13B on the SST-2 dataset.
T5-3B. We evaluate the performance and inference speed of our proposed method DEPT using the TS-3B model. We report the average performance on the Glue dataset as well as inference speed, measured in inference samples per second. As shown in Table 7, our findings indicate that DePT (m=60, r=30) outperforms PT in terms of inference speed by 37%. This suggests the advantage of DePT increases as the model size increases.
Method
Average Glue Performance
Inference samples per second
DEPT (m=60, r=30) PT (m=100)
86.4
8.9
85.6
6.5
Table 7: Test results using T5-3B on the Glue Benchmark.
Different prompt lengths. We have performed additional experiments regarding different prompt lengths, shown in the Table below. Specifically, we have increased the size of trainable parameters in both DEPT and PT by factor of two. We use the T5-BASE as the backbone. As shown in Table 8, we report the average performance on the Glue dataset as well as inference speed, measured in inference samples per second. Our findings indicate that DEPT (m=120, r=0) outperforms in terms of inference speed by 34%. We believe that this performance advantage can be further enhanced by reducing the value of m, which represents the length of the soft prompt. To provide a conerete example, the SST-2 dataset, DEPT can achieve inference speed of 77.2 samples per second, while PT can only infer 57.4 samples per second. This suggests the advantage of DEPT over PT increases the model size increases.
Method
Average Glue Performance
Inference samples per second
DEPT (m=120, r=60) PT (m=)00)
86.0
54.8 40.8
85,2
Table 8: The impact of using longer soft prompt length. Test results using T5-BASE the Glue Benchmark.
C DATASET
In this work, we use 23 popular datasets from previous few-shot learning and PEFT research. We limit the maximum training data number of Yelp-2 to 100k samples. We train MNLI with longer steps, 200k steps in total. For the GLUE dataset, use the HuggingFace dataset?. For the Super GLUE dataset, we use the HuggingFace d For MRQA 2019 Shared Task and other datasets, use the HuggingFace dataset!.
18

---

Published as a conference paper at ICLR 2024
GLUE Benchmark
Dataset
Source
Target
#Train
#Valid
#Test
Type
MNLI
31.8
1.0
392,702
9,832
9,815
NL.I
QQP
24.1
1.0
362,846
1,000
40,431
Paraphrase
QNLI
38.4
103,743
1,000
S,463
NLI
SST-2
10.4
1.0
66.349
1,000
872
Sentiment
STS-B
21.9
1.0
5,749
750
750
Sent. Similarity
MRPC
3,668
204
Paraphrase
RTE
54.4
1.0
2.490
138
139
NLI
CoLA
8.7
1.0
8,551
521
522
Acceptubility
SuperGLUE Benchmark
Dataset
Source
Target
#Train
#Valid
#Test
Type
MultRCC
286.1
1.0
27.243
2,424
2,424
Question Answering
BooiQ
108.3
1.0
9,427
1,635
1,635
Question Answering
WIC
18.4
1.0
5.428
319
319
Word Sense Disambiguation
1.0
554
52
Common Sense Reasoning
CB
64.6
1.0
250
28
28
NI.I
ReCoRD
210.7
1.5
137,484
1,370
15.176
Common Sense Reasoning
MRQA 2019 Shared Task
Dataset
Source
Target
#Train
#Valid
#Test
Type
NaturalQuustions
242.7
4.5
103,071
1,000
12836
Question Answering
HotporQA
225.7
2.6
71.928
1,000
5,901
Question Answering
SeurchQA
942.8
2.0
116,384
1,000
16,980
Question Answering
NewsQA
615.5
5.1
730160
1,000
4,212
Question Answering
Datasets
Dataset
Target
#Train
#Valid
#Test
Type
WinoGrande
23.8
1.0
39,398
1,000
1,267
Common Sense Reasoning
YelpPolarity
134.0
1.0
100,000
1,000
38,000
Sentiment
SciTail
30.8
1.0
23,596
PAWS
4,9401
8.000
Similarity
Vision Language Tasks (#images & #Texts)
Visual Question Answering MS CoCo Caption
113.2K/605. 1K
5.0K/26.7K
5.0K/26.3K
Question Answering
113. KK
5.0K/S.OK
S.0K/S.OK
Caption Generation
Table 9: The datasets evaluated in this work. Source indicates the average length of the source sentences in the training set. Target indicates the average length of the target sentences in the training set. STS-B real-valued regression task over the interval [0, 5)). Note that we only sample examples from the original training set in our few-shot experiments.
Hyperparameter
Assignment
number steps
30,000 steps (evaluate every 1,000 steps)
batch size
maximum learning rate (a1)
4e-1, 5e-1
maximum learning rate (oz)
lc-04, 5c-4, 1c-03
length of the soft prompt (vn)
20, 40, 60. 80
maximum sequence length
256
leamiigg rate optimizer
AdamW
Adam epsilon
le-6
Adam beta weights
0.9, 0.98
learning rate scheduler
Warmup linear
Weight decay
0.01
Warmup proportion
0.06
Table 10: Hyperparameters for Prompt Tuning and DEPT.
19

---

Published as a conference paper at ICLR 2024
D IMPLEMENTATION DETAILS
Our code is implemented using Pytorch', Huggingface Transformers', and Huggingface PEFT Below, we provide comprehensive list of the hyperparameters used in code. In our work, we mainly cite the experimental results from the previous works Asai et al. (2022); Wang et al. (2023b); Sung et al. (2022b). In addition, we train LoRA with up 200k steps. We search the learning rate within the set {5e-4, le-4, 5e-5, le-5]. We set the rank as 35. We choose a batch size of 32. We find that training LoRA on the MRQA dataset presents challenges, despite conducting thorough search for optimal learning rates and training steps. The reasons for these difficulties remain uncertain. For prompt tuning and DRPT, as shown in Table 10, we conduct grid search for learning rates. For the soft prompt, we search the learning rate within the set (3e-1, 4c-1, 5e-1]. J. For the low-rank matrice pairs, we search the learning rate within the set { {Ic-04, 5e-4, 1c-03, 5c-033. We choose batch size of 16. We typically use the max sequence length as 256 except for the SuperGLLUE-Muldgg where the max sequence length is 348. In each trial, we train the model for 30,000 steps, evaluate performance every 1,000 steps, and select the best checkpoint based on optimal performance on the evaluation set. For the large dataset with more than 100,000 training examples, we follow the prior work (Vu et al., 2022) to train the vanilla PT and our proposed method DEPT with up to 300,000 steps. Training more steps helps improve the performance of the vanilla PT for the large dataset. The best performance is determined by the relevant evaluation metric. We train the T5 model from the original checkpoint rather than the LM-adapted 1.1 version (Lester et al., 2021).
E FURTHER DISCUSSION
The intuition of DEPT is that (1) given the same number of trainable parameters, allow- ing some updates for word embeddings will improve the performance; and (2) a shorter soft prompt will improve the efficiency. To illustrate, the previous study (Wingate al., 2022) has shown that a soft prompt can interpolate between many token embeddings, enabling the representation of more abstract concepts compared to relying on a single discrete token. However, the soft prompt in the PT is consistently added at the beginning of the frozen word embedding, In contrast, we propose DEPT, which decomposes the long soft prompt into a short soft prompt and pair of low-rank matrices. This approach can (1) reduce the length of the soft prompt for better efficiency; and (2) permit rep- resentation updates within the frozen word embedding, thereby increasing the adaptability of input representations that were previously unavailable.
Intuition.
Related works with similar titles. The meaning of "compose" and the method are fundamentally different between previous works (Khot et al., 2022: Nayak et al., 2022) and our work. Specifically, Decomposed Promptin (Khot et al., 2022) focuses on in-context learning, without the need to update parameters. Decomposed Prompting aligns closely with the work of shain-of-thoughts and self- consistency. In addition, CSP (Nayak et al., 2022) treats the attributes and objects that are composed to define classes Icarnable tokens within the vocabulary. In contrast, our proposed method DePT does not train soft prompts associated with any vocabulary token, nor does it add additional tokens to the vocabulary. The main goal of DePT is to improve the efficiency of Prompt Tuning (PT) due to the increased input sequence issue.
Comparison between Prompt Tuning (PT) and LoRA. We would like to discuss the comparison between Prompt Tuning (PT) and A, as our work aims to improve the PT, in the following points:
Relative Performance of LoRA and PT. When adapting language models (LMs) to spe- cialised domains, like mathematical reasoning, which requires much different knowledge than what L.MSs have been trained on, LoRA may perform better than Prompt Tuning (PT). However, in case tasks have already been somewhat understood by LMs and the key chal- lenge is just to properly prompt the LMs, PT can be the better option. PT modifies minimal
hhtte:://yytorch.orrg//
20

---

Published as a conference paper at ICLR 2024
model parameters, focusing instead on improving the input prompt, which has been proven more effective than LoRA in prior studies (Asai et al., 2022; Wang et al., 2023b).
Specific Use Cases for PT. PT offers advantages in particular cases. For example, soft prompts can be used to compress few-shot examples in the prompt long context (Cheva- lier al., 2023; Wingate et al., 2022). While the number of trainable parameters is low, LoRA updates the weight matrices across the whole model. In contrast, PT only improves the input of the LM through the soft prompt, which helps the model focus on understanding the task and context better rather than learning new knowledge.
Parameter Efficiency. Unlike LoRA, which requires trainable parameters at each layer, PT's trainable parameters are more concentrated and less extensive.
Parameter-efffcientt transfer learning (PEFT) Framework. Framework. PETL frame- work (e.g.. Asai et al. (2022); Wang et al. (2023b)) can effectively improve the performance of the PT and make easier to use. our work, we have demonstrated that our approach is compatible with the PEFT framework.
21


Reviews:
[{"review_id": "P55ispDvmX", "reviewer": "Reviewer_yLsP", "summary": "The paper proposes a new PEFT method that falls under the prompt tuning category. The paper is motivated by the fact that soft prompts increase the sequence length which results in increased training and inference time. The main idea of the paper is to reduce the number of soft prompt tokens and use the remaining parameters to perform low-rank updates to the embedding matrix. The DePT method outperforms vanilla prompt tuning in almost all cases and is competitive with many other PEFT methods.", "questions": "**For me to retain my current score**\n\nQ: Many of the tables have inconsistent baseline and incomplete results. All the superglue results should be filled in Table 1. I understand that the numbers are not available in past papers but it is not hard to obtain these numbers. This is important because the LORA method is the closet peft method on the glue benchmark however it is missing when looking at Superglue. Similarly, Table 2 should have LoRA as a baseline.\n\nQ: IA3 is a recent PEFT method that is very parameter efficient and is missing from the baseline comparison. It is a crucial baseline, especially in the case of the few-shot adaptation. It should be added at least to Tables 4 and 5 and adding it to Tables 1-2 would also be good.\n\n\n\n**Other Questions**\n\nQ: Most of the improvement in the table-1 glue task seems to come from 1-2 tasks like cola, rte any specific reason for this?\n\n\nNote for other reviewers and AC: I am not sure of the related work section, there might be some relevant prompting-related baselines that I am not aware of.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "S1: The idea is very simple and leads to decent improvements over the baseline methods. Also, the paper is very easy to read and understand. \n\nS2: The experiments are thorough enough, however, I have some mild additional suggestions that might make the experimental section more complete.", "weaknesses": "W1: Some of the important baseline methods like IA3 are missing. See questions below. \n\nW2: The idea is interesting, however some more intuition on why this works might strengthen the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "auYorQiSuM", "reviewer": "Reviewer_BZux", "summary": "This paper proposes a novel parameter efficient tuning method for language models named decomposed prompt tuning or DePT. Compared with popular parameter efficient tuning method prompt tuning, deft aims to learn more compact soft prompt and a pair of low-rank matrices for updating the vocabulary embeddings. The key innovation for DePT is to offload the potential need for long soft prompts and decompose them into a pair of low rank matrices. Through experiments with both language models and vision-language models across various tasks, the authors find that DePT can achieve advantageous performance while saving approximately 20% memory cost.", "questions": "I appreciate the ablation experiments on the significance of different learning rates for the soft prompts and low rank matrices. I also wonder if there are other intuitive explanations.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "To the best of the reviewer’s knowledge, the method proposed in this paper DeFT is novel. The authors also provide solid intuitions and reasoning for this method. Besides the method constructions, the experiments are comprehensive. I also appreciate the authors’ efforts in organizing the anonymous project code that covers the experiments.", "weaknesses": "The key contribution of DePT lies in that it is both optimizing a soft context as long as the vocabulary in an efficient manner. The decomposition idea, although novel in its current form, is incremental to current PEFT methods. There is also existing work (e.g. [1]) that explored tuning subsets of vocabularies as a way of PEFT. That being said, DePT still has the advantage of efficient vocabulary tuning. The 20% efficiency advantage also is only revealed with one soft prompt length of 100. It would be appreciated if the authors are to disclose more performance comparison under different prompt length scenarios. \n\nRef: \n[1] Nayak, N. V., Yu, P., & Bach, S. (2022, September). Learning to Compose Soft Prompts for Compositional Zero-Shot Learning. In The Eleventh International Conference on Learning Representations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RCJ3b8YYKb", "reviewer": "Reviewer_mz3i", "summary": "The work introduces decomposed prompt tuning for parameter-efficient fine-tuning, or DEPT. They propose to reduce the prompt and add a decomposable matrix to the word embeddings. The experiments on GLUE and SuperGLUE show the method's effectiveness while being more efficient than prompt tuning. The few-shot learning experiments show that the method is competitive with other parameter-efficient fine-tuning methods with more parameters. Overall, the method improves prompt tuning while being 20% efficient in training.", "questions": "- In implementation details, you have included that the soft prompts and low rank matrices are initialized from soft prompts derived from one of the source tasks. Could you clarify this detail? It is unclear why this is necessary. \n- Do you think $\\mathbf{BA}$ could use a scaling factor like $\\frac{\\alpha}{r}$? It is possible that this could be causing the low performance due to when only the decomposed matrices are trained. \n- In the background, you have mentioned that $s$ is the maximum sequence length. How would you set $s$ for Llama2 or models that do not have a fixed maximum sequence length?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "**Paper quality.** The paper is well written. The organization of the paper is clear and well thought out. I enjoyed reading the paper. \n\n**Extensive experiments.** The paper extensively experiments with improved results compared to prompt tuning while being more efficient during training and inference. The authors have done a great job comparing the work with other recent methods in the literature and show that DEPT outperforms them on GLUE and SuperGLUE. They further provide evidence of their efficiency on GPT and Llama2 models. \n\n**Code.** The code is well organized and easy to understand (see https://anonymous.4open.science/r/DePT-8F43/README.md).", "weaknesses": "**The architecture is not well motivated.** \nThe architecture appears to be a combination of prompt tuning and LoRA. But, unlike LoRA, DEPT still suffers from prompt length compared to architectures at inference time. While DEPT can also achieve the same inference speed as the base model, like LoRA, when the prompt length is 0, in Figure 3, we see that the performance is about 20 points below the DEPT performance reported in Table 1. Furthermore, decomposing the prompts does not offer any conceptual understanding or insight into the learned prompts. \n\n**Experiments with only smaller models (<1B model parameters).**\nAll experiments in the main paper are conducted on smaller models which hurt the technical contribution of the work. The paper presents improved results over several parameter-efficient methods, by a small margin, on models that have less than 1 billion parameters. The authors have included results of Llama2 in Appendix B but show improvements over prompt tuning on only one dataset (SST-2). It would be of interest to the community to know if the work is applicable to larger models say T5-XL, T5-XXL, or Llama2 on more datasets.\n \n**Missing experiments.**\nI noticed that the authors do not include results on SuperGLUE for LoRA in Table 1. Since the results for DEPT and LoRA are so close on GLUE, I would be curious to see how close the results are on SuperGLUE. \n\n**Suggestions.**\n- Related work: The title of your work is similar to the below papers [a,b] but the proposed work differs. You could choose to cite them and clarify the differences in the next version of the paper. \n- Move the 3.3 #2 (DEPT grows more efficient with model size increases) to the Appendix and move Llama2 to the main paper as it is obvious that the inference time will improve with model parameter size. The Llama2 results, on the other hand, are more interesting.\n- For completeness, it would be great if the authors could include the GPT-2 results in the Appendix on GLUE and SuperGLUE. I’d be curious to see if there are any changes in the performance. \n\nReferences:\n\n[a] Decomposed prompting: A modular approach for solving complex tasks. ICLR 2023.\n\n[b] ​​Learning to compose soft prompts for compositional zero-shot learning. ICLR 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kMnHEm2Ljd", "reviewer": "Reviewer_XBEs", "summary": "The paper proposes Decomposed Prompt Tuning (DePT), which decomposes the soft prompt into a shorter soft prompt and a pair of low-rank matrices optimized with two different learning rates. The authors conduct experiments to evaluate the effectiveness and efficiency of the proposed DePT.", "questions": "See the questions mentioned above.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "- the paper is easy to follow\n- extensive experiments on simple datasets are conducted to evaluate the proposed method", "weaknesses": "- **motivation is weak**: \n  - soft prompts are already very parameter-efficient; it is not necessary to reduce the #parameters in prompt tuning\n  - usually, the length of soft prompts is small, e.g., 8, 16, 32; the additional inference cost is minor compared with the current LLM, which can accept thousands of tokens as input\n  - though prompt tuning is sensitive to initialization, there are some recent methods to deal with this problem, e.g.,\n    - Effective Structured Prompting by Meta-Learning and Representative Verbalizer, ICML 2023\n    - MetaPrompting: Learning to learn better prompts, COLING 2022\n- for the method, the idea of using a LoRA matrix to approximate part of soft prompts is odd to me. Can the authors visualize the learned A and B matrices to explain why DePT is better than Prompt tuning?\n- the tasks used in experiments are too simple; better to try more challenging tasks, like GeoQuery (Zelle & Mooney, 1996), NL2Bash (Lin et al., 2018), WebQS (Berant et al., 2013). \n- datasets in GLUE and SuperGLUE are sensitive to hyperparameters. What are the hyperparameters for each dataset?\n-  In Table 3, DePT is much worse than FT on VQA, but can perform better on MSCOCO; why?\n- how to initialize the soft prompt in prompt tuning? Usually, they can be initialized randomly or use the embeddings of label tokens.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)