Prompt Tuning Strikes Back: Customizing Foundation Models with Low-Rank Prompt Adaptation
Authors: Abhinav Jain
Swarat Chaudhuri
Thomas Reps
Chris Jermaine
Venue: NeurIPS 2024 poster (2024)
Keywords: Parameter-Efficient Fine-Tuning
Prompt tuning
Primary area: generative_models
TL;DR: Low-Rank Prompt Adaptation, soft-prompting approach for effective and efficient customization of large foundation models.

Abstract: Parameter-Efficient Fine-Tuning (PEFT) has become the standard for customising Foundation Models (FMs) to user-specific downstream tasks. However, typical PEFT methods require storing multiple task-specific adapters, creating scalability issues as these adapters must be housed and run at the FM server. Traditional prompt tuning offers a potential solution by customising them through task-specific input prefixes, but it under-performs compared to other PEFT methods like LoRA. To address this gap, we propose Low-Rank Prompt Adaptation (LoPA), a prompt-tuning-based approach that performs on par with state-of-the-art PEFT methods and full fine-tuning while being more parameter-efficient and not requiring a server-based adapter. LoPA generates soft prompts by balancing between sharing task-specific information across instances and customization for each instance. It uses a low-rank decomposition of the soft-prompt component encoded for each instance to achieve parameter efficiency. We provide a comprehensive evaluation on multiple natural language understanding and code generation and understanding tasks across a wide range of foundation models with varying sizes.

Prompt Tuning Strikes Back: Customizing Foundation Models with Low-Rank Prompt Adaptation
Abhinav Jain
Swarat Chaudhuri
Department of Computer Science Rice University aj70@rice edu
Department of Computer Science UT Austin swarat@cs. www.ss.eeeec...mmmm.. edu
Thomas Reps
Chris Jermaine
Department of Computer Science University of Misconsin-Madison repsêcs visc.edu
Department Computer Science Rice University cmj40rice.. edu
Abstract
Parameter-Efficient Fine-Tuning (PEFT) has become the standard for customising Foundation Models (FMs) to user-specific downstream tasks. However, typical PEFT methods require storing multiple task-specific adapters, creating scalability issues as these adapters must be housed and run at the FM server. Traditional prompt tuning offers potential solution by customising them through task-specific input prefixes, but it under- under-performs performs compared to other PEFT methods like LoRA. To address this gap, we propose Low-Rank Prompt Adaptation (LoPA), prompt- tuning-basee approach that performs on par with state-of-the-ee PEFT methods and full fine-tuning while being more parameter-efficient and not requiring server-basee based adapter. LoPA generates soft prompts by balancing between sharing task-specific information across instances and customization for each instance. It uses a low-rank decomposition of the soft-prompt component encoded for each instance achieve parameter efficiency. We provide comprehensive evaluation multiple natural language understanding and code generation and understanding tasks across wide range of foundation models with varying sizes.
Introduction
Language models exhibit remarkable few-shot learning capabilities, demonstrating strong perfor- mance across tasks, including those unseen during training 2 32 23]. Nevertheless, fine-tuning remains erucial for optimised performance creasingly challenging with larger models because updating all parameters impractical. Parameter Efficient Fine- Fine-Tuning (PEFT) presents promising solution, adjusting a limited subset of parameters while leaving the rest unchanged. This approach allows the personalisation of pre-trained Foundation Models (FMs) to multiple users simultaneously.
given downstream task. However, it becomes in-
In recent years, numerous PEFT approaches have been proposed || 12
16
17
18
30 36 each
varying in how and what they modify within FMs [10 91. These variations
be categorized by the position in the FMs where modifications are applied, the functions used for modification, and the methods of integrating the modifications. While effective, these methods require maintaining multiple adapter-like modules for each user-specific task on the FM server and the need to select and assemble subset of these modules every time batch of user requests is processed for inference
351 (see Fig. T].
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Prompt tuning [15] is simple approach that has certain advantages. First, it parameter- efficient, requiring only small set of vectors (soft prompts) that are prepended at the input layer and learned for specific task. Second, prompt-based personalization requires no task- specific processing the server. A task-specific prefix is added before processing, allowing the FM server perform the same processing gardless of the task. However, despite its advan- prompt tuning has been shown to under- perform compared to other methods for PEFT [TO] This gap in performance raises concerns about the viability of prompt tuning as solution. Recently, efforts have been made enhance the performance of prompt tuning by strategically inserting soft prompts into different layers of the transformer [20 38 19 41. However, this im- provement increases the number of parameters and again necessitates server-side modifications for serving multiple tasks. interesting recent work explored a simple method to make soft prompts input-dependent by using lightweight prompt generator for each sample (88. This approach achieved notable improvements, raising the question: Can we further improve the performance of prompt tuning while staying parameter-effiiem??
Trainable
Frozen
User
Server
X
task
Input Query
Task-speciffc Adapters
tasky
Response
Yes
Foundation Model
Code Completion
Figure 1: A schematic illustrating how typical PEFT methods like LoRA achieve personalization of foundation model for multiple tasks, such as Yes/No text classification or code completion, dur- ing inference.
To this end, we propose Low-rank Prompt Adaptation (LoPA), a new instance-aware prompt tuning- based approach LoPA constructs the soft prompt from two components: a task-specific element that shares task information across samples and an instance-specific element that incorporates information from each individual instance. Our extensive analysis reveals that relying solely on either component, as done previous works 115 38) is insufficient for outperforming other PEFT baselines. LoPA achieves its high performance by taking more balanced approach.
LoPA combines the task-specific and instance-specific components using gating function, which activates task-specific information conditioned on each instance. Additionally, it employs low-rank decomposition of the instance-level component to enhance parameter efficiency (see Fig. 2). Once trained, users can provide the learned soft prompts as a prefix with their input to the FM, incurring no additional computational cost the server.
We conducted extensive experiments on various models. These included six benchmark NLU tasks from the GLUE dataset and three Code Understanding and Generation tasks. Our results show that LoPA outperforms existing prompt-tuning methods. It often matches the performance of full fine-tuning and LoRA. In 11 out of 24 test cases, we found LoPA outperformed LoRA.
To summarize, the contributions of this work are as follows:
We propose LoPA, parameter-efficient and high-performing prompt-tuning strategy.
We verify its effectiveness by evaluating it against full fine-tuning and staeeoff-the-ar PEFT methods in nine tasks using seven different transformer backbones.
2
Related Work
Adapter-based. Several recent approaches have emerged to support parameter-efficient fine-tuning. These methods typically involve incorporating trainable adapter layers modules into the trans- former network [1]] TO [7]. Training such layers has been demonstrated to be computationally more economical than full fine-tuning while maintaining performance. Notably, LoRA (12) has gained prominence for its low-rank approximation of model updates, effectively capturing task- specific knowledge. Beyond LoRA, Compacter 14 introduces adapters parameterised by low-rank hyper-complex multiplication (PHM) layers
39 to achieve more optimal balance between task
'The code for LoPA be found here

---

performance and the number of trainable parameters. DoRA [8] is another recent approach that de- composes weights into their magnitude and directional components and employs LoRA for directional updates to minimise the number of trainable parameters efficiently.
Soft Prompting. Another line of work advocates for strategically inserting soft prompts into hidden states of the transformer instead of using trainable adapter modules. For example, prefix-tuning [16] and P-tuning-v2 (20) prepend trainable prefix vectors to keys and values matrices all transformer layers. Prompt tuning, P-tuning, and DePT 15
21 30 are special cases that operate by prepending prompt embeddings to the input at the first layer, with 30] being a sebrid-approace that further uses LoRA to learn updates for the input's embedding matrix. While these approaches are instance- agnostic and optimise speciipecific prompt, there are also methods to optimise an instance- aware soft prompt. For instance, IDPG (38] generates soft prompt for each sample, prepended either at the initial word-embedding layer (version-S) or all layers (version-M), L.T [9]] inserts a prompt into some intermediate layer (i) to eliminate gradient calculation below it for faster training and (ii) to retain more relatel information (which could be lost it had to be propagated through lower layers). SPT [4]] aims to be more intelligent by learning gating function to determine whether the soft prompt from the previous layer should be propagated or if a new one should be learned.
With the exception of prompt tuning and S-IDPG, PEFT approaches mostly operate by injecting prefixes and new trainable modules into deeper layers or doing low-rank re- re-parameterization of existing ones, necessitating the storage of PEFT parameters at the server to update the foundation model. In contrast, LoPA provides the soft prompt as a prefix that prepended to the input query, overcoming the need to store task-specific parameters on the server. Furthermore, we demonstrate that LoPA achieves a more balanced trade-off between specificity and generalization compared existing approaches for enhancing learned models, such as prompt tuning [15], which solely focuses on general task-specific soft prompt, and IDPG 38 which emphasizes an instance-specific prompt.
3 Proposed Methodology
In this section, we formally define the proposed approach, followed by an explanation of how affects model learning compared to existing soft- noft-prompinng approaches.
3.1 Preliminaries
Transformers. A transformer model consists of multiple stacked layers, where each layer has multiple heads (= NH). each performing self-attention over the input [33]. Let us consider single head, H parameterised by the query. key, and value weights as W W WV E Rd xd respectively, where d is the model dimension. In multi-headed attention, du is typically set to Nø For given sequence of n input vectors X the output of the head at position is
d
{x1 .....y ,...,,")) x and query vector x² with each E Rd
o¹ = Attteetion (x k X, WV x)= = softmax SramaxxxW9))) (W(K))
where, we have ignored the constant dH for notational convenience. For the first layer of the transformer network, the input X is the embedding matrix i.e. X = X, E Rdxn.
Full Fine-Tuning (FFT), With fine-tuning, the objective is to adapt the model to a new task with data D {(.y)). Formally, adaptation achieved by updating the weights (W) of the model to maximise the lig-likeldhood of the response y, i.c., maxw C = (xx,)>~((ogg Pw (y/X)].
Prompt-Tuning (PT). The objective of prompt-tuning is to achieve task task-specific model adaptation by learning a soft prompt" Z. Such a soft prompt is prepended to the input word embeddings and trained via propagappttt maxz L E[log Pw (y) cy/oonat,,Z X,)).. In PT, the soft prompt Z E Rax is parameterised by a set of m learnable vectors {z (z),...2 zz }. where m denotes its length [15]. Thus,
can be interpreted an embedding of virtual tokens that enable the model perform downstream task without updating its parameters (28 g)
With prompt tuning, the output of an attention head in the first layer is modified as follows:
OPT Attention Ox' W concat(Z, (e,)), W. (oncat(Z,X.))
3

---

By using the formulation of Attention from Eq.
1 this equation can be equivalently written as Au WV
Σ
+ Σ )
PT
bias
scale
exp( T
with
As
(3)
Σx exp( (T + D3 exp((WXX)) W
where, Aas is the attention transformer gives to the prefix vector Z₁ for given query vector x'. In Eq.
2 we can observe that the soft prompt linearly interpolates the head's position- wine output; where the
bias be considered in an offset subspace spanned by vectors {WV 2 1m kk=1
with dimension
m (or < m) 10
3.2 Low-rank Prompt Adaptation (LoPA)
LoPA constructs the soft prompt as Z = Zs (Zi), where Zg E Rdxm is the task-specific component and Z, E Rdxm is the instance-specific component. These are combined using the gating function g. implemented using sigmoid, where denotes the Hadamard product. Intu- itively, by sharing Zg across instances, it captures general information, adapting the model to user-defined tasks, while Z, fine-tunes the soft prompt for specific instances, acting as activations. Both Zş and Zr have their own dedicated parameters. Similar prompt tuning, Zs consists of m learnable vectors. Conversely, Z, is obtained from input us- ing the encoding function Raxx + Rdxxm_ However, en- coding matrix of size d can be expensive. For example, in [38]. an MLP layer with hidden dimension h requires O(hdm) parameters. To improve parame- ter efficiency, we assume a low- rank decomposition of Z, as Z, uu and encode the two matrices with rank r, u E gdxr and Rmxr, using separate MLP layers. This design makes f computationally cheaper, with O(hdm( + m r)) parameters, reduced by factor of ( . < 1 when is chosen be < min(d, m). The overall composition of Z can be represented as:
Trainable
Frozen
X
soft prompt
imput prompt
Z
Zi
14
MLP
000000
FM
25
Encoder
Input Query
Low-rank
dlcoeppstiinn
task
specifii
instance
specific
User
Figure An illustration LoPA. No task task-specific adapters need to be stored on the server. represents the concatenation of the soft prompt Z and the input prompt X, i.e. X concat (Z(X..)
Zs ogggggMM (MLP,(()) x ML),
Z (x=
where marks the three trainable components. The MLP head consists of a down- and up propreeection layer with non-linear activation between. It sits atop smaller language model, referred to the Encoder model, that gives the input representation X'. A visual illustration of the framework can be found in Fig. 2
Next, we provide an intuitive explanation why LoPA could be better than traditional prompt-tuning and existing instance-specifer approaches
15
38
Offset subspace induced by LoPA. From Eq.2, we can observe that the bias vector in the offset subspace is linear combination of {WV (W 3²} Vk E (1....,ml} m), with AçR. (Eq. 3) representing the to vary accordingly. Consequently, distinct offset sub-spaces emerge for different inputs. contrast, traditional prompt tuning maintains fixed vectors while allowing only the scalars to vary with input. As result, instance- sensitive approaches can exert greater influence on the attention patterns deeper layers of the transformer, thereby offering enhanced flexibility and responsiveness to varying inputs.
scalars 10 26]. In LoPA, the input influences Z, causing the vectors {WV z² V

---

Tuning
Tunable
SST-2
MNLI
MRPC
QNLI
QQP
RTE
Avg
Parameters
(acc)
(acc)
(acc, FI)
(acc)
(acc, FI)
(acc)
FFT
355M
95.99
90.40
90.81
94.60
90.39
85.92
91.35
p-tuning-v2
0.49M
89.91
88.13
70.18
85.21
84.63
53.43
78.58
prefx-tuning
25.75M
93.80
89.51
90.86
94.98
86.87
82.67
89.78
LoRA
2.36M
96.22
90.30
90.77
94.69
89.91
85.66
91.26
None
0
59.97
39.60
73.52
50.16
42.34
53.43
53.17
PT
10.2K
84.40
54.67
72.38
58.74
48.20
53.07
61.91
DePT
10.2K
89.68
71.51
72.97
57.09
45.81
53.79
65.14
S-IDPG
2.89M
95.30
84.50
78.60
90.48
84.88
77.26
85.17
Ours
1.60M
95.99
89.22
91.09
93.74
89.72
83.39
90.53
Table 1: Performance on GLUE tasks. We report the average of accuracy and F1 for both MRPC and
QQP. For all the other tasks, we report accuracy. Approaches below the dotted line do not require any modification to the model on the server side. Bold denotes the best-performing tuning method for the given model. Underline marks best result among all prompt tuning methods.
Coupled learning of Lg and Z₁. Let's examine the partial derivatives of the objective L with respect to Zg and Zj. Using the chain-rule on the formulation in Eq. with g(.) sigmoid(.).
Vzg L = VzL Vz, L (VzL Zs)
- -
These expressions suggest coupled learning process where changes in Z, (through the sigmoid function) directly impact the updates to Zs. and the updates to Z, are scaled by both Zg and the derivative of the sigmoid function. contrast, consider IDPG with bias term of the MLP layer can be viewed as task-specific element, such that the composition is Zs + Z, This approach results updates to Zs and Z, being independent of each other. Such linear-sum operation may fail to capture the complex relationships between the two components that LoPA can capture due LoPA's non-linear factorization.
MLP(X') [38]. Here, the
Experiments, Results, and Discussion
4.1 Experimental Setup
Tasks. We evaluate LoPA on (i) six Natural Language Understanding (NLU) tasks from the GLUE benchmark namely SST-2 BIJ MNLI [37]. MRPC [3], QNLI (2)]. QQP, and RTE (5): (ii) a gederaennration task that requires the model complete method bodies from MBPP benchmark [[]. and (iii) two code- under-sntstnnninn tasks namelyy CruxEval- (input prediction) and CruxEval-O (output prediction) from CruxEval benchmark [6]. These tasks assess the model's 's ability to reason about the execution of simple Python programs by asking it to predict the input given the output of function and vice-versa.
Baselines. We evaluate LoPA against the following baselines that represent different customisation
approaches. (1) FFT, (2) LoRA [12] (3) p-tuning-v2 200] and (4) prefix-tuning (h=512) [16] are selected as representatives of methods that customise models on the server side, with FFT representing full fine-tuning and the remainder showcasing parameter-eeffiiiet techniques. (5) Standard prompt- tuning [1). (6) S-IDPG 38 and (7) DePT [30 are chosen because they customise models from the user side, focusing on soft prompt insertion with DePT also learning updates to the input embedding matrix. For IDPG, we use DNN layers in MLP-head to encode the soft prompt for the input embedding
layer. We chose DNN over PHM layers (39 because we found them to exhibit better performance across our task set. Refer to Appendix 7.1 for the comparison. Additionally, the proposed LoPA also
uses DNN layers, facilitating fair comparison with S-IDPG-DNN (38. Lastly, we include results for the model without any fine-tuning to report its (8) zero-shot performance.
5

---

Code Understanding CruxEval-O
Code Generation
Model
Tuning
#Params
CruxEval-I
MBPP
FFT
350M
33.0
19.5
17.49
LoRA
1.3M
31.0
18.2
21.56
None
0
20.8
15.0
15.85
CodeGen-350M
PT
10.2K
32.8
13.8
15.20
S-IDPG
8.5M
16.9
13.2
17.04
Ours
4.4M
34.5
18.5
17.04
FFT
1.3B
45.0
34.8
44.76
LoRA
4.7M
35.5
36.0
44.14
None
0
26.8
29.8
34.08
DeepseckCoder-1J.3
20.5K
41.2
31.2
34.49
S-IDPG
16.3M
26.0
28.5
42.50
Ours
4.2M
43.0
34.5
44.66
FFT
2.7B
40.2
37.0
55.03
LoRA
7.9M
41.5
42.5
51.54
None
0
33.5
33.0
45.17
Phi2-2.7B
PT
25.6K
35.0
34.0
49.69
S-IDPG
20.3M
35.0
33.0
53.29
Ours
4.76M
43.0
37.2
52.15
FFT
3.8B
39.2
39.5
54.00
LoRA
6.3M
38.0
41.2
42.92
None
0
33.8
39.5
8.82
Phi3-3.8B
PT
30.7K
33.5
31.5
34.08
S-IDPG
24.2M
31.0
39.5
42.29
Ours
5.3M
42.2
42.5
44.35
FFT
7B
OOM
0OM
00M
LoRA
11.8M
47.5
49.8
53.38
None
0
39.3
44.0
50.51
DeepseekCoder-7B
PT
41.0K
45.8
44.8
37.47
S-IDPG
32.1M
40.5
41.5
53,59
Ours
6.35M
50.0
48.0
52.46
FFT
8B
OOM
00M
0OM
LoRA
9.4M
45.5
40.5
44.55
None
27.0
31.5
45.37
Llama3-8B
PT
41.0K
37.5
32.0
26.07
S-IDPG
32
29.2
35.2
33.05
Ours
6.4M
41.2
39.8
43.94
Table Performance comparison on CruxEval and MBPP tasks. We report average pass@l scores. Approaches below the dotted line prompt-cuinng methods, which do require any modification to the model on the server side. Bold denotes the best-performing tuning method for the given model. Underline marks the best result among all prompt-tuning methods. 00M indicates that the corresponding tuning approach exceeded the available GPU memory and ran out of memory.

---

Z =
SST-2
MNLI
MRPC
QNLI
QQP
RTE
Avg
(acc)
(ace)
(acc, F1)
(acc)
(ace, FI)
(acc)
concat(Za, Zs, ZT)
94.50
78.45
76.00
91.43
76.11
84.12
83.44
maxx(Zs: (7,,7,1
95.99
89.37
88.62
93.74
78.44
85.92
88.68
Z, 0 (Z.)
95.99
89.22
91.09
93.74
89.72
83.39
90.53
Table 3: Performance of LoPA on GLUE tasks with respect to function encoding Z. concat(.) operation. We
represents the concatenation of Zy and Z,- max(.) represents the element-wiss report the average of accuracy and F1 for both MRPC and QQP. For all the other tasks, we report accuracy. Bold denotes the best-performing encoding function for LoPA.
Backbone Architectures. For NLU tasks, we test all the tuning methods on 355M RoBERTa 22
similar to the setup opted by prior work [20] (38] 41]. For Code Generation and Understanding. we test subset of the baselines (FFT, LoRA, PT, S-IDPG) on range of FM backbones; starting from
smaller FMs: 350M CodeGen-mono [25] and 1.3B Deepseek-Coder
; mid-sized FMs: 2.7B
phi-2 13 3.8B phi-3 3 124): and larger FMs: 7B Codee and 8B Llama 3 (23
Implementation Details. For the GLUE tasks, we use the train-test splits pre-defined in the benchmark, while for the MBPP and CruxEval tasks, we employ a 50-50 split. Across all tasks and backbone models, the soft prompting baselines are implemented with m 10 virtual tokens representing the soft prompt. For NLU tasks, both IDPG and LoPA use RoBERTa + MLP (h=256) as the encoder, whereas in code tasks, 125M CodeSage MLP (h=124) is used. Additionally, for LoPA, the best-performing rank (r) of the low-rank decomposition is chosen from {1, 2, 4), and the corresponding number of tunable parameters is reported.
Training Configuration. For NLU tasks, training with FFT and LoRA was done for 10 epochs, while with prompt-tuning- based approaches it was done for 20 epochs. MBPP, all foundation model (FM) backbones were trained for 10 epochs across all tuning methods. In CruxEval Tasks across all PEFT methods, FM backbones under 7B were trained for 20 epochs, while larger FMs (27B) were trained for 10 epochs. Lastly, training with FFT on CruxEval tasks was done for 5 epochs. The learning rates for LoPA are set x 10 in NLU and 10 in Coding tasks. The baseline tuning methods use the following learning rates across all the tasks: FFT using 1 x 10-5 LoRA and the remainder of soft-prompting approaches using 1 10-4 All experiments are conducted on 40GB 2xA 100 GPUs.
Evaluation. We report binary multi-class classification accuracies and Fl scores for NLU tasks as provided in the GLUE benchmark. For coding tasks, we report the pass@1 scores computed using the best- pest-performing temperatures: 0.6 for MBPP and 0.2 for CruxEval.
4.2 Baseline Comparison
Performance on Natural Language Understanding. In Table I we can observe that the LoPA consistently outperforms the traditional prompt-tuning method and DePT by average margin of 28.62 points and 25.39 points respectively. This result demonstrates that conditioning the soft prompt instances enables it to influence the model's 's output more significantly. Furthermore, LoPA shows an average improvement of 5.36 points over IDPG, highlighting that the proposed factorisation captures complex relationships between task-specific (Zs)) and instance-specific (Z,) components. This improvement is particularly notable in limited-data settings, with a 12.5-point increase in MRPC and 6. 13-point increase in RTE.
Additionally, LoPA achieves performance close to FFT and LoRA, within point, while using 760k fewer parameters than LoRA owing to LoPA's low-rank decomposition of the soft prompt. This performance profile suggests that the LoPA is a parfmeter-efficient and high- gigh-performing alternative for NLU tasks. It outperforms existing prompt-tuning approaches and performs par with FFT and LoRA, making a compelling choice for efficient and effective model tuning.
Performance on Code Understanding. In Table 2. LoPA consistently improves the pass@1 1 score of the baseline with no tuning across all FM backbones. outperforms prompt-tunin on CruxEval tasks, with modest improvements of approximately 2 to 4 points smaller FMs like CodeGen
7

---

85
100
Relative Number of Parameters (%)
100
Relative Number of Parameters (%6
90.0
GLUE Performance (%)
-80
87.5
80
75
-60
62.5
60
70
60.0
77.5
60
20
75.0
55
10
25
50
100
25
50
Soft Prompt Length (m)
5oft Prompt Length (m)
(a) RTE
(b) MRPC
96
100
Relative Number of Parameters (9%)
100
Relative Number Parameters (9%)
94
GLUE Performance (%6
92
90
60
pass@1
25
86
40
40
84
20
15
20
80
5
10
25
50
100
5
10
25
50
100
Soft Prompt Length (m)
Soft Prompt Length (m)
(c) SST-2
(d) CruxEval-O
S-IDPG
PT
Ours
LoRA
FT Porams
Ours Params
S-IPG Params
Figure 3: Performance comparison of baselines as function of m on (a)-(c) GLUE benchmark and (d) CruxEval-O (with DeepseekCoder- 1.3B as FM). Tunable parameters shown relative the method with the most. Higher performance and fewer parameters indicate better results.
and DeepSeek 1.3B and larger improvements ranging from points on larger FMs like LLama-3 and Phi-3 in CruxEval-O. Furthermore, IDPG performs worse than PT for all models except Phi-3 in CruxEval-O. These results suggest that merely encoding an instance-sensitive soft prompt does not guarantee improvements and can degrade performance (e.g.. IDPG on CodeGen and DeepSeekCoder- Co. in CruxEval tasks). The poor generalization of the learned soft prompt, possibly due over-parand parameterization and resulting overfitting, might explain this behaviour. In contrast, LoPA, being more crrammttr--fffcienn explicitly incorporates task and instance information in its design of the soft prompt, leading to better performance.
LoPA also performs on par with LoRA, often within a range of to points of pass@1, while roughly using two-thirds of the parameters. Notably, LoPA outperforms LoRA across all models in CruxEval-I, except for LLama-3, with improvements approximately ranging from 2 points in DeepSeckCoder-7B to 4 points in Phi-3. This result might be attributed to the fact that many of the FMs considered here are good knowledge approximators, well-trained on diverse datasets, and demonstrate strong zero-shot generalization. Directly updating subset of their weights to catastrophic forgetting, where the models lose previously acquired knowledge [27]. such cases, soft prompting, employed by LoPA, can effectively elicit the necessary skills to solve tasks without compromising existing knowledge (26).
still lead
Performance on Code Completion. On the code completion task of MBPP, both IDPG and LoPA improve performance of the baseline model with nearly equal gains except for LLama-3. However, LoPA achieves this with significantly fewer tunable appprttiatsappoiitttlllyyel half the number used

---

1.6
94
1.4
42
GLUE Performance (%)
92
1.2
Para Ratametee (In millions
10
90
1.0
passip1
68
0,0
86
0.6
84
0.4
62
0.2
80
0.0
Rank (n)
Rank (r)
MRPC
RTE
SST-2
Params
Params/dd-1133 1.31
Peramsiph--2)
(a) NLU
(b) CruxEval
Figure 4: Performance of LoPA function of rank shown for m 10. (a) GLUE Benchmarks and (b) CruxEval tasks O) where ds-1.3 denotes DeepseckCoder- 1.3B and phi-2 denotes Phi2-2.7B models. Higher performance and fewer tunable parameters indicate better results.
in CodeGen and only one-fifth those used by IDPG in DeepscckCoder-765 7b. This demonstrates that LoPA scales well with the size of the foundation model, maintaining both performance and parameter efficiency. This efficiency is attributed to the low-rank approximation of the instance-specific matrix employed LoPA. For LLama-3, all tuning approaches led to drop in performance, possibly due to over-fitting, suggesting that LLama-3 might already be trained on MBPP
Overview of Results. Averaged over all tasks and foundation models, LoPA showed relative 52% over PT and 20. 16% over IDPG, while being outperformed by LoRA by only 0.54%. Notably, LoPA outperformed LoRA in 11 out of 24 cases. Thus, in the test cases we considered, there was no clear systematic advantage to LoRA in terms accuracy. Given task-specific processing at the server-the the prompt
percentage improvements
that LoPA requires even at the client, before being sent to the server for processing we believe LoPA may be a useful alternative to LoRA for some tasks.
be computed anywhere,
4.3 Ablation Study
Performance of LoPA as function of soft-prompt length. In Figure we study how the length of the soft prompt impacts the performance of LoPA compared to other prompting methods. Increasing the length of the soft prompt corresponds adding more vectors the set that represents the soft prompt, thus expanding the offset subspace (See Eq. 2 Whether the added vectors are mutually independent and provide additional useful information depends on the tuning approach learning them and the offset subspace of the FM.
For instance, PT and IDPG initially see performance improvements with increased prompt length, but performance eventually plateaus or drops due to over-fitting (See PT on SST-2 Fig. Bc and IDPG on CruxEval-O Fig. 3d In contrast, LoPA does exhibit significant performance fluctuations with varying prompt lengths (See Fig. 3a 3d). This stability is likely due to the shared component Zş acting as regularizer, preventing wittingting of the restance-speciffe soft prompts.
Morcover, LoPA with m = outperforms PT and IDPG even when they use longer prompts (m > 5) This result suggests that the dimension of the offset subspace much smaller, and LoPA can more accurately represent with its learned vectors.
(Refer Fig. 3a 3b
Performance of LoPA a function of rank. In Figure the instance-specific component affects the perfor-
examine how the rank of
the proposed low-rank decomposition mance. For NLU, we consider three tasks: MRPC, RTE, and SST-2, and observe the per- formance of RoBERTa as the rank increases from to 4 Performance consistently im- proves with increasing rank, showing gains of 1% 2% for SST-2 and up to 8% for MRPC.

---

In contrast, for CruxEval tasks, increasing the rank does not proportionally improve the per- formance. We attribute this behaviour to the size of the datasets used to approximate the low- rank matrices. NLU tasks provide thousands of samples, allowing for better approximation of higher-order matrices. However, CruxEval has only a few hundred samples, and increasing the rank introduces more parameters, possibly leading over-fitting.
Encoder
CruxEval-I 41.2
CruxEVal-O
CodeBert( (22MM)
32.8
CodeSage(130M))
43.0
34.5
CodeSage(365M)
42.2
34.5
Figure 5: Ablation for Encoder in LoPA with
DeepseckCoder-Ile as the foundation model.
Performance of LoPA a function of encoder network. In Figure 5 we study the impact on the performance by choosing different transformer backbones for the Encoder network in LoPA. For this experiment, we use 125M CodeBERT 44, and 130M and 365M CodeSAGE 40 encoder models to generate input encodings. X' for the CruxEval tasks. We observe that CodeSAGE models achieve a 2-point improvement over CodeBERT in both tasks. This improvement can be attributed to CodeSAGE's superior pre-training using Contrastive Learning, which allows for finer distinctions in code representations. Consequently, the soft-prompt vectors {z*] in LoPA, as functions of the input X', capture instance-specific nuances more effectively and accordingly exert influence on the model's 's output. We also tried tuning the encoder model while learning soft prompts but did not find any significant improvements in the performance.
Performance LoPA with respect function encoding Z. In Table we evaluate the effect of different functions encoding Z. as a combination of 7,, and Z,s on RoBERTa's performance in NLU tasks. The results show that simply concatenating Z.s and Z, performs the worst while the non-linear functions, such as max(.) and the proposed gating mechanism, yield the best performance. We leave the exploration of other non-linear functions for future work.
Conclusion
In this paper, introduced Low-Rank Prompt Adaptation (LoPA), instance-specific soft- prompting method that outperforms other methods in the prompt-tuning family, and performs on par with full fine-tuning and LoRA, while using fewer tunable parameters. LoPA first uses a low-rank approximation of the instance-specific soft prompt and combines it with a task-specific soft prompt via gating function. With more informed way of designing soft prompts, this work aims position prompt tuning as powerful alternative to adapter-but based methods for user-specific customization of foundation models.
Limitations and Future Work. The main limitation of LoPA is that its effectiveness as an alternative to LoRA was demonstrated user tasks where LoRA or even full fine-tuning might be necessary. Further investigation into its performance real-world tasks is part of our future work. In this work, we assumed the learned soft prompt to be prepended as a prefix to the input. Future research could explore the effects of positioning it suffix or randomly within the input. Finally, the foundation model combined with LoPA can be viewed as a Conditional Auto-Encoder, where soft prompt vectors exist in a latent subspace rather than an offset subspace. This viewpoint raises intriguing questions, such as whether the observed performance improvements result from inferring and compressing task-specific knowledge and providing it as additional information. Investigating this alternate perspective could lead to further performance enhancements by developing more sophisticated auto auto-encoding systems.
set of benchmark tasks, but this may not hold for obseure real-life
Broader Impact. Our contribution to new knowledge is the development of language-modes cus- tomization method that delivers strong performance while being aarameter-efficient The significance of our work lies in its potential to reduce training and maintenance costs associated with hosting and customizing foundation models. Furthermore, our method enhances user privacy by enabling task-specific customization on the user end rather than the server end.
6
Acknowledgments
This research was supported by the NSF under grant numbers CCF1918651, 2008240, 2131294, and 2212557, NIH CTSA award No. ULITRO03166 167, and US DOT Tier- UTC CYBER-CARE grant #69A3552348332.
10

---

References
[1] Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoe Le, al. Program synthesis with large language models. arXiv preprint (tt::2008007332 2021.
[2] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 43:1877-1900 2020.
[3] Bill Dolan and Chris Brockett. Automatically constructing a corpus of sentential paraphrases. In Third international workshop on paraphrasing (/WP2005). 2005.
[4] Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin. Ting Liu, Daxin Jiang, ct al. Codebert: A pre-trained model for programming and natural languages. arXiv preprint 20xx:2002.081555 2020.
[5] Danilo Giampiccolo. Bernardo Magnini, Ido Dagan, and William B Dolan. The third pascal recognizing textual entailment challenge. In Proceedings of the ACL-PASCAL workshop textual entailment and paraphrasing, pages 1-9, 2007.
[6] Alex Gu, Baptiste Rozière, Hugh Leather, Armando Solar-Lezama Gabriel Synnacve, and Sida Wang. Cruxeval: benchmark for code reasoning, understanding and exccution. arXiv preprint arXiv:4401.0306. 2024.
[7] Daya Guo, Qihao Zhu, Dejian Yang. Zhenda Xie, Kai Dong, Wentao Zhang. Guanting Chen, Xiao Bi, Wu, YK Li, et al. Deepscek-coder. When the large language model meets programming-the the rise of code intelligence. arXiv preprint arXiv:2401. (4t.:2401..1196, 2024.
[8] Karen Hambardzumyan, Hrant Khachatrian, and Jonathan May. Warp: Word-level adversarial reprogram- ming, arXiv preprint arXiv:2101.00113 101.00/1,, 2021.
[9] Zeyu Han, Chao Gao, Jinyang Liu, Sai Qian Zhang, et al. Parameter-efficient fine-tuning for large models: A comprehensive survey. arXiv preprint arXiv:2403. 14608, 2024.
[10] Junxian He, Chunting Zhou, Xuezhe Ma. Taylor Kirkkarrka,,,c,, and Graham Neubig. Towards a unified view of parameter-effiiient transfer learning. arXiv preprint arXiv:21 10,04366, 2021.
[11] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Ges- mundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International conference machine learning, pages 2790-2799. PMLR, 2019.
[12] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.0966 09685, 2021.
[13] Mojan Javaheripi. Sébastien Bubeck, Marah Abdin, Jyoti Aneja, Sebastien Bubeck, Caio César Teodoro Mendes, Weizhu Chen, Allie Del Giomo, Ronen Eldan, Sivakanth Gopi, et al. Phi-2: The surprising power of small language models. Microsoft Research Blog, 2023.
[14] Rabech Karimi Mahabadi, James Henderson, and Sebastian Ruder. Compacter: Efficient low-rank hypercomplex adapter layers, Advances in Neural Information Processing 3ystems,34:1022 14:1222-0 2021.
[15] Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-officien prompt tuning. arXiv preprint
2021.
[16] Xiang Lisa Li Percy Liang. Prefix- wreii-tuning: Optimizing continuous prompts for generation. arXiv preprint Xiv:2110 00190, 2021.
[17] Haokun Liu, Derek Tam, Mohammed Mugeeth, Jay Mohta, Tenghao Huang. Mohit Bansal, and Colin Raffel. Few-shot parameter-efficicnt fine-tuning is better and cheaper than in-context learning. Advances in Neural Information Processing Systems, 35:1950-1990 1965, 2022.
[18] Shih- Yang Liu, Chien-Yr Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang. Kwang- Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation. arXiv preprint 2mXiv:2402.0900 2024.
[19] Xiangyang Liu, Tianxiang Sun, Xuanjing Huang, and Xipeng Qiu. Late prompt tuning: A late prompt could be better than many prompts. arXiv preprint arXiv:2210.1129.. 2022.
11

---

[20] Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Lam Tam, Zhengxiao Du, Zhilin Yang. and Jie Tang. P-tuning v2: Prompt tuning can be comparable fine-tuning universally across scales and tasks. arXiv preprint arXiv:2110.07000 2021.
[21] Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujic Qian, Zhilin Yang, and Jie Tang. Gpt understands, too. Al Open, 2023.
[22] Yinhan Liu, Myle Ott. Naman Goyal. Jingfei Du, Mandar Joshi, Dangi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyerr and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint /97 2019.
[23] Meta.
Maamaa/ 2024.
[24] Microsoft. http:///wwatttt.oooooooooooooata 2024.
[25] Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. Codegen: An open large language model for code with multi-turn program synthesis. arXiv preprint arXiv:2203. 13474, 2022.
[26] Aleksandar Petrov, Philip HS Torr. and Adel Bibi. When do prompting and prefix-tuning work? theory of capabilities and limitations. arXiv preprint arXiv:2310. 19698, 2023.
[27] Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. Adaptcrfusion Non-destructive task composition for transfer learning. arXiv preprint rxXi:2005500247 2020.
[28] Guanghui Qin and Jason Eisner. Learning how to ask: Querying Ims with mixtures of soft prompts. arXiv preprint wrwii:2204.0066550
2021.
[29] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Pency Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint (60.:5660000550,, 2016.
[30] Zhengxiang Shi and Aldo Lipani. Dept: Decomposed prompt tuning for parameter-efficien fine-tuning. arXiv preprint 2xXi:2330900517 2023.
[31] Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang. Christopher D Manning, Andrew Y Ng. and Christopher Potts. Recursive deep models for semantic compositionality over sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1631-1642, 2013.
[32] Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju. Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, al. Gemma: Open models based on gemini research and technology. arXiv preprint prXiv:240300220 2024.
[33] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems. 30, 2017.
[34] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv: 1804.0746), 2018.
[35] Yeming Wen and Swarat Chaudhuri. Batched low-rank adaptation of foundation models. arXiv preprint axXiv:2312.05677, 2023.
[36] Yeming Wen and Swarat Chaudhuri. Batched low-rank adaptation of foundation models. In ICLR, 2024.
[37] Adina Williams, Nikita Nangia, and Samuel R Bowman, A broad-coverage challenge corpus for sentence understanding through inference. arXiv preprint 1oX.v::704.03420 2017.
[38] Zhuofeng Wu. Sinong Wang, Jiatao Gu, Rui Hou, Yuxiao Dong, VG Vydiswaran, and Hao Ma. Idpg: An instance-dependent prompt generation method, arXiv preprint orXiv:2204.04499 2022.
[39] Aston Zhang. Yi Tay. Shuai Zhang, Alvin Chan, Anh Tuan Luu, Siu Cheung Hui, and Jie Fu. Beyond fully-connected layers with quaternions: Parameterization of hypercomplex multiplications with 1/n parameters. arXiv preprint 1rXiv:2102.08520 2021.
[40] Dejiao Zhang, Wasi Ahmad, Ming Tan, Hantian Ding, Ramesh Nallapati, Dan Roth, Xiaofei Ma, and Bing Xiang. Code representation learning at scale. arXiv preprint arXiv:2402.00 2024.
[41] Wei Zhu and Ming Tan. Spt Learning to selectively insert prompts for better prompt tuning. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023.
12

---

7 Appendix
7.1
Comparison with Parameterized Hypercomplex Multiplication (PHM) layers
Tuning Method
Tunable
SST-2
MNLI
MRPC
QNLI
QQP
RTE (acc)
Avg
Params
(acc)
(acc)
(acc, F1)
(acc)
(acc, FI)
LoPA
1.60M
95.99
89.22
91.09
93.74
89.72
83,39
90.53
S-IDPG
FC
2.89M
95.30
84.50
78.60
90.48
84.88
77.26
85.17
PHM (n=8)
0.37M
95.07
83.46
76.12
85.45
77.35
67.14
80.77
PHM (n=16)
0.20M
94.61
83.66
76.68
81.16
80.39
65.34
80.31
PHM (n=32)
0.17M
94.72
81,45
74.99
84.59
81.90
68.23
80.98
Table 4:
Performance GLUE tasks. We report the average of accuracy and F1 for both MRPC and QQP. For all the other tasks, we report accuracy. Bold denotes the best- nest-performing tuning method for the given model. FC represents the fully-connected layers and PHM represents the hypercomplex multiplication layers parameterised by n..
In this section, we consider PHM layers as an alternative to low-rank decomposition in LOPA to reduce parameter complexity. We carry layers as Z. PHMw (X') with = >. A: X Br, where X represents the Kroneckr product between learnable matrices A,, B; and represents the hyper-paramett of factorisation in the Kronecker product. We can observe in Tahle while PHM layers reduce parameters, they also lead to significant performance drop of approximately 10 points average compared to LOPA. This drop may be due to the structural bias in W imposed by Kronecker factorisation of PHM layers. which could limit expressiveness [39] in comparison Pully-Connecee layers DNN. want to point out that for further reduction in trainable parameters, LoPA can also be used in conjunction with PHM layers.
ablation study NLU tasks where implement IDPG with PHM
halancing the parameter complexity extent
7.2 Convergence Analysis of Soft-Prompting Approaches
We present plots in Figures 6 S comparing the training loss and performance on NLU tasks (QQP, QNLI, MNLI) for Prompt Tuning (PT), IDPG, and LOPA. The results show that instance-dependent methods like IDPG and LOPA converge faster than traditional prompt tuning. Moreover, LOPA converges faster and achieves higher accuracy Fl scores compared IDPG.
2.4
154k
2044
(a) Training Loss
(b) Validation Accuracy
Figure 6: Convergence plots for the PEFT approaches Prompt-tuning (PT), IDPG and the proposed LOPA on the NLU task QQP.
13

---

Step
2004
2004
150k
200k
Training Loss
(b) Validation Accuracy
Figure 7: Convergence plots for the PEFT approaches Prompt-tuning (PT), IDPG and the proposed LOPA on the NLU task MNLI.
spectis Ios SL Siiteoiie
Stp
Stp
Training Loss
(b) Validation Accuracy
Figure 8: Convergence plots for the PEFT approaches Prompt-tuning (PT), IDPG and the proposed LOPA on the NLU task QNLI.
14

---

NeurIPS Paper Checklist
1. Claims
Question: the main claims made the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes, the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match thcorctical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Yes, the limitations are discussed in the conclusion section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed the paper.
The authors are encouraged create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results these assumptions (e.g.. independence assumptions, noiseless settings, model well well-specifiennt specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
violations of
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend implicit assumptions, which should he articulated.
The authors should reflect the factors that inffuence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
The paper does not present any theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
15

---

model The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by
Formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We include all the necessary set of parameerrsmeters and training guidelines to replicate the results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
If the contribution is datasct and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, contribution novel architecture, describing the architecture fully might suffice, or the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., the case of a large language model), releasing checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require relcasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution a new model (e.g.. a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (c.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, which case authors are welcome to describe the particular way they provide for reprodueibility. In the case of closed-source source models, it may be that access to the model is limited in some way (c.g.. to registered users). but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code. with sufficient instructions faithfully reproduce the main experimental results, as described supplemental material?
Answer:
[Yes]
Justification: The code is publicly released on Github.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurTPS code and data submission guidelines (https://nips... httppa//nnips.ccppuublls cctpublic///pp///ccccccccccc htt////ooSuuiieiiioooo
for more details.
While we encourage release of code and data, we understand that might not possible,
so "Na" an acceptable answer. Papers cannot rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source henchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: : ://nis///.////////// cc/public/ oui.dess/Cooeeuttmmiesssooooooo for more details.
16

---

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. If only subset of experiments are reprodueible, they should state which ones are omitted from script and why.
At submission time, to preserve anonymity, authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g.. data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: Yes, the hyper-parametee are discussed to replicate results.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined other appropriate informa- tion about the statistical significance of the experiments?
Answer: [No]
Justification: expensive.
Running multiple experiments on many large language models across all tasks is
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some paramcter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK report I-sigma enror bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, authors should be careful not to show in tables or figures
symmetric error hars that would yield results that are out of range (c.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
Experiments Compute Resources
Question: For each experiment. does the paper provide sufficient information the computer resources (type of compute workers, memory, time exccution) needed reproduce the experiments?
Answer: [Yes]
Justification:
The amount of computer resources used is discussed in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
17

---

The paper should provide the amount of compute required for each of the individual experimental well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform. every respect, with NeurlPS Code of Ethics
Answer:
[Yes]
Justification:
Research adheres to NeurIPS Code Ethics.
Guidelines:
The answer means that the authors have not reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain special circumstances that require deviation from the Code of Ethics.
authors should make sure preserve anonymity (e-g., if there is special consideration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss the potential societal impact of our work in the conclusion section.
Guidelines: Broader impact is discussed the paper.
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative socictal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deploy- of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will he foundational research and not tied particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example. it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it not needed point out that generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms could arise when technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitor- ing misuse, mechanisms to monitor how system learns from feedback time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place responsible release data or models that have a high risk for misuse (e.g.. pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
No new data or model is released.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example hy requiring that users adhere to usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
18

---

We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
Models and datasets with their licenses used are respected and cited.
Guidelines:
The answer means that the paper does not use existing assets.
authors should cite the original paper that produced the code package or dataset.
The authors should state which version the asset is used and, possible, include URL.
The name of the license (e.g.. g.. CC-BY 4.0) should be included for each asset. For scraped data from particular source (e.g.. website), the copyright and terms of service of that source should he provided.
If assets released, the license, copyright information, and terms of use the package should be provided. For popular datasets, paaperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new
Guidelines:
The NA means that the paper does release
Researchers should communicate the details of the dataset/code/model part of their sub- missions structured templates. This includes details about training. license, limitations, etc.
The paper should discuss whether how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Neither crowdsourcing nor research with human subjects was done,
Guidelines:
answer NA means that paper does not involve crowdsoureing nor research with human subjects.
Including this information in supplemental material is fine, but main contribution of paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Beard (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants. whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer:[NA]
19

---

Justification:
Neither crowdsourcing nor research with human subjects was done.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applica- ble), such as the institution conducting the review.
20


Reviews:
[{"review_id": "IlbUUATE85", "reviewer": "Reviewer_4fPU", "summary": "The paper introduces Low-Rank Prompt Adaptation (LOPA), a novel parameter-efficient fine-tuning (PEFT) approach that improves soft prompt tuning, delivering performance on par with LoRA and full fine-tuning methods. LOPA addresses scalability issues of traditional PEFT methods by using a low-rank decomposition for instance-specific soft prompts. This technique combines task-specific and instance-specific prompts through a gating function, offering a balance between customization and parameter efficiency. The paper demonstrates LOPA's effectiveness across various natural language understanding and code generation tasks, positioning it as a competitive alternative to adapter-based methods.", "questions": "LOPA constructs the sort prompt as $𝑍 = 𝑍_𝑆∘𝑔(𝑍_𝐼)$. Did you try an additive approach by concatenating the two vectors instead?", "limitations": "The author discussed several limitations of their approaches, including the effectiveness of LOPA on practical tasks, the assumed positioning of the learned soft prompt, and the need for further exploration of LOPA as a conditional auto-encoder.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper is well-written.\n* The proposed method is simple and the authors demonstrate its effectiveness across various natural language understanding and code generation tasks.\n* The authors provide a thorough analysis to understand the relative importance of several aspects of their approach.", "weaknesses": "I felt there were several obvious questions left unexplored, noted below, which raise concerns regarding the significance of the paper's contributions.\n\n* The authors only experimented with a rather small model, i.e., 355M RoBERTa, for classification tasks while trying much larger models (up to 8B) for code generation tasks, which raises concerns about whether the proposed method works with larger models for classification tasks.\n\n* The authors focused solely on a few classification tasks and two code generation tasks. This raises concerns about the proposed approach's effectiveness for other tasks, like open-ended generation, where prompt tuning often underperforms (An et al., 2022).\n\n* Finally, I am concerned about the practical adoption of the proposed approach since it is unclear whether the proposed approach performs better than LoRA generally.\n\nReferences:\n\nAn et al., 2022: https://arxiv.org/pdf/2203.03131", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "- *“The authors only experimented with a rather small model, i.e., 355M RoBERTa, for classification tasks while trying much larger models (up to 8B) for code generation tasks, which raises concerns about whether the proposed method works with larger models for classification tasks.”*\n\n  **Response:** Thank you for the observation. For natural language tasks, it is established that prompt tuning becomes competitive and comparable to fine-tuning with large models (>1B) (Lester et al., 2021; Liu et al., 2021). Therefore, similar to other recent works (Liu et al., 2021; Wu et al., 2022; Zhu et al., 2023), our focus is improving prompt tuning efficacy for medium-sized models (100M to 1B). Conversely, we experimented with much larger models (up to 8B) for code generation tasks, as the impact of prompt tuning methods in this area has not been extensively studied.\n\n&nbsp;\n\n- *“The authors focused solely on a few classification tasks and two code generation tasks. This raises concerns about the proposed approach's effectiveness for other tasks, like open-ended generation, where prompt tuning often underperforms (An et al., 2022).”*\n\n  **Response:** To address this concern, we conducted experiments on the standard E2E and WebNLG benchmarks for open-ended natural language generation. We followed the hyper-parameter setup of Hu et al. (2021) and fine-tuned GPT2-medium with LoRA, Prompt Tuning (PT), and our approach.\n\n  | | | | E2E | | |\n  | -------- | ------- | ------- | ------- | ------- | ------- | \n  | **Approach** | **BLEU $\\uparrow$** | **NIST $\\uparrow$** | **METEOR $\\uparrow$** | **ROUGE-L $\\uparrow$** | **CIDEr $\\uparrow$** | \n  | LoRA | 68.78 | 8.81 | 46.52 | 71.36 | 2.49 |\n  | PT(m=100) | 32.98 | 0.65 | 27.54 | 57.04 | 0.76 |\n  | Ours | 65.85 | 8.39 | 43.10 | 68.65 | 2.27 |\n\n  | | | | WebNLG | | | |\n  | -------- | ------- | ------- | ------- | ------- | ------- | ------- |\n  | **Approach** | **BLEU-U $\\uparrow$** | **BLEU-S $\\uparrow$** | **BLEU-A $\\uparrow$** | **TER-U $\\downarrow$** | **TER-S $\\downarrow$** | **TER-A $\\downarrow$** |\n  | LoRA | **46.89** | **63.27** | **55.85** | 0.45 | **0.33** | **0.39** |\n  | PT(m=100) | 29.59 | 31.98 | 30.94 | 0.54 | 0.54 | 0.54 |\n  | Ours | 44.78 | 55.46 | 50.65 | **0.44** | 0.37 | 0.40 |\n\n  The results show that while standard prompt tuning underperforms, our method significantly outperforms PT and closely matches LoRA's performance on both benchmarks. For WebNLG, we further report results across seen(S), unseen(U), and all(A) categories. Our approach demonstrates strong extrapolation performance on unseen WebNLG categories (see BLEU-U and TER-U), indicating its ability to handle diverse domains in the data without server-side personalization of the foundation model. This suggests that our method is also effective in open-ended generation scenarios. We will include this benchmark comparison and more baseline results in the paper.\n\n&nbsp;\n\n- *“Finally, I am concerned about the practical adoption of the proposed approach since it is unclear whether the proposed approach performs better than LoRA generally.”*\n\n  **Response:** True, we find no significant difference between LoRA and LOPA in terms of accuracy. However, LOPA has two key advantages. First, all other things being equal, a purely prompt-based method (such as LOPA) is preferable to one that requires integrating an adaptor with the model at the server (such as LoRA). LOPA allows the model to be specialized at the client (or via the use of a middleware), without modification at the server. LOPA does not require that any use-case-specific parameters be stored on the server, which can be costly, especially if the number of specializations is large. Second, we find that LOPA is more parameter-efficient than LoRA.\n\n&nbsp;\n\n- *“LOPA constructs the sort prompt as 𝑍=𝑍𝑆∘𝑔(𝑍𝐼). Did you try an additive approach by concatenating the two vectors instead?”*\n\n  **Response:** We experimentally found that the non-linear composition of $Z$ via $𝑍=𝑍_𝑆∘𝑔(𝑍_𝐼)$ is crucial for the performance gains observed with LOPA. See the following ablation study on a subset of NLU tasks, where we observe LOPA_add that opts for an additive approach underperforms.\n\n  | Approach | Params$\\downarrow$ | RTE$\\uparrow$ | MRPC$\\uparrow$ | SST-2$\\uparrow$ |\n  | -------- | ------- | ------- | ------- | ------- |\n  | LOPA(r=4) | 1.60M | **83.39** | **91.09** | **95.99** |\n  | LOPA_add(r=4) | 1.60M | 64.26 | 75.17 | 93.34 |\n  \n  We appreciate your feedback and will include these comparisons and numbers on the remaining tasks in the final manuscript.\n\n*References:*\n \n&nbsp;Lester, Brian, Rami Al-Rfou, and Noah Constant. \"The power of scale for parameter-efficient prompt tuning.\" arXiv preprint arXiv:2104.08691 (2021).\n\n&nbsp;Liu, Xiao, et al. \"P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks.\" arXiv preprint arXiv:2110.07602 (2021).\n\n&nbsp;Wu, Zhuofeng, et al. \"Idpg: An instance-dependent prompt generation method.\" arXiv preprint arXiv:2204.04497 (2022).\n\n&nbsp;Zhu, Wei, and Ming Tan. \"SPT: learning to selectively insert prompts for better prompt tuning.\" Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023.\n\n&nbsp;Hu, Edward J., et al. \"Lora: Low-rank adaptation of large language models.\" arXiv preprint arXiv:2106.09685 (2021).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KSWxrn3Vqi", "reviewer": "Reviewer_RWB9", "summary": "The paper introduces Low-Rank Prompt Adaptation (LOPA), an instance-aware prompt tuning-based approach. LOPA constructs soft prompts from a task-specific component (shared across samples) and an instance-specific component (unique to each sample), combining them using a gating function. It employs a low-rank decomposition of the instance-specific component to enhance parameter efficiency. Unlike Parameter-Efficient Fine-Tuning (PEFT) methods like LoRA, LOPA does not need to store adapter-like modules for each task as it is based on prompt tuning. The paper evaluates LOPA on natural language understanding and code tasks to demonstrate its effectiveness.", "questions": "Please refer to the weaknesses section.", "limitations": "NA", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. **Parameter Efficiency:** More efficient than traditional PEFT methods like LoRA.\n2. **No Server-Side Changes:** Once trained, LOPA's soft prompts can be used as input prefixes without additional server-side computational cost.", "weaknesses": "Major weaknesses of the paper is its lack of novelty. \n\n1. The proposed approach is quite similar to the IDPG method referenced in [36]. The IDPG approach also uses both instance-specific and task-specific prompts. While it is true that in IDPG, updates to $Z_{S}$​ and $Z_{I}$​ are independent of each other, this issue can be addressed by adding a non-linearity after the second layer in their prompt generator network. Additionally, to reduce the parametric complexity of $Z_{I}$​​, the authors have used a low-rank decomposition of $Z_{I}$​, similar to LoRA. Similarly, the IDPG paper uses Parameterized Hypercomplex Multiplication (PHM) Layers to reduce the complexity of $Z_{I}$​. There is no analysis in the paper comparing the novelty and benefits of low-rank decomposition, as in LoRA, to PHM Layers. \n\n2. Additionally, the performance of LoPA is inferior to LoRA on 6 out of 7 datasets in Table 1.\n\n3. Missing important experimental details: The paper does not indicate how many epochs each of the methods was trained. \n\n**Questions and Suggestions:**\n\na. Line 102: The paper states, \"IDPG [36], which emphasizes an instance-specific prompt.\" Also, on Line 156: \"existing instance-specific approaches [36].\" In general, IDPG uses both instance-specific and task-specific prompts. Therefore, referring to it solely as an instance-specific approach may not be correct.\n\nb. Lines 138-140: \"However, encoding a matrix of size d×m can be expensive.\" Why not use a linear layer of dimension n×m as an encoding function f? This might not be as expensive.\n\nc. How different and efficient is the low-rank decomposition of $Z_{I}$​​ compared to the Parameterized Hypercomplex Multiplication (PHM) Layers proposed in the IDPG paper (Section 3.2.1)? PHM layers also optimize the prompt generator network.\n\nd. There is no discussion about the convergence of the proposed method, as prompt tuning is known for its slower convergence. It will good to show convergence of proposed method. \n\ne. Line 122: $z_{k}$ should be $z^{k}$.\n\nf. Line 197: \"Evaluation.For\" -> space is missing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "- *\"The proposed approach is quite similar ... decomposition, as in LoRA, to PHM Layers.\" \"How different and efficient is the ...also optimize the prompt generator network.\"*\n\n  **Response:** Thank you for the detailed comparison. Here are the key distinctions and considerations:\n  1. **Encoding of $Z_S$ and $Z_I$:** The encoding of $Z_S$ and $Z_I$ differs fundamentally between IDPG and LOPA. In IDPG, $Z_S$ is the bias term from the last layer of the prompt generator, constructing $Z = Z_S + Z_I$. In LOPA, $Z$ is constructed as $Z = Z_S \\circ g(Z_I)$, making $Z_S$ and $Z_I$ co-dependent through a gating function $g(.)$. We experimentally found this non-linear interaction crucial for the performance gains observed with LOPA, which was absent in existing soft-prompt-based learning approaches like IDPG, PT, etc.\n  2. **PHM Layers vs. Low-Rank Decomposition:** We acknowledge that PHM layers can reduce parameter complexity, similar to low-rank decomposition in LOPA. We have carried out an ablation study using PHM in IDPG to construct $Z_I$ with $n$ representing the hyper-parameter balancing the parameter complexity and extent of factorisation in the Kronecker product $W = \\sum_i^n A_i \\bigotimes B_i$. Our ablation study on three NLU tasks shows that while PHM layers reduce parameters, they also lead to a significant performance drop (~15pt in RTE and MRPC) compared to LOPA. This drop may be due to the structural constraints of PHM layers imposed by Kronecker factorisation, which could limit expressiveness (Zhang et al 2021).\n  | Approach | Params$\\downarrow$ | RTE$\\uparrow$ | MRPC$\\uparrow$ | SST-2$\\uparrow$ |\n  | -------- | ------- | ------- | ------- | ------- |\n  | LOPA(r=4) | 1.60M | **83.39** | **91.09** | **95.99** |\n  | IDPG+FC | 2.89M | 77.26 | 78.60 | 95.30 |\n  | IDPG+PHM (n=8) | 0.37M | 67.14 | 76.12 | 95.07 |\n  | IDPG+PHM (n=16) | 0.20M | 65.34 | 76.68 | 94.61 |\n  | IDPG+PHM (n=32) | 0.17M | 68.23 | 74.99 | 94.72 |\n  3. **Future Work:** We want to point out that PHM can be used in conjunction with low-rank decomposition in LoRA to reduce trainable parameters further. See below for a comparison of trainable parameter complexity. We can observe that while using either PHM or FC layers, LOPA can still be more parameter efficient by a factor of $(\\frac{r}{d}+\\frac{r}{m})$ (refer to Sect. 3.3 for notations). This is an interesting experiment that we leave for future work.\n   - IDPG + FC : $\\mathcal{O}(hdm)$\n   - LOPA + FC : $\\mathcal{O}(hdm(\\frac{r}{d}+\\frac{r}{m}))$\n   - IDPG + PHM : $\\mathcal{O}(n^3 + \\frac{hdm}{n})$\n   - LOPA + PHM : $\\mathcal{O}(n^3 + \\frac{hrm}{n} + n^3 + \\frac{hrd}{n}) = \\mathcal{O}(n^3 + \\frac{hdm}{n}(\\frac{r}{d}+\\frac{r}{m}))$\n\n&nbsp;\n- *\"Additionally, the performance of LoPA is inferior to LoRA on 6 out of 7 datasets in Table 1.\"*\n\n  **Response:** This is true. However, the performance difference in Table 1 is marginal (< 1% in 5 of 6 tasks where LoRA is better). Given the additional 18 cases in Table 2, it is even less clear that there is any meaningful advantage to LoRA in terms of accuracy, as LOPA outperformed LoRA in 11 out of 24 cases. Given the other advantages of LOPA (parameter efficiency, no need for deployment on the server), we argue that the method has significant value.\n\n&nbsp;\n- *“Missing important experimental details: The paper does not indicate how many epochs each of the methods was trained.”*\n\n  **Response:** Thank you for pointing this out. Here are the training details. In NLU Tasks, FFT and LoRA were trained for 10 epochs, while prompt-tuning approaches were trained for 20 epochs. In MBPP, all methods were trained for 10 epochs across all foundation model (FM) backbones. In CruxEval Tasks, for FM backbones under 7B, PEFT approaches were trained for 20 epochs, while larger FMs (≥7B) were trained for 10 epochs. FFT on CruxEval tasks for FM backbones under 7B was trained for 5 epochs. We will include these details in the final manuscript.\n\n&nbsp;\n- *“Lines 138-140: \"However, encoding a matrix of size d×m can be expensive.\" Why not use a linear layer of dimension n×m as an encoding function f? This might not be as expensive.”*\n\n  **Response:** The IDPG baseline considered in the paper indeed uses a linear layer of dimension $d \\times m$ as an encoding function $f$. Resultantly, it is more computationally expensive than LOPA’s encoding of $f$. Consider the following - If the input features have dimension h, the parameter complexity of $f$ in IDPG is $\\mathcal{O}(hdm)$. In comparison, LOPA uses two low-rank factors encoded with linear layers of sizes $d \\times r$ and $m \\times r$, reducing the parameter complexity to $\\mathcal{O}(hdm(\\frac{r}{d}+\\frac{r}{m}))$ by a factor of $\\frac{r}{d}+\\frac{r}{m} < 1$.\n\n&nbsp;\n- *“There is no discussion about the convergence of the proposed method, as prompt tuning is known for its slower convergence. It will good to show convergence of proposed method.”*\n\n  **Response:** Thank you for highlighting this aspect. Refer to the enclosed pdf for the training plots. Faster convergence is indeed a key benefit of LOPA. We present enclosed plots comparing the training loss and performance on NLU tasks (QQP, QNLI, MNLI) for Prompt Tuning (PT), IDPG, and LOPA. The results show that instance-dependent methods like IDPG and LOPA converge faster than traditional prompt tuning. Moreover, LOPA converges faster and achieves higher accuracy or F1 scores compared to IDPG. We appreciate the suggestion and will include this analysis in the paper.\n\n\n*References:*\n \n&nbsp;Zhang, Aston, et al. \"Beyond fully-connected layers with quaternions: Parameterization of hypercomplex multiplications with $1/n $ parameters.\" arXiv preprint arXiv:2102.08597 (2021).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "r8ZKSJrG5C", "reviewer": "Reviewer_4xA9", "summary": "This paper proposed a new prompt-tuning based approach called Low-Rank Prompt Adaptation (LOPA), which performs comparably to the state-of-the-art PEFT methods without the need for a server-based adapter. LOPA balances between sharing task-specific information across instances and customization for each instance to generate soft prompts, utilizing the low-rank decomposition for parameter efficiency. The effectiveness of proposed method is validated on multiple natural language understanding datasets.", "questions": "See the weaknesses above.", "limitations": "See the weaknesses above.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The author considers a novel perspective to achieve fine-tuning for downstream tasks without manipulating the foundation model.\n2. The method is simple, effective, and easy to implement.\n3. The writing is clear and easy to understand.", "weaknesses": "1. The comparison with the PEFT methods only considered LoRA, other representative methods such as Adapter-tuning, P-Tuningv2, etc., were not taken into account. Additionally, methods mentioned in the related work such as LPT and SPT that may compete with LoPA were not compared in the experimental tables.\n2. The ablation experiments are not comprehensive enough. The cost savings and performance sacrifices of using low-rank decomposition have not been discussed.\n3. Would it be more accurate to replace \"Foundation Models\" in the title with \"Language Models\"? Although the proposed method appears to be a general approach, it was only validated on natural language datasets.\n4. For the analysis of method principles, such as the offset subspace induced by LOPA, could some quantitative/visual verification be provided to demonstrate the changes brought about by the introduction of LOPA?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "- *“The comparison with the PEFT methods only considered LoRA, other representative methods such as Adapter-tuning, P-Tuningv2, etc., were not taken into account. Additionally, methods mentioned in the related work such as LPT and SPT that may compete with LoPA were not compared in the experimental tables.”*\n\n  **Response:** Thank you for this feedback. We chose LoRA as a representative baseline because it requires storage of user-specific parameters on the server for LLM personalization. Additionally, we focused on soft-prompting methods (e.g., PT, IDPG) that enable model customization on the user side, without necessitating server-side modifications. Soft-prompting methods like P-TuningV2, LPT, and SPT require inserting prefix vectors within intermediate layers of the transformer network, necessitating server-side changes for every user query.\n\n  To address your concern, we showcase a comparison with P-tuning v2 (Liu et. al 2021), Prefix tuning (Li et. al 2021) and a recent parameter-efficient baseline DePT (Shi et. al 2023). We can observe that while DEPT and P-tuning v2 reduce parameters, they also lead to a significant performance drop  (~21 and 16 pts average drop respectively) compared to LOPA. On the other hand, Prefix-tuning performs similarly to LOPA but does so at the cost of 16x more parameters.\n  \n  | Approach | Params$\\downarrow$ | RTE$\\uparrow$ | MRPC$\\uparrow$ | SST-2$\\uparrow$ | QNLI$\\uparrow$ | Average$\\uparrow$ |\n  | -------- | ------- | ------- | ------- | ------- | ------- | ------- |\n  | LOPA | 1.6M  | **83.39** | **91.09** | **95.99** | 93.74 | **91.05** |\n  | DEPT | 10.2K  | 53.79 | 72.97 | 89.68 | 57.09 | 68.38 |\n  | P-tuning v2 | 0.49M | 53.43 | 70.18 | 89.91 | 85.21 | 74.68 |\n  | Prefix-tuning |  25.75M  | 82.67 | 90.86 | 93.80 | **94.98** | 90.58 |\n\n  We will include this comparison and more task results in the paper.\n\n&nbsp;\n\n- *“The ablation experiments are not comprehensive enough. The cost savings and performance sacrifices of using low-rank decomposition have not been discussed.”*\n\n  **Response:** \n  1. **Ablation study - cost vs. performance trade-off:** In the paper, Figure 4 studies this cost-performance tradeoff as a function of rank. Bar plots show the training costs in terms of number of trainable parameters and Line plots show performance.\n  | Approach | Params$\\downarrow$ | RTE$\\uparrow$ | MRPC$\\uparrow$ | SST-2$\\uparrow$ |\n  | -------- | ------- | ------- | ------- | ------- |\n  | LOPA(r=4) | 1.60M | **83.39** | **91.09** | **95.99** |\n  | LOPA_add(r=4) | 1.60M | 64.26 | 75.17 | 93.34 |\n  | IDPG+PHM (n=8) | 0.37M | 67.14 | 76.12 | 95.07 |\n  | IDPG+PHM (n=16) | 0.20M | 65.34 | 76.68 | 94.61 |\n  | IDPG+PHM (n=32) | 0.17M | 68.23 | 74.99 | 94.72 |\n\n  2. **Ablation study - PHM Layers vs. Low-Rank Decomposition:** We compare low-rank decomposition in LOPA with Parameterized Hypercomplex Multiplication (PHM) Layers implemented for IDPG baseline that serves as an alternate for parameter efficiency.  $n$ representing the hyper-parameter balancing the parameter complexity and extent of factorisation in the Kronecker product $W = \\sum_i^n A_i \\bigotimes B_i$. Our study on three NLU tasks revealed that while PHM layers reduce parameter count, they also result in a significant performance drop (~15 points in RTE and MRPC), likely due to the structural constraints of Kronecker factorization limiting expressiveness (Zhang et al., 2021).\n\n  3. **Ablation study - Non-linear Composition of Z:** We also compared LOPA with LOPA_add, an additive approach for composing Z. The non-linear composition in LOPA, expressed as 𝑍 = 𝑍𝑆 ∘ 𝑔(𝑍𝐼), outperformed LOPA_add, indicating the importance of non-linear interaction for performance gains.\n\n&nbsp;\n\n- *\"Would it be more accurate to replace \"Foundation Models\" in the title with \"Language Models\"? Although the proposed method appears to be a general approach, it was only validated on natural language datasets.”*\n\n  **Response:** As the method is general, our preference is to use the term 'foundation model,' but the reviewer correctly points out the fact that the method was only evaluated with LLMs. We are happy to follow the reviewer's advice here. \n\n  The proposed approach has been validated on both natural language and code-generation datasets. Refer to Table 2 for evaluation of MBPP and CruXEVAL datasets.\n\n&nbsp;\n\n- *“For the analysis of method principles, such as the offset subspace induced by LOPA, could some quantitative/visual verification be provided to demonstrate the changes brought about by the introduction of LOPA?”*\n\n  **Response:** This is an excellent suggestion, especially the development of an appropriate visizalization to illustrate the LOPA functionality.  We plan to explore this idea and hope to add illustrative viz to an appendix.\n\n*References:*\n \n&nbsp;Zhang, Aston, et al. \"Beyond fully-connected layers with quaternions: Parameterization of hypercomplex multiplications with $1/n $ parameters.\" arXiv preprint arXiv:2102.08597 (2021).\n\n&nbsp;Shi, Zhengxiang, and Aldo Lipani. \"Dept: Decomposed prompt tuning for parameter-efficient fine-tuning.\" arXiv preprint arXiv:2309.05173 (2023).\n\n&nbsp;Liu, Xiao, et al. \"P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks.\" arXiv preprint arXiv:2110.07602 (2021).\n\n&nbsp;Li, Xiang Lisa, and Percy Liang. \"Prefix-tuning: Optimizing continuous prompts for generation.\" arXiv preprint arXiv:2101.00190 (2021).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Thank you for the insightful comments. We enclose the convergence plots of the prompt-tuning based baselines and the proposed approach on a subset of NLU tasks.

Meta-review: This paper proposes Low-Rank Prompt Adaptation (LOPA), which achieves competitive results compared with state-of-the-art PEFT approaches and full fine-tuning while being more parameter-efficient and not requiring server-based adaptors (drawbacks of PEFT methods like LoRA). Reviewers agree that the paper is well-written and easy to understand, and proposes a novel approach. The contribution of an effective prompt tuning approach is exciting! Most concerns (e.g., experiments on more tasks) in the original reviews have been addressed in the rebuttal. The authors should follow the discussions to improve the paper.
Decision: Accept (poster)