Improving generalization in large langue model by learning prefix subspaces
Authors: LOUIS FALISSARD
Vincent Guigue
Laure Soulier
Venue: EMNLP 2023 Findings (2023)
Keywords: Deep learning
parameter efficient fine-tuning
prefix-tuning
subspace learning
natural language processing
TL;DR: Learning simplexes of prefixes for large language models

Abstract: This article focuses on large language models (LLMs) fine-tuning in the scarce data regime (also known as "few-shot learning setting"). We propose a method to increase the generalization capabilities of LLMs based on neural network subspaces. 

This optimization method, recently introduced in computer vision, aims to improve model generalization by identifying wider local optima through the joint optimization of an entire simplex of models in parameter space.

Although this property would be highly beneficial in the context of training large language models in the “few-shot learning” setting, its adaptation to massive, pretrained transformers poses some challenges. First, their considerable number of parameters make it difficult to train several model jointly, and second, their deterministic parameter initialisation schemes make them unfit to the subspace method as originaly proposed.
We show in this paper that its application to "Parameter Efficient Fine-Tuning" (PEFT) methods, however, is relatively natural, and we propose to apply it to prefix-tuning, by learning entire simplexes of continous prefixes. 

We test our method on a variant of the GLUE benchmark adapted to the few-shot learning setting, and show that both our contributions (learning prefix simplexes, and non-deterministic validation metric inference) jointly lead to a gain in average performances compared to state of the art methods.

Improving generalization in large language models by learning prefix subspaces
Louis Falissard Vincent Guigue" Laure Soulier'
(1) Sorbonne Université, CNRS, ISIR, 75005 Paris, France
(2) Bibliothèque nationale de France, 75013 Paris, France
(3) AgroParisTech, UMR MIA-PS, 91120 Palaiseau, France
louis. falissard@gmail. com, vincent. vincent.guigun guigue@isir... upmc fr laure. souliereisir.. wtl.iiereiiiruuum........ fr
Abstract
Extending these discrete methods the learning of continuous prompts (Lester al., 2021), which conceptually falls within the framework so- called "Parameter Efficient Fine-Tuning (PEFT) methods (Houlsby et al., 2019; Bapna and Firat, 2019), poses certain challenges in the context of few-shot learning. One such challenge is the issue of model adjustment guidance through validation metric during gradient descent (Mao et al., 2022). Traditionally, in the process of model fitting, ap- proximately one-third of the training dataset ex- cluded beforehand to create a validation (or devel- opment) set dedicated to inferring an unbiased esti- mation of the model's performance (Hastie et al., 2001). This metric is utilized both during gradient descent (to estimate convergence of the descent al- gorithm inform early stopping heuristics), and subsequently guide hyperparameter searches typ- ically employed in the fine-tuning of large language models. However, the validity of this approach re- lies on the assumption that the distribution the validation set is representative of the real observed phenomenon. This assumption quickly loses its rel- evance in the context of few-shot learning, where available
This article focuses on large language models (LLMs) fine-tuning in the scarce data regime (also known as the "few-shot" learning setting). We propose a method to increase the general- ization capabilities of LLMs based on neural network subspaces. This optimization method, recently introduced computer vision, aims improve model generalization by identify- ing wider local optima through the joint op- timization of an entire simplex of models in parameter space. Its adaptation to massive, pretrained transformers, however, poses some challenges. First, their considerable number of parameters makes it difficult to train sev- models jointly, and second, their deter- ministic parameter initialization schemes make them unfit for the subspace method as origi- nally proposed. We show in this paper that "Pa- rameter Efficientt Tine-Tuning" (PEFT) meth- ods. however, are perfectly compatible with this original approach, and propose to learn en- tire simplex of continuous prefixes. We test GLUE bench-
our method on a variant mark adapted to the few-shot learning setting, and show that both our contributions jointly lead gain average performances com- pared to sota methods. The implementation can be found at the following link: https: //gi thub. com/ lllllolluuu subbppce
at most few tens of observations for estimating the validation metric. This notion has become problematic enough in present times that a portion of academic literature on continuous learning with small datasets presents experiment results utilizing validation sets that are unrealistic and artificial, containing several orders of magni- tude more observations than the training set used for the model adjustment itself (Mao et al., 2022).
1 Introduction
The emergence of large language models (Devlin et al., 2019; Radford et al., 2019; Raffel et al., 2019) in recent years has significantly transformed the ap- plications of deep learning methods in natural lan- guage processing. These models, pretrained in an unsupervised fashion on massive textual datasets, enable the fine-tuning of powerful models with just few thousand -or even hundred- observations. They achieve generalization performances that re- quired millions of observations just a few years ago, particularly when used in conjunction with discrete instruction prompts (Brown et al., 2020).
In the machine learning community, character- izing local minima with desirable generalization properties has been a topic of interest for decades (Garipov al.. 2018: Zhang et al., 2021; Hochre- iter and Schmidhuber, 1997). From flat minima (Hochreiter and Schmidhuber, 1997) to mode con- nectivity (Garipov et al., 2018), this body of work

---

has provided the basis for several practical obser- vations regarding the connection between the prop- erties of local minima and model's generalization abilities.
model) through single optimization loop as fol- lows:
A simplex of n models is built through ran- dom initialization of each of its vertices using
The concept of learning neural network sub- spaces (Wortsman et al., 2021) is example of method built using these considerations. This approach proposes find not just local minimum of the cost function in the model's parameter space, but an entire simplex associated with low values of this objective. This additional constraint is meant to bias the descent algorithm towards wider min- ima, empirically associated with better generaliza- tion (Dziugaite and Roy, 201 8a). In addition, the availability of this entire simplex of models allows for the inference of not only one scalar develop- ment metric, but an entire distribution, at any given moment during model tine-tuning These two phe- nomena, become particularly relevant when viewed through the lens of large language models, and most especially for few-shot learning problems, where the model's ability to generalize concept class from limited number of examples is crucial.
standard random initialization schemes.
For each gradient descent iteration, model is built as a weighted average of all vertices (to sample uniformly from the simplex they define)
The sampled model is used for inference and cost function computation
The gradient is backpropagated through all vertices to update all of their parameters
So far, the sampling procedure does not depend at all on the connectionist aspect of neural net- works and simply considers a model as a vector of learnable parameters. However, the vast ma- jority of deep learning models are defined as se- quences of non-linear transformations (Goodfellow et al., 2016). Therefore, it seems natural to in- corporate, in one way or another, this sequential structure of neural models into the sampling pro- cedure. To do so, Wortsman et al. 2021 propose to sample each layer's parameters independently. This variant, known as the by-yerrb--lyyrr method, is empirically associated with better generalization performances.
The contributions of this article are as follows, First, we introduce the first adaptation of the sub- space method to large language models through subspace adjustment of prefixes (a PEFT method similar to the Sute-off-the-as continuous prompt adjustment in current academic literature). Next, this article proposes to leverage certain natural ad- vantages offered by the subspace method to revisit the concept of guiding model adjustment through the validation metric. We will empirically demon- strate that the combination of these two ideas leads to significant improvement in terms of average prediction on natural language understanding tasks provided by the GLUE benchmark (Wang et al., 2018). Finally, an ablation study will be presented provide some insights into the mechanisms un- derlying this prediction improvement.
After model fitting, the simplex can be used ei- ther in the context of ensemble methods, or simply by using the simplex's 's centroid as the final model (Wortsman et al., 2021). The latter case is the one we focus on in this article, mainly because of the generalization properties it empirically displays.
Subspace centroid and generalization. Several explanations have been proposed to explain these interesting generalization properties. One possible justification for this property, visualized in 1, lies the idea that a model obtained through traditional training would be located at the periphery of a local minimum of the objective function, typically more susceptible to generalization emrors (Izmailov et al., 2018: Dziugaite and Roy, 2018b). On the contrary, moving within the subspace allows us "cross" the local minimum, in order to obtain a model associated with a more stable region of the objective function (Dziugaite and Roy, 3018a).
2 Background
In this section, we review the two main concepts used in this article, neural network subspaces (Wortsman et al., 2021) and prefix-tuning (Li and Liang, 2021).
2.1 Mode connectivity and network subspaces
Learning neural network subspaces. The sub- space method proposes to obtain the simplex of solutions (in the parameter space of the studied

---

adjust them while keeping the language model's pretrained parameters frozen. The main advan- tage of this approach lies in the ability of these "adapted" models to replicate (or even improve in contexts associated with small sample sizes) the performances of language models while reducing the number of learnable parameters by several or- ders of magnitude. In addition, some of these ap- proaches (L.i and Liang, 2021) typically require ran- dom initialization of the additional parameters they introduce into the model, making them particularly promising candidates for adapting the subspace method to large language models.
occu acy screes simplex (5T2. shot leaming)
1844
2342
2030
2304
2117
Figure 1: Generalization evolution of a language model adjusted on a prefix line, with alpha the weighting between both line extremities. Generalization perfor- mance follows a curve similar parabola with max- imum at its center.
Prompt-based approaches (Liu et al., 2022), on the other hand, are based on the adjustment n embedding vectors [E i = 1", typically concate- nated at the beginming of the input embedding se- quence of the language model I.Mo, parameterized by 4. In other words, for input sequence of L tokens, (T.)== we construct predictive model based not on the output of the language model it- self:
Application LLMs From the definition of the subspace method, it becomes clear why has never been applied (at least to our knowledge) to large language models. First, it requires storing not just one model during the optimization loop, but all the vertices of the studied simplex. This additional memory complexity constraint is likely manage- able in the case of adjusting small convolutional network in computer vision. However, language models are known for their substantial size, reach- ing up to hundreds of billions of parameters, to the extent that the traditional fine-tuning of a sin- gle model already poses considerable technical challenge for most specialized computing infras- tructures. Therefore, the idea of simultaneously adjusting not just one but up six models (the typical number of simplex vertices used in the sub- space method) appears be impractical.
LM ((Ti))))
(1)
but on
(coooca(((((E))) (4H)))
(2)
The adjustment of the predictive model is done solely by adjusting the virtual tokens (Ei) while keeping the parameters of the language model tb frozen.
To increase the expressiveness of this approach (which is particularly limited in terms of the num- ber of learnable parameters), prefix tuning (Li and Liang, 2021), chosen in this article candidate for applying the subspace method, proposes con- catenate these virtual tokens not to the input se- quence of the model, but to the Key and Value sequences used as inputs to the multiplicative atten- tion modules in each layer of the language model.
In addition, and more fundamentally, this ap- proach relies on random initialization of the de- scent algorithm to construct initial simplex of models. In contrast, pretrained language models are inherently initialized in a deterministic manner. Their entire transfer learning capabilities rely on the data representations captured into their parame- ter vectors during pretraining.
In a similar approach to continuous prompt fine- tuning, the adjustment of prefixes is done solely by adjusting (via gradient descent) the virtual to- kens, while keeping the parameters of the language model itself frozen. However, directly learning these embeddings proves be particularly unsta- ble (Li and Liang, 2021). Therefore, it is customary not to adjust them directly, but instead to use repa- rameterization trick, which involves concatenating transformed versions of the prefixes to the Key and
2.2 Prefix-tuning
On the other hand, continuous prompt adjust- ment methods and, by extension, PEFT methods (Houlsby et al., 2019; Hu al., 2021; Li and Liang. 2021; Liu et al., 2022), propose not to directly fine- tune language models, but instead introduce new learnable parameters (such as the embeddings of virtual tokens in continuous prompt learning) and

---

Value sequences. This transformation is parame- terized by two-layer feed-forward network, as follows:
Ps and Pk the prefixes prepended the Val-
ues and Keys sequences in the model, respec- tively
P. - (B) and Pa- (E)
(3)
E, the sequence of embedding vectors associ- ated with simplex vertex
With:
P. and Ps the prefixes prepended to the Val- ues and Keys sequences the model, respec- tively
MLP and M..Ps. the reparametrization perceptrons associated with simplex vertex for the Values and Keys prefixes, respectively
E a sequence of embedding vectors
It is also important consider the adaptation of the method's "layer-wise" variant. Indeed, the prefix adjustment does not rely on introducing a conventional perceptron structure into the language model, but rather on modifying the operation of the multi-head attention module. In this article, propose to extend this "layer-wise" variant to each layer's Keys and Values prefixes. Thus, dur- ing each descent iteration, the Keys and Values prefixes of each layer will be independently sam- pled. Moreover, this sampling will be performed independently for all observations, unlike the tradi- tional approach that prefers creating a single model per descent iteration step.
ML,, and MLP; the reparametrization per- ceptrons for the Values and Keys prefixes, re- spectively
3 Learning prefix subspaces
From this definition. it appears that the two main issues that made applying the subspace method to large language models cumbersome are both allevi- ated when using prefix tuning. Indeed, as PEFT method, the number of trainable in the prefixes is orders magnitude lower than the model's param- eters, allowing to easily store simplexes memory. Moreover, these prefixes are basically embedding vectors that require random initialization before model fine-tuning, which allows us to easily sam- ple an initial simplex, by randomly initializing all its vertices.
Additionally, the prediction head of the model is typically randomly initialized. Therefore, we choose apply the subspace method to the pre- diction head as well, described in Part 2. For consistency, the variant of parameter sampling at the observation level, rather than the batch level. will also be applied to it.
3.1 Model formalization
The adaptation of the subspace method to prefix-tuning can be done through two distinct ap- proaches:
In summary, we propose to adjust a simplex with prefix vertices as follows:
Independent initialization of n reparameteri- zation systems
1. Application to the learnable parameters of the model itself, namely the initial embedding and the reparameterization perceptron.
Computation of the n vertices of the simplex for each descent iteration
2. Application of the method to the prefixes themselves, specifically the output of the repa- rameterization module.
Construction of prefixes used for cost function inference and gradient calculation through in- dependent uniform sampling for each obser- vation, each layer, as well as for the prefixes of the Key and Value sequences.
In this article, propose to investigate the second option, essentially considering the reparameteriza- tion module training artifact, and build our proposed prefixes follows:
3.2 Subspace learning and stochastic inference of development metrics
P. ER MLP,,
(4)
The adjustment of large language model is (5) typically guided by estimating performance met- ric on a validation set, both during hyperparameter (6) search and the descent process itself, where the best model according to this sealar value selected as
Ps a (E,)
[[a]i E [0, 1]" Ejo; 1}
With:

---

the final model. The estimation of this metric in subspace learning framework raises questions. In- deed, adjusting not just a single model, but an entire simplex, results in potentially estimable validation metrics.
suitable to our problem using a methodology simi- lar to that presented in (Mao et al., 2022). Namely, we build few-shot learning classification datasets with varying sample sizes (50, 100, 200, and 500 observations) through random sampling. However, our method of constructing these corpora differs from the original authors' on several key points.
Since we limit ourselves in this article to us- ing this method to extract the centroid associated with better generalization performance, it would be natural to estimate the metric with respect to said centroid. However, the existence of not just a single model but this simplex, and the additional information it provides about the nature of the ob- tained local minimum, might be interesting. This is particularly the case in few-shot learning context. As mentioned earlier, for validation datasets with small sample sizes (typically <100), estimating this metric can become unreasonably noisy.
First, the authors chose to construct validation sets of 1,000 observations for all their training datasets, which, in our opinion, is not realistic in few-shot learning context (a validation set gen- erally does not contain ten times more examples than its training set). Secondly, they use the GLUE benchmark's s validation sets as test sets. However, some of these validation sets have small sample sizes (277 for RTE, 408 for MRPC), which could potentially introduce noise in the estimation of per- formance metries. Therefore, for each reference dataset, dataset of sample size K is constructed as follows:
Therefore, propose using the entire simplex to "augment" the development metric's estimation. This will be done by not using the simplex' cen- troid for inference but by using multiple randomly sampled models for each observation from the vali- dation set. In other words, for every development metric estimatiion we propose to concatenate the development set multiple time, and to perform in- ference under the same conditions as during gra- dient descent iterations, meaning with randomly sampled models used for each observation. We set the number n of developmlent set concatenation to 10 in all experiments presented in this article that employ this stochastic inference approach.
The training and validation datasets are con- catenated into a single dataset.
Half of the observations (capped at 5,000 ob- servations) are excluded from this dataset to construct test dataset that is common all experiments.
K observations are then selected through uni- form sampling and divided into a training and validation dataset following a 70/30 propor- tion.
Nevertheless, we still select the centroid of the simplex as the final model. Indeed, the determin- ism of a model remains desirable property in production settings.
For each task in the benchmark and for each selected sample size, 10 datasets are constructed using this methodology to allow for replication of experiments on different datasets, enable estima- tion of average performance, and test the signifi- cance at 5% threshold of the obtained differences (via bootstrap).
4 Experimental protocol
4.1 Datasets
All experiments described in this article eval- uate the predictive performance of the proposed method are conducted with BERT-base-cased on datasets constructed from the GLUE benchmark (Wang et al., 2018), which consists of 8 English language comprehension tasks, all formulated as classification problems. However, these datasets have significantly larger sample sizes than what would be expected the few-shot learning setting, and they do not make their test datasets available. As a consequence, we do not directly use these datasets, but instead adapt them to a format more
4.2 Baselines
All experiments described this article to evaluate the predictive performance of the proposed method are conducted with BERT-base-cased. We compare method to 5 baseline fine-tuning approaches, in- cluding standard fine-tuning and 4 alternate PEFT methods. Similar to prefix-tuning, these alterna- tive fine-tuning approaches are based on the idea of freezing the language model's parameters and introducing a fraction of new adjustable parameters

---

(lypically with a cardinality several orders of mag- nitude lower than that of the model itself), but they differ how they introduce these new parameters into the model:
Same method without prediction head sub- spaces
Same method without prefix subspace (i.e., only on prediction heads)
Standard Adapter (Houlsby et al., 2019), which typically involves introducing one or more two-layer bottleneck perceptrons at dif- ferent stages of Transformer layer. This was the first PEFT method to be introduced and is the most recognized.
Results
The performances of all selected PEFT methods, as well as the proposed approach, are presented in Table 1 for all different tasks of the GLUE benchmark and for the different selected sample sizes. Overall, the method sig- nificantly outperforms most baselines for all sam- ple sizes, and notably outperforms significantly all baselines for K 500. However, the method no- tably shows a higher gain for lower sample sizes (K' < 200), which strongly implies that experiment results still suffer from high variance in this regime
Overall effectiveness
Low-Rank Adaption (LoRA) (Hu et al., 2021), which reparametrizes the projection matrices of Values and Queries prior to the multi-head multiplicative attention module using two- layer linear bottleneck perceptrons. This was the first PEFT method to propose different transformations for different elements of the attention module.
The comparison between the proposed approach and prefix tuning particularly interesting. Indeed, both approaches have the same exact functional form. terms of statistical significance, the pro- posed method outperforms its classical counterpart 12 times:
UniPELT (Mao et al., 2022), a fusion method combining adapters, LoRA, and prefixes to benefit from the advantages of each method (without suffering from their potential respec- tive drawbacks).
On QNLI, SST-2, and STS-B for K=50 and K=100
Standard prefix tuning, a crucial reference method to estimate the portion the perfor- mance of the proposed method that can be attributed to it.
On MRPC and QQP for K=200
On MNLI, QNLI, MRPC, and STS-B for K=500
For the proposed approach and all aforemen- tioned baselines, follow the same procedure for model fitting and hyperparameter search as pro- posed by (Mao al., 2022), and experiment set- tings can be found in the annex. ensure optimal comparability, the hyperparameter choices for the proposed method will be selected to exactly match those of the prefix tuning baseline, which were also determined for the first time in a text classification framework by (Mao al., 2022). The subspaces adjusted in the experiments all simplexes with 6 vertices.
However, is statistically surpassed only once, on MRPC for K=50, which is even more surprising considering that the difference between the two methods in this experiment is 0.4%. Moreover, the proposed method becomes significantly superior again on this task once the sample size increases up 500 observations, showing significant increase in generalization performances.
Comparison between PEFT methods More broadly, the proposed method is significantly sur- passed only 6 times across all experiments:
4.3 Model variants
order to better identify the impact of the dif- ferent aspects of the proposed method, we also experiment with the following variants:
On MRPC by the prefix and LoRA methods for K=50
On RTE by the Adapter method for K=50
Same method with 2-vertex simplexes (i.e., line)
On STS-B by the UniPELT method for K=50
Same method without stochastic validation inference
On CoLA by the Adapter method for K=200
and K=500

---

Method (number of params.)
MNLI
QNLI
SST-2
QQP
CoLA
STS-B
MRPC
RTE
Avg.
[K 50/
Fine-ture tuning (108M)
35.5
65.9
57.57.
45.6,
3.5
45.1.
81.1
50.6
48.1,
Adapter (5M)
35.6
62.6,
64.7.
35.3,
0.0
59.7
80.2
53.1*
48.9,
LoRA (0.3M)
35.7
63.9.
68.4.
47.
1.0
56.5.
81.4
52.8
50.8.
UniPELT (L8M)
35.3
62.4.
73.1.
42.3.
1.1
64.3*
80.7*
51.8
51.4.
Prefix-tuning (0.9M)
37.8
63.5,
74.9.
53.1
1.8
59.2.
80.4*
52.6
52.9
Prefix subspaces (0.9M)
36.6
66.6
80.1
54.3
61.1
80.0
52.2
54.0
[K 700]
Fine-tuning (108M)
35.5.
68.9.
73.9.
52.6.
3.0.
64.1.
81.3
52.1
53.9.
Adapter (5M)
36.3.
66.7.
72.8.
54.0,
7.2
63.8.
80.5
53.0
54.3,
LoRA (0.3M)
37.3
64.9.
73.2.
54.2.
7.3
60.4.
81.3
52.9
53.9.
UniPELT (188))
37.7
66.9.
79.1.
53.6,
5.1
68.4
79.7,
52.0,
55.3,
Prefix Prefix-tuning (0.9M)
38.3
69.4.
80.8,
57.2
8.1
66.6,
81.1
54.2
57.0
Prefix subspaces (0.9M)
38.5
70.8
82.5
59.6
68.3
54.1
57.9
[K 200]
Fine-tuning (108M)
42.3
71.9
80.8.
63.0
20.2
69.0.
80.8
54.6
60.3.
Adapter (5M)
42.7
69.1,
83.1,
59.5,
26.5'
70.3,
56.2
61.0
LoRA (0.3M)
41.0
67.1.
82.2.
61.2.
19.8
67.8.
80.1
54.5
59.2.
UniPELT (L8M)
41.6
70.2
82.8.
58.7.
16.4
72.8
81.7
54.9
59.9.
Prefix-tuning (0.9M)
44.9
71.4
84.2
63.0,
22.2
71.3
79.6,
56.0
61.6
Prefix subspaces (0.9M)
44.7
71.2
84.1
64.4
21.1
72.3
81.6
55.9
61.9
IK 500G
tunint (108M)
52.7.
74.3.
85.4,
66.8
32.2,
78.0
82.5
59.8
66.5.
Adapter (SM)
51.1.
72.4.
85.4.
65.7.
38.9"
76.1,
81.9,
59.8
66.4.
LoRA (0.3M)
50.1
73.6,
84.6,
66.5
35.3
75.6,
82.3,
58.3,
65.8.
UniPELT (188)
50.7.
74.2.
85.4,
63.4,
34.2
77.2
57.8_
65.6.
Prefix-tuning (0.9M)
54.0-
74.7.
85.6.
66.2
35.7
77.8
82.
60
67.0.
Prefix subspaces (0.9M)
55.7
75.4
86.1
67.2
36.0
78.1
83.
60.8
67.8 8
Table 1: Experiment results. F1 scores are reported for QQP and MRPC. Spearman correlation is reported for STS-B. Matthews correlation reported for CoLA. Accuracy measures reported for the remaining tasks. Results in bold and underlined correspond to the first and second best performances, respectively. Results followed by asterisk as a subscript or superscript correspond to significantly higher or lower results compared to those of the proposed method, respectively.
is noteworthy that most of these occurrences are observed for K=50 (and therefore validation sets of 15 observations), where model fitting be- comes particularly challenging.
methods. Both groups feature an equal number of similar tasks and imbalanced datasets.
Comparison between approach variants The results for the investigated variants, which are dis- played in Table 2, can be summarized as follows:
On the other hand, the proposed method sig- nificantly outperforms one of the other baseline methods in the conducted experiments total of 80 times, demonstrating a clear advantage in terms of predictive power.
The use of two-vertex simplexes shows slightly inferior performance compared to the proposed method for K=50, and similar per- formance thereafter.
is particularly noticeable that the majority of experiments where the proposed method outper- forms the reference methods are mainly on three datasets: QNLI, SST-2, and QQP. Furthermore, the ability of the proposed method to significantly sur- pass the reference methods on these tasks does not seem to depend on the sample size of the datasets.
The use of the validation guided subspace method estimated deterministically collapses for K=50, K=100, and K=200 (cases where the performance is even lower than the clas- sical prefix fitting method) and eventually be- comes equivalent the proposed method.
However, difficult to identify what distin- guishes these datasets from those where the pro- posed method remains comparable to the reference
The use prefix subspaces without a head

---

Method
K = 50
K = 100
K 200
K 500
Proposed approach
54.0
57.9
61.9
67.8
Line subspace
53.6
57.9
62.1
67.6
Deterministic
49.5x
56.0
61.4
67.8
Head subspace
53.5
56.8
61.3
67.2
Prefix only subspaces
52.6
57.7
62.2
67.6
Table 2: Results of the ablation study. The reported scores correspond to the average predictive performances across all tasks in the GLUE benchmark.
subspace consistently surpassed by the pro- posed method.
characterize the obtained local minimum.
4.
The use of prediction head subspace coupled with classical prefixes considerably sur- passed for K=50 (which the only statisti- cally significant result) and similar to the pro- posed method when the sample size increases.
6
Conclusion
In this article, we introduced two innovative ideas. The first one, an adaptation of the subspace method for training large language models through the PEFT method, is, to our knowledge, the first exam- ple of its use in the academic literature on natural language processing. The second idea, proposing an alternative way to estimate development metries, represents an original application of the subspace method and not specific to problems encountered in textual data analysis. The combined use of these two methods leads to a significant improvement in the performance of common language models such as BERT on language comprehension tasks proposed by the GLUE benchmark, rephrased in "few-shot learning" context. The ablation study presented at the end of the article also allows us to quantify the impact of these two contributions. The performance gains observed on very small datasets (< 100) seem to be mainly explained by the finer information extracted from the validation set through the stochastic metric estimation method. However, this gain appears diminish for larger sample sizes, where the subspace method applied to prefix tuning seems to be sufficient its own to achieve performance gains over standard PEFT methods as well as classical model training.
These observations, taken as a whole, provide sev- eral pieces of evidence regarding the relevance of using the notion of stochastic validation metric in- ference in few-shot learning. In particular, Obser- vation 3 shows that adjusting the prefix subspace with classical validation metric estimation is asso- ciated with performance gains similar to the pro- posed method only from K = 500 onwards. More- over, the fact that the performance of this variant is even lower than that achieved by classical prefix adjustment further supports the importance of the proposed method of stochastic validation metric inference.
Subsequently, Observations 2, and provide slightly weaker arguments regarding the impor- tance of simplex size in the context this stochas- tic estimation. Although the simplex size does not appear to have an effect for K > 50 (strongly in- dicating that learning lines is preferable for these sample sizes, which are significantly more memory- efficient), it seems to have an impact for very small sample sizes. This observation could be explained by the richness of information extracted from the validation dataset through stochastic estimation, due to a larger simplex. However, the results pre- sented in this article are insufficient to confirm or refute this hypothesis. Similarly, Observations 2 and particularly highlight the importance of ad- justing the entire set of learnable parameters of the model through the subspace when K 50. This could also be explained by suggesting that restrict- ing the stochastic validation metric estimation to a subset of learnable parameters limits the ability to
Finally, applying subspace learning to PEFT methods also enables the training of powerful pre- dictive models while significantly reducing the computational resources typically required for training large language models. This approach preserves the fundamental efficiency goal of these methods. Learning prefix subspaces remains acces- sible even in situations where resources, both terms of data and computational power, are limited.

---

7 Limitations
learners. Advances neural information processing systems, 33:1877-1901.
The proposed approach is meant to improve the per- formances of large language models in the context of few-shot learning. As a consequence, it becomes increasingly dependent on the type of representa- tions the model's pretraining was able to capture. In other words, it would be highly unreasonable expect the proposed method perform well in highly complex tasks that cannot be easily captured by current unsupervised pretraining schemes.
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long and Short Papers), pages 4171-41888 Minneapolis, Minnesota. Association for Computational Linguistics.
In addition, the fact that this method allows for fine-tuning without high sample sizes in the de- velopment set might let people use it without any additional test set, and thus any model validation of any sort, which might lead to the implementation of highly biased models.
Gintare Karolina Dziugaite and Daniel Roy. 2018a. Est optimizes the prior of pac-bayes bound: Generalization properties of entropy-sge and data-dependent priors. In International Conference on Machine Learning, pages 1377-1386. PMLR.
Gintare Karolina Dziugaite and Daniel Roy. 2018b. Entropy-SGD optimizes the prior of a PAC-Bayes bound: Generalization properties of entropy- -SG and data-dependent priors. In Proceedings of the 35th International Conference on Machine Learn- ing. volume 80 of Proceedings of Machine Learning Research, pages 1377-1386. PMLR.
8 Ethical considerations
This article introduces a method for text classifica- tion that has the same exact functional form prefix fine-tuned large language model. As con- sequence, they get the same exact ethical issues, such as socially biased classification algorithms. In addition, the method's increased generalization abilities make it so that these algorithms might be built with fewer observations, which can lead to ill- defined objectives. These concerns call for thought and caution when implementing tools using the proposed model.
Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P Vetrov, and Andrew G Wilson. 2018. Loss surfaces, mode connectivity, and fast ensembling of dnns. Advances in neural information processing systems, 31.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. 2016. Deep Learning. MIT Press. http:// //www. deeplearningbook.ogg
9 Acknowledgements
Trevor Hastie, Robert Tibshirani, and Jerome Friedman. 2001. The Elements of Statistical Learning. Springer Series in Statistics. Springer New York Inc., New York. NY. USA.
We would like to thank the Sorbonne Center for Artificial Intelligence for funding Louis Falissard's post-doctoral contract within the MLIA laboratory of the Institute of Intelligent Systems and Robotics, We would also like to thank the ANR JCJC project SESAMS (Projet-ANR CIEE--ARIIICCEEE C--233300))),
Sepp Hochreiter and Jürgen Schmidhuher. 1997. Flat Minima. Neural Computation, 9(1):1-42.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo. Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for NLP. In Proceedings of the 36th International Conference Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 2790-2799. PMLR.
References
Ankur Bapna and Orhan Firat. 2019. Simple, scal- able adaptation for neural machine translation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- pages 1538-
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li. Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models.
guage Processing (EMNLP-IJ 1548, Hong Kong, China. Association for Computa- tional Linguistics.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot
Pavel Ixmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. 2018 Averaging weights leads to wider optima and better generalization.

---

Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. prompt
A Appendices
The power of scale for parameter-effie tuning. In Proceedings of the 2021 Conference Empirical Methods in Natural Language Processing, pages 3045-3059, Online and Punta Cana, Domini- can Republic. Association for Computational Lin- guistics.
All models were trained for 50 epochs using the default settings of the Huggingface Trainer, along with early stopping with patience of 10 epochs, The batch size is fixed at 16 for all experiments, and hyperparameter searches are performed through exhaustive search within the following values:
Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume I: Long Pupers), pages 4582- 4597, Online. Association for Computational Lin- guistics.
Traditional fine-tuning: Learning rate from [Te 5, 2c 5]
Standard adapter: Learning rate of le and reduction factor from [3, 6, 12]
Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Tam, Zhengx- iao Du, Zhilin Yang, and Jie Tang. 2022. P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 61 61-68, Dublin, Ireland. Association for Computational Lin- guistics.
LoRA: Rank and alpha value fixed 8, learn- ing rate from [le -, 4,
UniPELT: Prefix length fixed 10, adapter reduction factor fixed 16, and LoRA with rank and alpha value fixed at 8. Learning rate from [ee 4, 5e - 4]
Yuning Mao, Lambert Mathias, Rui Hou, Amjad Alma- hairi, Hao Ma, Jiawei Han, Scott Yih, and Madian Khabsa. 2022. UniPELT: A unified framework for parameter-efffcient language model tuning. In Pro- ceedings of the 60th Annual Meeting of the Associa- tion for Computational Linguistics (Volume 1: Long Papers), pages 6253-6264, Dublin, Ireland. Associa- tion for Computational Linguistics.
Prefix-tuning: Prefix length fixed at 50, learn- ing rate from [le 4, 2e - 4, 5e - 4]
Alec Radford, Jeff Wu, Rewon Child, D. Luan, Dario Amodei, and Nlya Sutskever. 2019. Language Models are Unsupervised Multitask Learners.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text trans- former.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: multi-task benchmark and analysis platform for nat- ural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353 355- Brussels, Belgium. Association for Com- putational Linguistics.
Mitchell Wortsman, Maxwell C Horton, Carlos Guestrin, Ali Farhadi, and Mohammad Rastegari. 2021. Learning neural network subspaces. In Pro- ceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 11217-11227. PMLR.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. 2021. Understanding deep learning (still) requires rethinking generaliza- tion. Commun, ACM, 107-1155


Reviews:
[{"review_id": "EuvmqW9cH7", "reviewer": "Reviewer_vny6", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank you for your review. Please, find below the responses to your different comments.\n\n[1 - Reject experimental setting from previous work]\n\nOur main issue with experimental settings displayed in most of the recent few-shot learning academic literature lies in the fact that they use validation datasets that are orders of magnitude larger than their corresponding training datasets. This is highly unrealistic, since the validation dataset is typically comprised of observations that are gathered from the same data generation process as the training dataset. As a consequence, we firmly believe that any insight gained from such an experimental setup would not be applicable to real-life scarce data regime problems.\nWith this article, we also mean to establish a new standard in methodology for gradient-based few-shot learning. One that we feel is actually realistic, and could actually transfer to real-life few-shot learning situations. We could reproduce experiments with settings displayed in previous works if that were to change your evaluation of our paper. However, this would be at the expense of a considerable amount of computational cost (the experiments displayed in our paper required the fine-tuning of approximately 6 thousand models), which is why we chose not to do so in the original paper.\n\n\n[2 - Need for experiments on larger LLMs]\n\nReplicating the experiment on larger pre-trained LLMs would be highly desirable since they typically behave better in the scarce data regime. We have not done so yet due to computational and time limitations. Again the experimental setting displayed in our paper required us to fine-tune approximately 6K LLMs, which is highly demanding in terms of resources. We could not do these experiments in the amount of time available for the rebuttal. However, you are perfectly right, and investigating how our proposed approach translates to different LLMs (bigger, generative, with different pretraining) will be the object of future work.\n\n\n[3 - Missing number of trainable parameters in Table 1]\n\nThank you for this remark, the number of trainable parameters will be included in Table 1. In the meantime, we will display it to you here:\n\nStandard fine-tuning: 108M\n\nAdapter: 5M\n\nLoRA: 300K\n\nUniPELT: 11M\n\nPrefix-tuning : 10M\n\nPrefix subspace: 10M (The final model derived from our method is functionally identical to one derived through standard prefix-tuning)\n\n\n[4 - Limited novelty]\n\nWhile it is true that PEFT and subspace learning are not new contributions of this article, we feel that another contribution of our paper, that is not mentioned in your summary of our article, lies in our proposal of using subspaces to derive distributions of validation metrics. To the best of our knowledge, this approach is truly novel and not limited to PEFT methods. It can indeed be applied to any neural architecture compatible with the subspace method (i.e. any perceptron, convolutional, or even transformer model), and its application to prefix-tuning is an experimental choice rather than a limitation. In addition, our experiments (especially our variants analysis) show their contribution to improving model performances, especially in the few-shot learning setting, which seems particularly significant to us.\nFinally, this paper is, to our knowledge, the first application of subspace learning to NLP and large language models. The contribution of subspace learning on LLMs, and of validation metrics, especially in the few-shot learning setting, seems particularly significant to us. To the best of our knowledge, we have found no trace of any existing work on the subject, and it is in addition key to avoiding the unrealistic validation sets currently used in the literature.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The proposed method obtains good results according to the experiment section in the paper.", "reasons_to_reject": "It may not be appropriate to directly reject setting from previous works and only provide results of newly constructed evaluation dataset. Comparison with previous methods under their settings are still suggested for fair comparison.\n\nExperiment on larger pre-trained language models are suggested. \n\nSince the proposed method is claimed to be efficient, comparison of number of parameters are suggested to be added into Table 1.\n\nThe novelty is limited, the proposed framework is based on previous methods with little modification.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper proposes PEFT which is a method for fine-tuning language models with subspace method can prefix tuning.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "PC8jpUr0Js", "reviewer": "Reviewer_1Q35", "summary": "", "questions": "On line 434, the authors note that they used classical fine-tuning as a baseline comparison without specifying whether Adam or AdamW optimizer was used, given that the latter can be expected to perform better in low-resource fine-tuning [1]. In turn, that could modify the results presented in Table 1, although without affecting overall results or diminishing the authors' work importance and impact, which is the reason this remains a suggestion.\n\n[1] Zhenxun Zhuang, Mingrui Liu, Ashok Cutkosky, and Francesco Orabona. Understanding AdamW through Proximal Methods and Scale-Freeness. arXiv:2202.00089.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "[1 - Reasons to reject]\n\nThank you for your appreciation of our work and your comments.! The code source was uploaded to a GitHub that will be cited in the revised version of the paper upon acceptance.\n\n[2 - Question for authors]\n\nWe apologize for this lack of precision. We indeed used AdamW, and not Adam, for all experiments in the article.\n\n[3 - Missing references]\n\nThe two references you mentioned will be included in the paper, and the mention of mode connectivity being a recent notion will be altered to account for these works.\n\n[4 - Typos Grammar Style and Presentation improvement]\n\nThank you for these ideas, we will include them in the revised version of our article.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is exceptionally well-written and easy to follow. Including replicates and indicating statistical significance is a welcome addition, allowing the authors to better support their claims and demonstrate the advantages of their method.\n\nThe problem of data and resource-efficient transfer learning on LLMs is highly pressing, given the general public interest in fine-tuning pretrained LLMs to suit their specific applications better, and issues with the classical fine-tuning, namely the additional memory required for the optimizer and data batches, on top of storing non-quantized LLMs and the amount of data required for fine-tuning.\n\nThe 5-fold replication and associated statistical significance indication is a welcome improvement on the status quo in the algorithms research in ML and provides excellent support to author's conclusions and results.\n\nTo the best of my knowledge, the approach used by the authors is novel and is supported by the current understanding of interconnected and flat minima.\n\nThe authors' approach is elegant, and the evidence provided to support the authors' claims is convincing. ", "reasons_to_reject": "At this stage the only element missing for a complete accept is the lack of attached source code to be able to validate authors' approach and allow future work to build on top of it. ", "questions_to_authors": "On line 434, the authors note that they used classical fine-tuning as a baseline comparison without specifying whether Adam or AdamW optimizer was used, given that the latter can be expected to perform better in low-resource fine-tuning [1]. In turn, that could modify the results presented in Table 1, although without affecting overall results or diminishing the authors' work importance and impact, which is the reason this remains a suggestion.\n\n[1] Zhenxun Zhuang, Mingrui Liu, Ashok Cutkosky, and Francesco Orabona. Understanding AdamW through Proximal Methods and Scale-Freeness. arXiv:2202.00089.", "paper_topic_and_main_contributions": "The authors of the paper propose a novel method for few-examples fine-tuning of LLMs (few-shot learning, also referred to as Parameter-Efficient Fine-Tuning, PEFT). Their approach is based on a coordinated training of several models (model simplex / neural network subspaces) to improve the model's generalization. To go around the large parameter number of LLMs and the fact that the NN subspaces approach requires random initialization, the authors apply the model simplex approach to an existing approach in PEFT - prefix tuning, which trains a smaller model from randomization. This smaller model modifies the behavior of multihead attention modules, allowing a differentiable tuning of the LLM in a data-efficient way. \n\nThe authors refer to the resulting approach as the \"Prefix Subspaces.\" They then demonstrate the performance of their approach compared to existing PEFT methods across a panel of tasks from the well-established GLUE benchmark, making sure to account for random fluctuation by 5-fold replication.\n\nWhile not perfect, their method compares favorably to most existing methods. The authors then perform a performance regression analysis, showing that random initialization at the heart of the subspaces approach is critical for the performance of their method.", "excitement": 5, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "The hypothesis of a connection between flatter minima and better generalization in ANNs is not \"recent,\" unlike what authors claim on L 074-079, and is generally traced at least to [2] and had substantial developments since, e.g. [3].\n\nI believe at least those two citations are needed to make the theoretical foundation of the approach more clear and allow readers to better connect the proposed approach with existing body of knowledge, although a more in-depth introduction, both on flat minima and minima interconnectness would be welcome. \n\n[2] Sepp Hochreiter and Jürgen Schmidhuber. Flatminima. Neural Computation, 9(1):1–42,1997.\n[3] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. 2021. Understanding deep learning (still) requires rethinking generalization. Commun. ACM 64, 3 (March 2021), 107–115.", "Typos_Grammar_Style_and_Presentation_Improvements": "- Regarding Table 1. While the information is comprehensive and is an appropriate amount of evidence for the authors' claims, I suggest that the authors add a set of heatmaps to convey the same information in a more rapidly understandable format.\n\n- I would also suggest that authors provide a cut across the NNs in the simplex, showing the distribution of performance metrics, given their claim that taking into account all the models in the simplex rather than the mean was essential for the performance.\n\n- L 553 - authors seem to refer to the ablation study as \"investigated variants.\" I would suggest using a consistent naming scheme to make the reading of the paper easier."}}, {"review_id": "OFTIgRpRWu", "reviewer": "Reviewer_bmEc", "summary": "", "questions": "Please refer to **Reasons To Reject**. If the author can help understand their contribution clearly, I would be willing to raise my score.", "limitations": "", "rating": 2, "confidence": 1, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "[1 - Limited improvement]\n\nThank you for your comments. We would like to point out that our proposed approach significantly outperforms the prefix-tuning baseline on 6 experiments with K>100, and moreover significantly outperforms prefix-tuning on average when K=500. We think that these experimental results strongly support the fact that our method offers some advantages over standard methods even for K>100.\n\nIn addition, the case where K<=100 remains of particular interest to us since it is actually fairly common when designing real-world machine learning-based systems. We think that our experiments show the increase in stability and efficiency that our method yields in this specific setup.\n\n\n[2 - Limited application scenario]\n\nAlthough we only applied it to prefix-tuning, the subspace learning method can be applied directly to all the other PEFT baselines selected in our paper without loss of generality. \n\nFirst, P-tuning can be seen as a “simplified” version of prefix-tuning, where only one prefix is created, and concatenated to the model’s input, instead of having prefixes for each multi-head attention module. P-tuning prefixes are built using a reparameterization trick identical to the one used in prefix-tuning, making it directly compatible with our proposed approach. \n\nIn addition, subspace learning was first derived for multilayer perceptrons and convolutional networks (Wortsman et al., 2021), and as such its application to Adapter or LoRA methods (both based on inserting small bottlenecked perceptrons in the pre-trained LLM) is straightforward.\n\nThe choice of limiting ourselves to one PEFT method was made for the sake of conciseness, and to limit the computations in our experiments. Prefix-tuning was chosen as our candidate solely based on the fact that it was found to yield the best performances among all others in the scarce data regime.\n\n1 - Mitchell Wortsman, Maxwell C Horton, Carlos Guestrin, Ali Farhadi, and Mohammad  Rastegari, 2021, Learning neural network subspaces\n\n\n[3 - Lack of experiments]\n\nWe are sorry that you feel the experiments we displayed aren’t enough to demonstrate our method’s interest. We would like to emphasize the fact that PEFT methods are widely used techniques that have shown their interest in a broad range of applications, especially on very large models such as Llama, whose size renders traditional fine-tuning unreasonable.\n\nOur contribution, evaluated on prefix-tuning but potentially applicable to most (if not all) PEFT methods, constitutes a solution of choice for fine-tuning and stabilizing LLMs in the few-shot learning setting. We are convinced that this method has numerous potential applications and are currently working on some of them. We hope that the results we obtained so far have already demonstrated the value of this approach.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The motivation is clear and reasonable. \n\nThe writing is clear.", "reasons_to_reject": "Actually I am not familiar with this topic, but in my opinion, I think there exists some drawbacks:\n\n (1) The improvements is limited. From the Table 1, I notice that only when K is small (<100), the gain over the \"Prefix-tuning\" baseline is significant.\n\n(2) Application scenarios are limited. As there are many types of tuning techniques (e.g. prefix/p-tuning/LoRA/..), the proposed method can only be applied on prefix methods.\n\n(3) Due to the limited application, the experiments also seem to be  not enough.", "questions_to_authors": "Please refer to **Reasons To Reject**. If the author can help understand their contribution clearly, I would be willing to raise my score.", "paper_topic_and_main_contributions": "This paper introduce the first adaptation of the subspace method to large language models through subspace adjustment of prefixes. \n\nThis paper propose to leverage certain natural advantages offered by the subspace method to revisit the concept of guiding model adjustment through the validation metric. And the author show that two ideas leads to improvement on natural language understanding tasks.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "none.", "Typos_Grammar_Style_and_Presentation_Improvements": "none."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper combines the "Learning neural network subspaces" approach and the "Prefix tuning" to improve generalization of language models in few-shot learning setting. Experiments focus on K-shot (K=50,100,200,500) adaptation of Bert-base on GLUE benchmark. Reviewers find the existing evaluation sound, however recommend adding (a) experiments with full GLUE dataset (b) applying it with another PEFT method.  I recommend authors to consider adding these experiments as these experiments would help reader to understand limitations and potential of the main hypothesis of the paper better. These experiments should require << 6k experiments and worth the time.
Decision: Accept-Findings