Exploring the Impact of Model Scaling on Parameter-Efficient Tuning
Authors: Yusheng Su
Chi-Min Chan
Jiali Cheng
Yujia Qin
Yankai Lin
Shengding Hu
Zonghan Yang
Ning Ding
Xingzhi Sun
Guotong Xie
Zhiyuan Liu
Maosong Sun
Venue: EMNLP 2023 Main (2023)
Keywords: Parameter-efficient fine-tuning
Pre-trained language model

Abstract: Parameter-efficient tuning (PET) methods can effectively drive extremely large pre-trained language models (PLMs) by training only minimal parameters. Different PET methods utilize different manually designed tunable modules. In small PLMs, there are usually noticeable performance differences among PET methods. Nevertheless, as the model scale increases, the performance differences become marginal. Hence, we hypothesize that model scaling mitigates the impact of design differences on PET methods. To investigate this hypothesis, we introduce a more flexible PET method called Arbitrary PET (APET) method. The APET method is compatible with a tunable module, which consists of any number of parameters distributed in arbitrary positions. Then, we utilize it and conduct experiments on $11$ NLP tasks across $3$ representative PLMs. Our investigations reveal that model scaling (1) mitigates the effects of the positions of tunable parameters on performance, and (2) enables tuning methods to achieve performance comparable to full-parameter fine-tuning by optimizing fewer tunable parameters. Intriguingly, we also observe that tuning methods optimize the similar number of tunable parameters to exceed random guess performance on different tasks. We collectively discuss this phenomenon and the two aforementioned findings from an optimization perspective to understand the underlying mechanisms. These conclusions enhance our understanding of the impact of model scaling on PET and assist in designing more effective and efficient PET methods for PLMs of different scales. The source code can be obtained from this GitHub repository: \url{https://github.com/yushengsu-thu/PET_Scaling}.

Exploring the Impact of Model Scaling on Parameter-effcient Tuning
Yusheng Su' Chi-Min Chan' Jiali Cheng', Yujia Qin', Yankai Lin', Shengding Hu',
Zonghan Yang', Ning Ding', Xingzhi Sun Guotong Xie*. Zhiyuan Liu Maosong Sun DDepartment of Computer Science and Technology. Tsinghua University ?University of Massachusetts Lowell
Gaoling School of Artificial Intelligence, Renmin University *Ping An Technology
yushengsu. thu@gmail.ccmm com
Abstract
Performance
Parameter- Efficiinn Tuning
BrtFit
Purameter-efficient tuning (PET) methods can effectively drive extremely large pre-trained language models (PLMs) by training only min- imal parameters. Different PET methods uti- lize different manually designed tunable mod- ules. In small PLMs, there are usually notice- able performance differences among PET meth- ods. Nevertheless, as the model scale increases, the performance differences become marginal. Hence, we hypothesize that model scaling miti- gates the impact of design differences on PET methods. To investigate this hypothesis, we introduce more flexible PET method called Arbitrary PET (APET) method. The APET method compatible with a tunable module, which consists of any number of parameters dis- tributed arbitrary positions. Then, we utilize it and conduct experiments across 3 representative PLMs. Our investiga- tions reveal that model scaling (1) mitigates the effects of the positions of tunable param- eters on performance, and (2) enables tuning methods to achieve performance comparable to full-parameter fine-tuning by optimizing fewer tunable parameters. Intriguingly, we also ob- serve that tuning methods optimize the similar number of tunable parameters to exceed ran- dom guess performance on different tasks. We collectively discuss this phenomenon and the two aforementioned findings from an optimiza- tion perspective to understand the underlying mechanisms. These conclusions enhance understanding of the impact of model scaling on and assist in designing more effec- tive and efficient PET methods for PLMs of different scales. The source code can be ob- tained from this GitHub repository: https:// gi titubb.co com/ wwwooyyusengguuu thulPET_Scalinng.
(PET) Methods
Transtomerr
BitFit Aco Layer
Small
Adapler
Adapter
Food Fonward Network
BitFit
LoRA
Medium
Adapter
Mulli-Hood Hoad Attintion
Prompt
PC
LoRA
LoRA
Large
[Hidden States
Prompt
Model Parameters (Freezing) <b
Trainable Modules
(Optimizing) 0
Scale
Figure Different PET methods have distinct tunable modules, which typically result in noticeable perfor- mance differences. However, as the model scale in- creases, these differences become less significant.
11 NLP tasks
and T5 (Raffel et al., 2020), have achieved great success on various natural language processing (NLP) tasks. Despite their effectiveness, fine- tuning (FT) these large-scale PLMS with full param- eters incurs both unaffordable computational and storage costs. To solve this problem, researchers have proposed series of parameter-efficient tun- ing (PET) methods (Houlsby et al., 2019a; Li and Liang, 2021; Mahabadi et al., 2021a: Lester et al., 2021; Mahabadi et al., 2021b; Hu et al., 2022a; Ben Zaken et al., 2022; He et al., 2022b) which only update assigned tunable module consist- ing of minimal parameters while freezing the rest parameters in a PLM during model adaptation.
Although these existing representative PET methods can reduce computational and storage costs, there are usually noticeable performance dif- ferences among these representative PET methods on downstream tasks. Intriguingly, as the scale of a PLM increases, the performance differences among PET methods become narrower, as illus-
1 Introduction
Pre-trained language models (PLMs), such as GPT (Radford et al., 2018), BERT (Devlin et al., 2019),
The first two authors contributed equally. Corresponding author: Z. and M.Sun.

---

trated in Figure These findings are interesting and worth exploring because the existing represen- tative PET methods are designed with disparate philosophies, e.g. tunable modules that are com- posed of different numbers tunable parameters distributed in arbitrary positions. Hence, we hy- pothesize that model scaling mitigates the effects of the above design differences among the PET meth- ods on performance. To validate this hypothesis, we further conduct two lines of ablation analyses:
tends to be lower on the larger models. Namely, PET methods can optimize fewer tunable param- elers to achieve full full-purameter fine-tuning perfor- mance on the larger models. On the other hand, when the number of tunable parameters exceeds the low parameter threshold, all APET methods outper- form random guess performance. We find that the low thresholds are nearly identical across the same models, even for different tasks. This suggests that across different tasks, PET methods can optimize a similar number of tunable parameters on the same PLM to surpass random guess performance.
(A1) Whether the model scale mitigates the perfor- mance differences resulting from the position of tunable parameters.
In summary, introduce more flexible PET methods APET methods to conduct the extensive ablation analyses and reveal the impact of model scaling on PET design, e.g., (1) the position of tunable parameters ($ 5.2) and (2) the number of tunable parameters ($ 5.3). (3) Furthermore, we discuss the findings of ablation analyses from the perspective of optimization ($ 6). We hope these conclusions not only encourage more researchers to explore the impact of model scaling tuning methods from theoretical perspective, but also provide guidance for designing tuning methods for models of different scales.
(A2) Whether the model scale mitigates the perfor- mance differences resulting from the number of tunable parameters.
However, solely investigating the four representa- tive PET methods (see Figure 1) might be insuffi- cient to encompass an adequate range of parameter positions for the ablation analyses (A1). Addition- ally, the tunable modules of these four PET meth- ods are constrained to be composed of layer-level tensors or matrices, making challenging to pre- cisely control the number of tunable parameters at the -re-grained (parameter) level in the ablation analyses (A2). To facilitate the ablation analyses, we develop more flexible Arbitrary Parameter- Efficient Tuning (APET) method ($ 5.1), which can be compatible with any number of tunable pa- rameters distributed in arbitrary positions.
2 Related Work
Parameter-Efficht Tuning (PET) Methods With larger PLMs continuously being developed, fine-tuning all of the parameters and storing the adapted weights become increasingly cumbersome. To address the issue, researchers propose PET methods which keep most of the parameters of PLMs frozen and optimize only a tunable module consisting of a few parameters during downstream adaptation. Over the recent years, many differ- ent designs of PET methods have emerged. For instance, some PET methods insert the external tunable modules after the feed-forward and atten- tion layers PLM (Houlsby et al., 2019a; Pfeif- fer et al., 2021: Mahabadi et al., 2021c); others prepend the tunable modules into attention layers (Li and Liang, 2021; Hu et al., 2022a) or the embed- ding layer (Lester et al., 2021). Another line of PET method selects the existing parameters in PLM (Ben Zaken et al., 2022; Guo et al., 2021) the tunable module optimize. To further enhance the performance of PET methods, some works propose automatic selection strategies (Hu et al., 2022c; Chen et al., 2023; Lawton et al., 2023; Zhou et al., 2023) for tunable parameters.
In analysis (A1), we compare the performance of APET methods with an equal number of tunable parameters distributed in different positions. Based on the experimental results, we observe smaller dif- ferences in the performance of these APET meth- ods on larger models. This finding suggests that scaling the model miligates the effects caused by the position of tunable parameters on performance. In analysis (A2), we compare the performance of the same APET methods with varying numbers of tunable parameters. Based on the experimen- tal results, we observe that model scaling does not mitigate the effects caused by the number of tun- able parameters on performance. Furthermore, we have observed two interesting phenomena when the number of tunable parameters reaches two thresh- olds: the high threshold and the low threshold. When the number of tunable parameters equals the high threshold, APET methods can achieve the full hull-eeraeeter fine-tuning performance of the cor- responding backbone model, and the high threshold

---

PET Methods
Unified View of PET Methods Positions Tunable Modules (W,, w
Prompt (Lester et al., 2021)
W will be concatenated input hidden states W will be plugged between SclfAttn//FNN. layers W will be plugged into SelfAttn layers will add into Bias terms
Adapter (Houlsby et al., 2019a)
hout =(hh)) Ah
LoRA (Hu et al., 2022a)
BitFit (Ben Zaken et al., 2022)
We uniformly re- frame the transformations of PET methods as modifications Ah of specific hidden states in the corresponding PLM layer (f) where W is introduced in computing Åh, as suggested by (2022a); Hu et al. (2022c). Each PET method has tunable weights W in designed positions. Hence, represent each PET tunable module as (W1, 2..... Wp).
Table I:
Although these PET methods have distinct tun- able modules, they can be unified into a similar form. He et al. (2022a) formalize PET methods unified framework to study the connections among PET methods. Yi et al. (2022) also conduct the same study and further indicate that the opti- mization of different PET methods can be unified in similar subspace. In this paper, we leverage these unified perspectives to explain the impact of model scaling on PET the final discussion ($ 6).
state is applied with the sequence of layers. For simplicity, we formalize the transformation of each layer as
h ondi f(h").
(1)
Under the layer as the operator f. the input hidden state h c RRsxd is transformed into the output hidden state h hou E RRxxdout where is the input length and dins dout are dimensions.
3.2 Parameter Efficient Tuning (PET)
The Power of Model Scaling With the scaling of
Different PET methods are equipped with diverse modules 0 as shown in Figure 1. These modules are composed of tunable parameters W that modify the original layers and the corresponding transfor- mations in PLMs. To make comparisons, we follow unified view (He et al., 2022a; Hu et al., 2022c) to re- frame the transformations of all PET methods as the modifications Ah of specific hidden states in the corresponding PLM's layers as follows:
model size, PLMs emerge numerous capabilities, including reasoning ability (Wei al., 2022b,a), and can achieve state-of-the-s results in various understanding and generation tasks (Du et al., 2022: Chowdhery et al., 2022).
In the adaption perspective, some researchers find that performing some PET methods (Lester et al., 2021; Ding et al., 2023; Su et al., 2022) lcrge models almost achieve the full- parameter fine-tuning performance. In this paper, we further find that as the model scale increases, the performance differences among distinet PET methods become smaller ($ 4). Hence, we study the impact of model scaling on PET methods ($ 5) to fathom this phenomenon and explain from the optimization perspective ($ 6).
h out (h Ah.
(2)
In the training process, given a downstream task D = (X,Y), we only optimize all tunable pa- rameters of the module 0 for each PET method to generate desired outputs Y of downstream task while freezing the rest of the parameters & in a PLM M, shown in Figure 12. Formally, the training objective to minimize as follows:
3 Preliminary
this section, we first introduce the Transformer framework ($ 3.1) and the most representative PET ($ 3.2).
ming (M (X),))
(3)
4 Main Experiments
To explore the impact of model scaling on these PET methods, we first introduce the investigated tasks, PLMs, and settings of the existing represen- tative PET methods in the experiments ($ 4.1,, and then report the main experimental results ($ 4.2).
3.1 Transformer Framework
The Transformer model (Vaswani et al., 2017) is the mainstream architecture for most powerful PLMs. The model is stacked of L blocks. each of which consists of a sequence of layers, including self- attention and forward network. During the forward pass through each block, the input hidden
MMor implementation details are in appendix "The manipulations, including addition, concatenation, and plugging, are discussed in $ 5.1.

---

4.1 Experimental Settings
[b] BERT (6 tasks of types)
Investigated NLP Tasks We investigate 11 tasks, which can be divided into 5 categories: (1) Sen- timent Analysis (SA), including SST-2 (Socher al., 2013), IMDB (Maas al., 2011), and Rotten Tomatoes (Pang and Lee, 2005); (2) Natural Lan- guage Inference (NLI), including MNLI (Williams et al., 2018), QNLI (Wang ct al., 2019), and RTE (Bos and Markert, 2005); (3) Paraphrase Identifi- cation (PI), including MRPC (Dolan and Brockett, 2005) and QQP (Sharma et al., 2019); (4) Question Answering (QA), including NQ-Open (Lee et al., 2019); (5) Summarizatioo (SUM), including SAM- Sum (Gliwa al., 2019) and Multi-News (Fabbri al., 2019). More details are in appendix A.
Average Performance
85
13
+
T>
BERT
BER BER Ta
S.D. 5,08
S.D. 3.63
S.D. 2.65
Avarage Performance (5)
[b] BLOOM (11 tasks of types)
Adapter
(FT)
70-
50
BLOOMssori
BLODM S.D. 3,00
BLOOM S.D. 2.50
S.D. 3,46
[a] T5 (11 tasks of types)
as
Pronp:
Adacter
Average rerformance
1 +
Investigated PLMs We will experiment three series of PLM backbones: BERT (Devlin al., 2019), BLOOM (Seao al., 2023), and T5 (Raf- fel et al., 2020) representing encoder-base model, decoder-based model, and requence-to-sequence based model, respectively. Since BERT has fixed- length output limitation, we only investigate SA, Pl, and NLI categories of tasks on it. Differently. BLOOM and T5 models have fixed-length out- put limitation; thus, we investigate all tasks on them.
TSawu: S.D.
T5
T5 1.72
Figure 2: We investigate the average performance of the tuming methods, including Prompt, BitFit, LoRA, Adapter, and full full-parameter fine-tuning, on three series of models. As the model scaling increases, the perfor- mance differences (standard deviation (S.D.)) among tuning methods become smaller.
Training Details of PET Methods We select four representative PET methods: Prompt (Lester et al., 2021), BitFit (Ben Zaken al., 2022), Adapter (Houlsby et al., 2019a), and LoRA (Hu et al., 2022a), for conducting analysis experiments. To ensure the consistency of the PET methods' per- formance, we maintain the original design of each method, including the positions of tunable param- eters and the number of trainable parameters, reported in the respective original papers. Addition- ally, we train each PET method on 11 tasks using 3 different random seeds and report their average performance. Further details regarding the training configurations can be found in appendix B.
(S.D.)) from each other on the general-seale models (BERT sMALL and BERT bASE in the sub-figure [a]: BLOOM560M and BLOOM in the sub-figure [b]; T5 and T5 in the sub-figure [c]). This phenomenon is intuitive and demonstrates the critical impact of design differences (the position and quantity of parameters in the tunable module) the performance of PET methods. This find- ing has been consistently found in numerous prior works (Ding et al., 2023; Hu al., 2022c).
However, we find that as the model scaling in- creases (from BERT SMALL to BERT LARGE in the sub- figure [a]; from BLOOM560M to BLOOM in the sub-figure [b]; from T5 to T5xxL. the sub-figure [c]), the performance discrepancies among PET methods diminish across all types of models, evidenced by the decreasing standard deviation (S.D.) (from 5.08 to 2.65 on [a] BERT; from 3.46 2.50 on [b] BLOOM: from 2.75 1.72 on [c] T5). This finding implies that the larger model scaling can mitigate the impact of the design differences among the PET methods on performance.
4.2 Model Scaling Impact on PET Methods
To investigate the impact of model scaling on PET methods, we arrange the Pre-trained Language Models (PLMs) in ascending order based on their model scale, and report the performance of PET methods each type of PLM.
Results are reported in Figure/ 2. First, we can observe that the PET methods exhibit notice- able performance differences (standard deviation

---

5 Ablation Analyses
The design differences among the PET methods mainly lic in the tunable module's s parameter po- sition and parameter quantity. To further verify whether the model scaling will respectively remove the effects of the above differences on PET meth- ods, we conducted two ablations to investigate whether model scaling can mitigate (1) the impact of tunable parameter position and (2) the impact of tunable parameter quantity.
Arbitrary Parameter
Module
Efficient Tuning (APET)
A0d
APET
Torwort Nerwor
Turable
Weige
W
CONCAT
Add Layer Norm
1 PLM Layor Waur
Mussiited
Antention
However, only investigating the above four re- spective PET methods is insufficient to cover enough variations of parameter position for abla- tion study (1). This limitation makes us hard to preciously control the number of tunable parame- ters at the fine- fine-grained (parameter level) in abla- tion study (2). Hence, develop more flexible PET method, Arbitrary Parameter-Efficieet Tuning (APET) method. Its tunable module can be arbi- trary structure ($ 5.1) that facilitates us to explore various parameter positions in the ablation study ($ 5.2) and easier control the number of tunable parameters in the ablation study ($ 5.3).
PLUG
W:
Posisons
Tunable Parameters
Hiltien Statos
XL
Frazen Parameters
Figure The tunable modules of APET methods (0 [W,, W..... WL are composed L tunable weights W with arbitrary structures. There are three operations (ADD, CONCAT, PLUG) for inserting these tunable weights W into a PLM.
Note that the inserted tunable weights W are not limited the aforementioned structure as shown in Figure 3; they can be arbitrary structures. Ac- cording to the inserted tunable weights and the corresponding modifications, the transformations of a PLM layer for APET method can be expressed as:
5.1 Arbitrarily Garameter-Efficiae Tuning (APET)
Similar to PET methods, the APET method is equipped with arbitrary module 0 which is com- posed of L tunable weights W distributed in any position of model. Here, APET have three op- erations to insert the tunable weight W into any position of the PLM, thereby modify the specific layers and their corresponding transformations as follows:
W f(W2) oh W."' off(h" Ws)
h out ff +
(7)
ADD The tunable weight W will be into the PLM layer. The corresponding transformation of
By comparing Equation 7 with the equations of pre- viously introduced Equation 2. it is obvious that the PET methods are special cases of APET method. The module 0 of APET are composed of arbitrar- ily inserted weights W. which can be expressed (W1, W2, ..., Wz). ). the training process, we follow Equation (3) only to optimize while freezing the rest of the parameters (T) in a PLM.
PLM layer be denoted
out f(h")) W,
(4)
The tunable weight W will be con- catenated with the hidden state or the layer in the PLM. The corresponding transformation of a PLM layer can be denoted as:
CONCAT
5.2 The Impact of Differences in Parameter Position on Performance
f(W2) out f(h")) + W W.
(5)
To investigate whether model scaling can mitigate the impact of parameter position in PET, ini- tially freeze other significant factors, i.e. the num- ber of tunable parameters, that could potentially affect the performance. Given that the tunable pa- rameters of the four aforementioned PET methods
PLUG
The tunable weight W will be plugged
between PLM layers. The corresponding transfor- mation PL.M layer can be denoted as:
put = f(h + (f(hh") Ws) Ws. (6)

---

[a.1] BERTown
[a.2] BERTIANSE
Bosing Refomaree (6)
Rancum Swad
Randoom Soadl
Randram Sered
Randiom Saud
Random Sindd
Randim Seed
Aeergg Perfommnce
gt
$36 St
8AS d
71
777
80
+
APETworuti
APET SDD 0.25
APET S.D. = 0.32
APET APETwoeeri 0.76
APET Proeeeei
APET S.D. 1.03
APETo
APET S.D. 0.49
S.D. 1.81
S.D. 0.64
S.D. 0,29
[b.1] BLOOMsccM
[b.2] BLOOM
Aucrogo Peformrcc (30
Rerformnnc (%)
Rancom Scod
Random Soad
Random Scad
Random Socd 1
Random Sood
Rancom Sood
60
067
at- $1.5
75-
68.1 19
23
80
APET S.D 2.65
APET S.D. m 0.28
Reaptri) S.D. 2.82
APET Troomaii
APET S.D. - 1.08
LPET S.D. 0.39
APET S.D. 1.63
S.D. = 3.45
S.D. = 2.81
[c.1] T5SmAL
[c.2] T5
Rancium
Rardlom
Randion
Random Saad
Aussage herforeen Perfrttonn
16.9
11.
761
13.1
66.9
momati
APETaundi S.D. 1.57
APET 10.90
Trrmmed S.D. 1.85
APETa S.D. 0.36
APETAsveteri S.D. 0.30
S.D. = 4.10
S.D. 2.21
S.D. 0.18
Figure 4: The parameter quantity each group (bar: corresponds aforementioned four PET methods'. We denote the APET methods with the coresponding numbers of parameters as APET Prompt. APET Bara, APET and APET AdETed respectively. Each APET method will arbitrarily select tunable parameters with different random seeds, each random seed representing different parameter distribution. Here, S.D. means the standard deviation. As the model scaling increases, the impact caused by the parameter position on the performance becomes minor.
are fixed in the same positions, it challenging for us to precisely conduct an experiment as- sess the impact of position. Under this limitation, we then employ the APET method to arbitrarily select tunable parameters with different random seeds, each random seed representing a different parameter distribution, and train them on the tasks.
from [b. 1] [b. 2]: T5: from [c. 1] to [c. 2]), the performance differences (standard deviation (S.D)) APET methods within each group de- crease. Based on this findings, we argue that larger models demonstrate greater effectiveness in miti- gating the impact of differences in parameter posi- tion on performance.
In the experiments, we set the number of tun- able parameters for the APET methods in four groups. The parameter quantity in each group (bar: corresponds to that of the aforemen- tioned four PET methods' (Prompt, BitFit, LoRA, Adapter). We denote these APET methods with varying numbers of parameters? APET Prompt. APETBatitt APETLORA. and APETer respec- tively. Besides, we conduct the ablation study on three series of models (BERT, BLOOM, and T5) and report task (SST, RTE, and MRPC) average performance.
In addition, we have observed that despite the dif- ferent number of tunable parameters in four differ- ent groups (bar: of APET methods, they have fewer performance differences on the larger model. We will delve into this finding further and provide explanation for this phenomenon in $ 5.3.
5.3
The Impact of Differences in The Number of Tunable Parameters on Performance
In this section, given the APET method under dif- ferent numbers of tunable parameters, we observe their performance to conduct an ablation study.
Performance Comparison As shown in Figure 4, there are four groups of comparisons in each sub-graph. We can observe that as a PLM size scales (BERT: from [a. 1] to [a. BLOOM:
From the reported results in Figure 5, we can find that (1) on the smaller models, e.g., BERT sMaLL. (- -), BLOOM560M (---), T5 (---) when the tunable parameters of tuning methods are fewer than a certain number, the performance will drop to randomly guess performance; (2) similarly,
"The number of tunable parameters are left in appendix D.

---

Pin Tune (20t)
APST (ecRTana)
Pho Ture (ELOCM
AFET (5400M
Teo Tand (TBrea) Fine Tame (17m))
Afct (Paama)
Pros Turn (2E3)om))
APET (eciT
Pree Taaw (5
ARET (5400M
ARET (Plpcl
5512
2004
A yerr
(64)) Kiz
Low
(63) MAPC
(1.5)
[c3]
2301
Menbor
Figure 5: Given the different numbers of tunable parameters, we observe APET performance on three series of models and tasks. We find that (1) the model scaling can make tuning methods optimize fewer necessarily tuned parameters to reach full-pane parameeeetter fine-tuning performance (--- and (2) APET methods require the similiar number of tunable parameters (low parameter thresholds lie the similiar range) to exceed random guess performance on the same models.
this phenomenon still holds on the larger models, BERT (-), BLLOMM,,1B (-), X5xxL. (- ). Based on these findings, we can argue that that model scaling cannot adequately eliminate the im- pact of the number of tunable parameters the performance of PET methods.
than the high threshold of the smaller model. This phenomenon holds true across all tasks (SST2, RTE, MRPC), and for all series of models, as depicted in all sub-graphs. Therefore. we can conclude that model scaling enables tuning methods train fewer necessary parameters while achieving the fine-tuning. This conclusion can intuitively explain why APET methods can achieve relatively similar per- formance on larger models, especially on T5XXL- as illustrated in the aforementioned [c.2] in Figure 2. This is due to the fact that the number of tun- able parameters in each group of APET methods surpasses the high parameter thresholds on T5xxL; hence, they all achieve the similar performance of calameaaeettrr fine-tuning.
Interestingly, we find two parameter thresholds for tunable parameters in all models and name them as low parameter threshold (Low, Low, Low, Low, Low, Low) for necessary tuned parameters and the high parameter threshold (High, High, High, High, High, High) for necessary tuned pa- rameters, respectively in Figure 5. When tunable parameters are more than low parameter thresh- old, the APET method
similar performance
exceed random per- formance (c.g., Numher label types % on BERT, 0% on BLOOM, and 0% on T5); when the tunable parameters are more than high parameter thresh- old, the APET method can almost achieve the full- parameter fine-tuning (FT) performance. Further- more, we find that the model scaling affects the two parameter thresholds. Hence, we explore this phenomenon in the following paragraphs.
1x000
Low Threshold of Necessary Tuned Parameters From the above results, we find that APET methods will exceed the random guess performance (0% on T5; 0% on BLOOM; 50% on BERT) and immedi- ately reach the 80-90% full-parameter fine-tuning performance when the tunable parameters more than low thresholds. However, the low thresholds are relatively higher on larger models (BERT LARGE- BLOOMM T5xx)). Namely, APET methods require more tunable parameters to exceed the ran- dom guess performance. This phenomenon is con-
High Threshold of Necessary Tuned Parameters Based on the experimental results in the sub- graph [c. 1] (SST2) of Figure 5, we find that the high threshold of the larger model is consistently lower

---

sistent over all tasks on all series of models. Hence, we can infer that the model scaling cannot reduce the number of necessary tuned parameters to drive PLMs to perform downstream tasks.
arbitrary selection of tunable parameters for tun- ing without greatly degrading performance (Desai et al., 2019; Chen et al., 2020; Prasanna et al., 2020; Evei et al., 2020); thus, controllers (modules) might have higher degrees of freedom.
Furthermore, it is worth noting that the low parameter thresholds of the APET methods al- most lie in the same range on the same models. Specifically, the range of low thresholds are in BERT and [4. 0e+2 1. 1.6e+3] on SMALL [8. 2e+3, 4. 1e+4] on BLOOM [8. 2e+3, 4. 3] on 5L00M
This might explain why the aribitray positions of the tunable parameters have less impact such that all PET methods can achieve the similar perfor- mance on the larger models, It is worth noting that even though the distribution of tunable parameters have less impact on the performance, it still affects converge speeds. Thus, finding a better parame- ter distribution to improve the converge speeds for PET methods is a direction worthy of exploring.
[8. 18.0e+2, 3.2e+3]
17.9e+3, 8. 3e on X5xXI. [8.0e* 0e+2, 7. 3e+3] on T5 We will explain this phenomenon from the optimization perspective in $ 6.
6 Discussing the Ablation Results from the Optimization Perspectives
Why model scaling leverages the fewer tun- able parameters to achieve full-parameter fine- tuning performance? Tuning @ to steer PLM towards downstream NLP tasks can be seen adaptations. From the perspective of representa- tion space, the adaptations of PET methods can be re-parameterized into a unified low dimensional subspace (Qin et al., 2021; Aghajanyan et al., 2021; Yi et al., 2022). Aghajanyan et al. (2021) further demonstrate that adaptation on larger PLM can be re- parameterized into the lower dimensional space; this implicitly explains why PET methods can op- timize fewer parameters on larger-scale models, e.g., X5xxL to meet the full-parameter fine-tuning performance on tasks.
The objectives of all parameter-efficient tun- ing methods (PET, APET) can be expressed as ming [ (( (X), Y) as introduced in Equation (3), where 0 is a tunable module. The module 0 of different PET methods consists of different structures and varying numbers of tunable param- eters. In this paper. we investigate the impact of model scaling on different modules, which possess varying numbers tunable parameters distributed across multiple positions. We find that the larger model scaling can (1) mitigate the effects caused by the difference positions of tunable parameters (S 5.2) and (2) make PET methods optimize fewer tunable parameters to achieve full- uull-parmmeter fine- tuning performance 5.3). To further fathom these phenomena, we will investigate the underlying rea- sons from an optimization perspective. (3) Besides, we also observe that PET methods can optimize almost the similar number of necessarily tuned pa- rameters to exceed random guess performance on the same backbone models ($ 5.3). Although phe- nomenon (3) is not caused by model scaling, we also explain it from the optimization perspec- tive. Next, together discuss and the above two findings (1) and (2) in the following paragraphs.
Why can PET methods optimize the similar numbers of tunable parameters exceed ran- dom guessing? As stated above, the adaptations of the PET methods can be re-parameterized into a unified subspace. Qin et al. (2021) show that this low dimensional subspace is shared among all NLP tasks for the same PET methods. Yi et al. (2022) further suggest that this subspace is also shared among various PET methods. This might implicitly explain why all PET methods can tune the similar numbers of necessary tuned parameters to exceed the random guessing performance on the same models, even for the different tasks (S 5.3).
Why model scaling mitigates the effects caused by the differences in positions of tunable param- eters on the PET performance? From the op- timal control perspective, tunable module (0) of tuning method can be seen as a controller ( (Yang and Liu, 2022: Ding et al., 2023) to drive PLMs towards downstream tasks. As the model scale increases, the larger model has higher parameter redundancy (Aghajanyan et al., 2021), allowing
Conclusion
The realm of model scaling for LLMs presents important and intriguing directions for the LLM community. The increasing of model scale unveils numerous emerging capabilities and advantages. this work, our primary emphasis is on the impact of model scaling as it pertains to PET methods.

---

Through our comprehensive observation studies and in-depth discussions from optimization per- spectives, we gain deeper insights into the effects of model scaling on PET and the reasons behind the observed phenomena. We believe that our findings will serve as catalyst, inspiring further meticulous research and exploration in this area.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko. Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vin- odkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, An- drew M. Dai, Thanumalayan Sankaranarayana Pil- lai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee. Zongwei Zhou, Xuezhi Wang. Brennan Sacta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellster Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022. Palm: Scaling language mad- eling with pathways.
8 Limitations
This paper might have some possible limitations as follows: (I) we only explore the effects of the scaling law on performance. There might be other research points worth exploring, such as the power of model scale to convergence speed; (2) we study the power of model scale with comprehensive em- pirical experiments and explain the findings from the optimization perspective. There might be more theoretical proofs to explain these exciting findings.
Shrey Desai, Hongyuan Zhan, and Ahmed Aly. 2019. Evaluating lottery tickets under distributional shifts. In Proceedings of the 2nd Workshop on Deep Learn- ing Approaches for Low-Resource NLP (DeepLo 2019), pages 153-162, Hong Kong, China. Asso- ciation for Computational Linguistics.
References
Armen Aghajanyan, Sonal Gupta, and Luke Zettle- moyer. 2021. Intrinsic dimensionality explains the Pro-
effectiveness of language model fine-tuning. ceedings of the 59th Annual Meeting of the Associa- tion for Computational Linguistics and the 1 1ith Inter- national Joint Conference on Natural Language Pro- cessing (Volume 1: Long Papers), pages 7319-7328, Online. Association for Computational Linguisties.
Jacob Devlin, Ming-Wer Wei Chang. Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long and Short Papers), pages 4171-41 Minneapolis, Minnesota. Association for Computational Linguistics.
Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. 2022. BitFit: Simple parameter-efficient fine-tuning for transformer-based masked language-models, In Proceedings of the 60th Annual Meeting of the As- sociation for Computational Linguistics (Volume 2: Short Papers), pages 1-9, Dublin, Ireland. Associa- tion for Computational Linguistics.
Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zong- han Yang. Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, Jing Yi, Weilin Zhao, Xiaozhi Wang. Zhiyuan Liu, Hai-Tac Zheng, Jianfei Chen, Yang Liu, Jie Tang. Juanzi Li, and Maosong Sun. 2023. Parameter-efficient fine-tuning of large- scale pre- pre-trained language models. Nature Machine Intelligence.
Johan Bos and Katja Markert. 2005. Recognising tex- tual entailment with logical inference. In Proceed- ings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing, pages 628-635, Vancouver, British Columbia, Canada. Association for Computa- tional Linguistics.
William B. Dolan and Chris Brockett. 2005. Automati- cally constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005).
Jiaao Chen, Aston Zhang, Xingjian Shi, Mu Li. Alex Smola, and Diyi Yang. 2023. Parameter-efficient fine- tuning design spaces. In The Eleventh International Conference Learning Representations.
Nan Du, Yanping Huang, Andrew M. Dai, Simon Tong. Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yangi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus, Maarten P. Bosma, Zongwei Zhou, Tao Wang, Yu Emma Wang, Kellie Webster, Marie Pellat, Kevin Robinson, Kathleen S. Meier- Hellstern, Toju Duke, Lucas Dixon, Kun Zhang. Quoc V. Le, Yonghui Wu, Zhifeng Chen, and Claire
Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Zhangyang Wang, and Michael Carbin. 2020. The lottery ticket hypothesis for pre- trained BERT networks. In Advances in Neural In- formation Processing Systems 33: Annual Confer- ence on Neural Information Processing Systems 2020. NeurIPS 2020, December 6-12, 2020, virtual.

---

Cui. 2022. Glam: Efficient scaling of language mod- els with mixture-of cfture-of-expeeee -f-eeptsts...... In International Con- ference on Machine Learning, ICMI. 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 5547-5569. PMLR.
Neil Houlsby, Andrei Giurgiu. Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Ges- mundo, Mona Attariyan, and Sylvain Gelly. 2019b. Parameter-effiiien transfer learning for NLP. In Pro- ceedings of the 36th International Conference on Ma- chine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 2790-2799. PMLR.
Utku Evci, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. 2020. Rigging the lottery: Making all tickets winners. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 2952- PMLR.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. 2022a. Lora: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.
Alexander Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir Radev. 2019. Multi-news: A large-scale multi-document summarization dataset and abstrac- tive hierarchical model. In Proceedings of the 57th Annual Meeting the Association for Computational Linguistics. pages 1074-1084, Florence, Italy. Asso- ciation for Computational Linguistics.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. 2022b. Lora: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25- 2022. OpenReview.net.
Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Alek- sander Wawer. 2019. SAMSum corpus: A human- annotated dialogue dataset for abstractive summa- rization. In Proceedings of the 2nd Workshop on New Frontiers in Summarization, pages 70-79, Hong Kong, China. Association for Computational Linguis- tics.
Shengding Hu, Zhen Zhang, Ning Ding, Yadao Wang. Yasheng Wang. Zhiyuan Liu, and Maosong Sun. 2022c. Sparse structure search for delta tuning. In Advances in Neural Information Processing Systems.
Neal Lawton, Anoop Kumar. Govind Thattai, Aram Galstyan, and Greg Ver Steeg. 2023, Neural archi- tecture search for parameter-efficient fine-tuning of large pre-trained language models. In Findings of the Association for Computational Linguistics: ACL 2023, pages 8506-8515, Toronto, Canada. Associa- tion for Computational Linguistics.
Demi Guo, Alexander Rush, and Yoon Kim. 2021. Parameter-efficient transfer learning with diff prun- ing. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Lan- guage Processing (Volume 1: Long Papers), pages 4884-4896, Online. Association for Computational Linguistics.
Kenton Lee, Ming- Wei Chang, and Kristina Toutanova. 2019. Latent retrieval for weakly supervised open domain question answering. In Proceedings of the 57th Annual Meeting of the Association for Computa- tional Linguistics, pages 6086-6098 Florence, Italy. Association for Computational Linguistics.
Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg- Kirkpatrick, and Graham Neubig. 2022a. Towards a unified view of parameter-efficient transfer learning. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25- 29, 2022. Openkeview.net.
Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficieet prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3045-3058 Online and Punta Cana, Domini- can Republic. Association for Computational Lin- guistics.
Shwai Hc, Liang Ding, Daize Dong, Jeremy Zhang, and Dacheng Tao. 2022b. SparseAdapter: An easy approach for improving the parameter-efficiencyy of adapters. Findings of the Association for Computa- tional Linguistics: EMNLP 2022, pages 2184-2190, Abu Dhabi. United Arab Emirates. Association for Computational Linguistics.
Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguistics and the I 11th International Joint Conference on Natural Language Processing ( (Volume 1: Long Papers), pages 4582- 4597, Online. Association for Computational Lin- guistics.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Ges- mundo, Mona Attariyan, and Sylvain Gelly. 2019a. trrameter-effiiion transfer learning for NLP. In Pro- ceedings of the 36th International Conference on Ma- chine Learning, ICMI. 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 2790-2799. PMLR.
Andrew L. Maas, Raymond F. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 201 Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the

---

Association for Computational Linguistics: Human Language Technologies, pages 142-150, Portland, Oregon, USA. Association for Computational Lin- guistics.
Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. J. Mach. Learn. Res, 21: 21:140:1-140:67
Teven Le Scao, Angela Fan, Christopher Akiki, El- lie Pavlick, Suzana Ilié, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, Jonathan Tow, Alexander M. Rush, Stella Biderman, Albert Webson, Pawan Sasanka Am- manamanchi, Thomas Wang. Benoît Sagot, Niklas Muennighoff, Albert Villanova del Moral, Olatunji Ruwase, Rachel Bawden, Stas Bekman, Angelina McMillan-Major Iz Beltagy, Huu Nguyen. Lucile Saulnier, Samson Tan, Pedro Ortiz Suarez, Vic- tor Sanh, Hugo Laurençon, Yacine Jernite, Julien Launay, Margaret Mitchell, Colin Raffel, Aaron Gokaslan, Adi Simhi, Aitor Sorca, Alham Fikri Aji, Amit Alfassy, Anna Rogers, Ariel Kreisherg Nitzav, Canwen Xu, Chenghao Mou, Chris Emezue, Christopher Klamm, Colin Leong, Daniel van Strien. David Ifeoluwa Adelani, Dragomir Radev, Ed- uardo González Ponferrada. Efrat Levkovizh, Ethan Kim, Eyal Bar Natan, Francesco De Toni. Gérard Dupont, Germán Kruszewski, Giada Pistilli. Hady Elsahar, Hamza Benyamina, Hicu Tran, Ian Yu, Idris Abdulmumin, Isaac Johnson, Itziar Gonzalez-Dios Javier de la Rosa, Jenny Chim, Jesse Dodge, Jian Zhu, Jonathan Chang, Jörg Frohberg, Joseph Tobing, Joy- deep Bhattacharjee. Khalid Almubarak, Kimbo Chen. Kyle Lo, Leandro Von Werra, Leon Weber, Long Phan, Loubna Ben allal, Ludovic Tanguy, Manan Dey, Manuel Romero Muñoz, Maraim Masoud, María Grandury, Mario Šaško, Max Huang, Max- imin Coavoux, Mayank Singh, Mike Tian-Jian Jiang, Minh Chien Vu, Mohammad A. Jauhar, Mustafa Ghaleb, Nishant Subramani, Nora Kassner, Nuru- laqilla Khamis, Olivier Nguyen, Omar Espejel, Ona de Gibert, Paulo Villegas, Peter Henderson, Pierre Colombo, Priscilla Amuok, Quentin Lhoest, Rheza Harliman, Rishi Bommasani, Roberto Luis López. Rui Ribeiro. Salomey Osei, Sampo Pyysalo, Se- bastian Nagel, Shamik Bose, Shamsuddeen Hassan Muhammad. Shanya Sharma, Shayne Longpre, So- maich Nikpoor, Stanislav Silberberg, Suhas Pai, Syd- ney Zink, Tiago Timponi Torrent, Timo Schick, Tris- tan Thrush, Valentin Danchev, Vassilina Nikoulina, Veronika Laippala, Violette Lepercq. Vrinda Prabhu, Zaid Alyafeai, Zeerak Talat, Arun Raja, Benjamin Heinzerling, Chenglei Si. Davut Emre Taşar, Eliz- abeth Salesky, Sabrina J. Mielke, Wilson Y. Lee. Abheesht Sharma, Andrea Santilli, Antoine Chaffin, Amaad Stiegler, Debajyoti Datta, Eliza Szezechla, Gunjan Chhablani. Han Wang. Harshit Pandey, Hen- drik Strobelt, Jason Alan Fries, Jos Rozen, Leo Gao, Lintang Sutawika, M Saiful Bari, Maged S. Al-shaibani, Matteo Manica. Nihal Nayak. Ryan Teehan, Samuel Albanie, Sheng Shen, Srulik Ben- David, Stephen H. Bach, Taewoon Kim, Tali Bers, Thibault Fevry, Trishala Neeraj. Urmish Thakker, Vikas Raunak, Xiangru Tang. Zheng-X0 Xin Yong. Zhiqing Sun, Shaked Brody, Yallow Uri, Hadar Tojarich, Adam Roberts, Hyung Won Chung, Jac- sung Tae, Jason Phang, Ofir Press, Conglong Li. Deepak Narayanan, Hatim Bourfoune, Jared Casper,
Rabech Karimi Mahabadi, James Henderson, and Se- bastian Ruder. 2021a. Compacter: Efficient low-rank hypercomplex adapter layers. In Advances in Neural Information Processing Systems 34: Anmual Confer- ence on Neural Information Processing Systems 2021, NeurlPS 2021, December 6-14, 2021, virtual. pages 1022-1035.
Rabech Karimi Mahabadi, James Henderson, and Se- bastian Ruder. 2021b. Compacter: Efficient low-rank hypercomplex adapter layers. In Advances in Neural Information Processing Systems 34: Annual Confer- ence on Neural Information Processing Systems 2021, NeurlPS 2021, December 6-14, 2021, virtual. pages 1022-1035.
Rabech Karimi Mahabadi, James Henderson, and Se- bastian Ruder. 2021c. Compacter: Efficient low-rank hypercomplex adapter layers. In Advances in Neural Information Processing Systems 34: Annual Confer- ence on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual. pages 1022-1035.
Bo Pang and Lillian Lee. 2005. Seeing stars: Exploit- ing class relationships for sentiment categorization with respect to rating scales. In Proceedings of the 43rd Annual Meeting of the Association for Compu- tational Linguistics (ACL '05), pages 115-124, Ann Arbor, Michigan. Association for Computational Lin- guistics.
Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé Kyunghyun Cho, and Iryna Gurevych. 2021. AdapterFusion: Non-destruction task composition for transfer learning. In Proceedings of the 16th Con- ference of the European Chapter of the Association for Compulational Linguistics: Main Volume, pages 487-503, Online, Association for Computational Lin- guistics.
Sai Prasanna, Anna Rogers, and Anna Rumshisky, 2020. When BERT Plays the Lottery, All Tickets Are Win- ning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3208-3229, Online. Association for Computational Linguistics.
Yujia Qin. Xiaozhi Wang, Yusheng Su, Yankai Lin, Ning Ding, Zhiyuan Liu, Juanzi Li, Lei Hou, Peng Li, Maosong Sun, et al. 2021. Exploring low- dimensional intrinsic task subspace via prompt tun- ing. ArXiv preprint, abs/21 10.07667.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language under- standing by generative pre-training.
Colin Raffel, Noam Shazeer. Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yangi Zhou,

---

Jeff Rasley, Max Ryabinin, Mayank Mishra, Minjia Zhang, Mohammad Shoeybi, Myriam Peyrounette, Nicolas Patry. Nouamane Tazi, Omar Sanseviero, Patrick von Platen, Pierre Cornette, Pierre François Lavallée, Rémi Lacroix, Samyam Rajbhandari, San- chit Gandhi, Shaden Smith, Stéphane Requena, Suraj Patil, Tim Dettmers, Ahmed Baruwa, Amanpreet Singh, Anastasia Cheveleva, Anne-Laure Ligozat, Arjun Subramonian, Aurelie Névéol, Charles Lover- ing. Dan Garrette. Deepak Tunuguntla, Ehud Reiter, Ekaterina Taktasheva, Ekaterina Voloshina, Eli Bog- danov, Genta Indra Winata, Hailey Schoelkopf, Jan- Christoph Kalo, Jekaterina Novikova, Jessica Zosa Forde, Jordan Clive, Jungo Kasai, Ken Kawamura, Liam Hazan, Marine Carpuat, Miruna Clinciu, Na- joung Kim, Newton Cheng, Oleg Serikov, Omer Antverg, Oskar van der Wal, Rui Zhang, Ruochen Zhang, Sebastian Gehrmann, Shachar Mirkin, Shani Pais, Tatiana Shavrina, Thomas Scialom, Tian Yun, Tomasz Limisiewicz. Verena Rieser, Vitaly Protasov, Vladislav Mikhailov, Yada Pruksachatkun, Yonatan Belinkov, Zachary Bamherger, 7eennč Kasner, Al- ice Rueda, Amanda Pestana, Amir Feizpour, Am- mar Khan, Amy Faranak, Ana Santos, Anthony Hevia, Antigona Unldreaj. Arash Aghagol, Are- zoo Abdollahi, Aycha Tammour, Azadeh HajiHos- seini, Bahareh Behroozi, Benjamin Ajibade, Bharat Saxena, Carlos Muñoz Ferrandis, Danish Contrac- tor, David Lansky, Davis David, Douwe Kicla, Duong A. Nguyen, Edward Tan, Emi Baylor, Ez- inwanne Ozoani, Fatima Mirza, Frankline Onon- iwu, Habib Rezanejad, Hessie Jones, Indrani Bhat- tacharya, Irene Solaiman, Irina Sedenko. Isar Ne- jadgholi, Jesse Passmore, Josh Seltzer, Julio Bonis Sanz, Livia Dutra, Mairon Samagaio, Maraim EI- badri, Margot Mieskes, Marissa Gerchick, Martha Akinlolu, Michael McKenna, Mike Qiu, Muhammed Ghauri, Mykola Burynok, Nafis Abrar, Nazneen Ra- jani. Nour Elkott, Nour Fahmy, Olanrewaju Samuel, Ran An, Rasmus Kromann, Ryan Hao, Samira Al- izadeh, Sarmad Shubber, Silas Wang, Sourav Roy, Sylvain Viguier. Thanh Le. Tobi Oyebade. Trieu Lc. Yoyo Yang. Zach Nguyen, Abhinav Ramesh Kashyap, Alfredo Palasciamo, Alison Callahan, Anima Shukla, Antonio Miranda-Escalldaa Ayush Singh. Benjamin Beilharz, Bo Wang, Caio Brito, Chenxi Zhou, Chirag Jain, Chuxin Xu, Clémentine Fourrier, Daniel León Periñán, Daniel Molano, Dian Yu, Enrique Manjava- cas, Fabio Barth, Florian Fuhrimann, Gabriel Altay, Giyaseddin Bayrak, Gully Burns, Helena U. Vrabec, Imane Bello, Ishani Dash, Jihyun Kang, John Giorgi. Jonas Golde. Jose David Posada, Karthik Ranga- sai Sivaraman, Lokesh Bulchandani, Lu Liu, Luisa Shinzato, Madeleine Hahn de Bykhovetz, Maiko Takeuchi, Mare Pamies, Maria A Castillo, Mari- anna Nezhurina, Mario Sänger, Mathhas Samwald, Michael Cullan, Michael Weinberg, Michiel De Wolf, Mina Mihaljeic, Minna Liu, Moritz Freidank, Myungsun Kang, Natasha Seelam, Nathan Dahlberg, Nicholas Michio Broad, Nikolaus Muellner, Pascale Fung, Patrick Haller, Ramya Chandrasckhar, Renata Fisenberg, Robert Martin, Rodrigo Canalli, Rosaline Su, Ruisi Su, Samuel Cahyawijaya, Samuele Garda,
Shlok S Deshmukh, Shubhanshu Mishra, Sid Ki- blawi, Simon Ott, Sinee Sang- -rooossooiiii Srishti Ku- mar, Stefan Schweter, Sushil Bharati, Tanmay Laud, Théo Gigant, Tomoya Kainuma, Wojciech Kusa, Ya- nis Labrak. Yash Shailesh Bajaj, Yash Venkatraman, Yifan Xu, Yingxin Xu, Yu Xu, Zhe Tan, Zhongli Xie, Zifan Ye, Mathilde Bras, Younes Belkada, and Thomas Wolf. 2023. Bloom: A 176b- webb-paraeeter open-access multilingual language model.
Lakshay Sharma, Laura Graesser, Nikita Nangia, and Utku Evci. 2019. Natural language understanding with the quora question pairs dataset. ArXiv preprint, abs/1907.01041.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013, Recursive deep models for semantic compositionality over sentiment treebank. In Proceedings of the 2013 Conference on Empiri- cal Methods in Natural Language Processing, pages 1631 1-31-1 Seattle, Washington, USA, Association for Computational Linguistics.
Yusheng Su, Xiaozhi Wang. Yujia Qin, Chi-Min Chan, Yankai Lin, Huadong Wang, Kaiyue Wen, Zhiyuan Liu, Peng Li, Juanzi Li, Lei Hou, Maosong Sun, and Jie Zhou. 2022. On transferability of prompt tuning for natural language processing. In Proceedings of the 2022 Conference of the North American Chap- of the Association for Computational Linguistics: Human Language Technologies, pages 3949-3969, Seattle, United States. Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones. Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Pro- cessing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA. pages 5998-6008
Tu Vu, Brian Lester, Noah Constant, Rami AI-Rfou', and Daniel Cer. 2022. SPoT: Better frozen model adaptation through soft prompt transfer. In Proceed- ings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa- pers). pages 1039,,509,,1 Dublin, Ireland. Association for Computational Linguistics.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. GLUE: A multi-task benchmark and analysis plat- form for natural language understanding. In 7th In- ternational Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph. Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022a. Emer- gent abilities of large language models. Transactions on Machine Learning Research.

---

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian lchter, Fei Xia, Ed Chi, Quoe Le, and Denny Zhou. 2022b. Chain of thought prompting clicits reasoning in large language models.
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sen- tence understanding through inference. In Proceed- ings of the 2018 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies, Volume I (Long Papers), pages 1112-1122, New Orleans, Louisiana. Association for Computational Linguis- tics.
Zonghan Yang and Yang Liu. 2022. On robust prefix- tuning for text classification. The Tenth Inter- national Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022, Open- Review.net.
Jing Weize Chen, Yujia Qin, Yankai Lin, Ning Ding. Xu Ham, Zhiyuan Liu, Maosong Sun, and Jie Zhou. 2022. Different tunes played with equal skill: Explor- ing unified optimization subspace for delta tuning.
Han Zhou, Xingchen Wan, Ivan Vulić, Anna Korho- nen. 2023. Autopeft: Automatic configuration search for parameter-efficht fine-tuning arXiv preprint arXiv:2301.22132.

---

A Task and Dataset
where n is sequence length, d is the hidden di- mension, W Ruxd is query, W k Raxd is key, and W E Raxd is value. The output from each attention head will be concatenated and fur- ther transformed by W. E gd and be denoted
We use various NLP tasks to evaluate the APET methods, which can be divided into the following 5 categories:
q
Sentiment Analysis (SA) SA tasks evaluate if a model can correctly predict the sentiment labels of input sentence. this paper, we choose SST-2 (Socher et al., 2013), IMDB (Maas al., 2011). and Rotten Tomatoes (Pang and Lee, 2005).
ĪAHA = concat(h h2,... ha) W Or
(9)
where hμи E RRnd is the output hidden state of attention layer. After that, h will be fed into a two-layer feed-forward network
Natural Language Inference (NLI) NLJ tasks evaluate model's ability to correctly classify if hypothesis can be entailed or not given a premise. In this paper, choose MNLI (Williams et al., 2018), QNLI (Wang et al., 2019), and RTE (Bos and Markert, 2005).
hFFN = ohW,, + b1) b2,
(10)
where Wi E RRdxdm W₂ E IR xd b E IR b E Rd and dm > d is an integer.
During the forward pass through each (trans- former) block, the input hidden state is applied with the sequence of layers. For simplicity, we formalize the transformation of each layer as
Paraphrase Identification (PI) tasks evalu- ate if a model can correctly identify paraphrases, which means two sentences are identical in se- mantic meaning. In this paper, we choose MRPC (Dolan and Brockett, 2005), and QQP (Sharma et al., 2019).
h = f(h
(11)
Under the layer as the operator f. the input hidden state h E Rxd is transformed into the output hidden state hou E Baxd where is the input length, and d is the dimension.
Question Answering (QA) QA tasks evaluate a model's s ability to answer questions. Context may be present. In this paper, we choose NQ-Open (Lee et al., 2019), an open-world QA dataset without context.
B.2 Implementation Details of PET Methods
Summarization (SUM) SUM tasks evaluate model's ability summarize long paragraph into shorter abstract without loosing the semantics of the original text. In this paper, we choose SamSUM (Gliwa et al., 2019), and Multi-News (Fabbri et al., 2019) in our experiments.
Prompt Prompt-tuning (Lester al., 2021) prepends Np tunable soft tokens, i.c. embeddings, to the input sentences and asks the model to predict the probability of the next word. During training, only the newly added embeddings are optimized and the backbone model is frozen.
B Parameter-efficient Tuning (PET) Methods
BitFit BitFit (Ben Zaken al., 2022) is method that only tunes all the bias terms W,, Rd in the PLM, which lie the self-attention and layer norm layers.
Here, first recap the PLM (transformer) layer. Then, we describe the detail and training configu- rations of the PET methods shown in Figure 1.
LoRA LoRA (Hu et al., 2022b) is method that adapts PLM in a low-rank space. It down-projects the attention weights into a lower dimension and up-projects it back to the original dimension. Only these projection weights are optimized.
B.1 Transformer Architecture
A PLM generally a stack of multiple Transformer layers, each composed of a multi-headed attention and a feed-forward network. The multi-headed attention contains h attention heads working in parallel. Specifically, given an input X E Roxd the i-th attention head works follows:
Adapter Adapter (Houlsby et al., 2019b) is method that only tunes the inserted adapter modules, which consist of down projection, non- linear transformation, up projection, and skip- connection. For each existing Transformer layer in a PLM, the adapter modules are inserted at two
(WW
X
h = sof tmax
(XW))), (8)
Vd/h

---

Module
C Arbitrary Parameter-Efftciet Tuning (APET) Methods
ADD
We introduce a more flexible PET method, Arbitrary Garameter-Effictee Tuning (APET) method. Its tunable module be arbitrary struc- ture that facilitates structures (parameter position) and easier control the number of tunable parameters.
Tunable
explore various module
Weight
W
CONCAT
C.1 Implementation Details of APET Methods
PLM
Layer
W PLM
PLUG
As previously introduced in 5.1, the tumable module of the APET method is composed of tun- able weights. Each tunable weight can be expressed as W. Here, we have three operations insert the tunable weight W into the PLM to modify the spe- cific layers and their corresponding transformations as follows:
Tunable Parameters
Parameters
Figure 6: The tunable modules of APET methods are composed p tunable weights W, which can be pressed as / = [1 W2,... Wz)- We introduce three operations to insert the tunable weight W into the PLM and the corresponding transformation.
ADD We will add the tunable weight W into the PLM layer. The corresponding transformation can be denoted as out,
1(h WI.
(12)
CONCAT We will concatenate the tunable weight W and the hidden state or the layer in the PLM. The corresponding transformation can be denoted hout
locations: (1) after the first feed-forward layer, and (2) after the two consecutive feed-forward layers. During training, only the adapter modules are opti- mized and the rest of the PLM is frozen.
f(W,) A(h + ah WW, W4
(13)
B.3 Training Configurations of PET Methods
PLUG We will plug the tumable weight W be- tween PLM layers. The corresponding transforma- tion can be denoted as hou.
The tunable module of PET method is com- posed of I. tunable weights W (all tunable weights) of the specific PET method, which can be expressed (W1, W2; .... W.). We also follow Equa- tion (3) to train the PET method. During training, we only optimize while freezing the rest of the parameters in the PLM. We adopt batch size of 32 and have no warm-up for most the PET mod- els and tasks. The maximum input length is 128 for single sentence tasks (SA) and 256 for multi- sentence tasks (NLI, PI, QA, SUM). The maximum generation length is 1 for classification tasks (SA, NLI, PI), 64 for Multi-News, and 128 for SAM- Sum. On the BERT, BLOOM, T5 models, we set their learning rates as (3e-4), (3e-4, 5e-5), (le-4, 1c-3, 1c-2) respectively. Then, we choose the best performance to report.
s(h²) ( (I(h") W,Ws).
(14)
According to these operations and the corre- sponding transformations, APET methods as hnd.
express the
W f(W2)
f(h²m) +
oh W, W. a(Shhm W:
(15)
By comparing the Equation (15) with the equations of the previously introduced PET methods, we can clearly find that the PET methods are special cases of APET methods.

---

IBERT
100
-10
300
12
11
16
92
73
3X0
47
306
300
60
50
72
109
100
43
20
C0
SEX
53
XCC
33
24
42
23
342
33
300
43
26
73
200
CC
204
205
37
34
50
SCC
33
43
23
5L
50
200
74
33
58
100
52
ICC
45
52
96
100
76
43
74
[a.1] APT ChNesor
I APT
[ APATTOO Toooe Iian
APT Touer 11yml
5010
13
1161
IEC
18
1463
1:330
91
53 33
K5
:20 93
0900
100
39
144
ATT
56 59
500
57 56
53
500
79
14
LCC
24
33
94
100
25
10 Doen
56
WD-5pm
14
15
57
50 50
57
73
20
24
60
29
120
39
27
21
47
27
40
100
33
20
50
22
25
56
100
Spent
TargesTaee
Target Tace
(a) BERT SMAZ and T5SMAL
(b) BERT ARGE and T5xx..
Figure 7: Relative performance (zero-shot transfer performance original performance) (%) on the target tasks (columns) of the APET methods trained on the source tasks (rows). Colors of the task names indicate task types.
C.2 Training Configurations of APET methods
APET
APETmaFa
APETLAA
APETadaa
BERT
5.14
1.444
6.6e14
2.0c45
The tunable module of APET method 0 is com- posed of L tunable weights W, which can be ex- pressed as 0- = (W1, W., Wi). We also fol- low Equation (3) to train the APET method. During training, we only optimize 0 while freezing the rest of the parameters in the PLM.
BERT,
1.0c55
1.7e45
7.9++5
2.4++6
BL00Ms0M
1.0455
2.7e45
7.9c+5
2.4c+6
BLOOM
4.1et5
1.446
3.9c+6
1.2c47
15
5.1e44
1.2045
2.3e+5
8.0c45
T5xaL
4.1045
2.5046
6.3e+6
1.9c47
Table 2: The table shows the numbers of tunable param- eters of tunable parameters in Figure 4 experiment.
Besides, we adopt a batch size of 32 and have no warm-up for most of the APET models and tasks. In addition, The maximum input length is 128 for single sentence tasks (SA) and 256 for multi-sentence tasks (NLI, PI, QA, SUM). The maximum generation length is for classification tasks (SA, NLI, PI), 64 for Multi-News, and 128 for SAMSum. On the BERT, BLOOM, T5 models, we set their learning rates as (3e-4), (3e-4, 5e-5), (le-4, le-3, le-2) respectively. Then, we choose the best performance to report.
between the NLP tasks in the zero-shot setting (Vu et al., 2022; Su et al., 2022; Ding al., 2023). In the experiments, we first train the parameters of APET methods on the source tasks and directly reuse them on the target tasks in zero-shot set- ting. We will investigate two series of PLMs T5 (T5 and T5xx1) and BERT (BERT SMALL and BERT,ARGE and report the relative performance.
D Number of Tunable Parameters of APET
Note that for different types of tasks, they are expected to share different groups of label sets (e.g. for task like SA, the labels are usually posi- tive/negative, whereas, for tasks like NLI, the labels usually entailment/not entailment). Reusing the parameters trained on the source task to test on the target task will naturally fail since the model is not able to generate the labels they have never seen in the training stage. To this end, we gener- ally map the original label sets to a unified label
Here, the Table 2 shows the number of tunable parameters of APET for each group in Figure 4.
E Power of Model Scale to Transferability
Furthermore, to explore whether the power of model scale can also facilitate generalization ability of tuning methods, we explore the transferability

---

set (e.g. negative/not entailment/falle -> O, posi- tive/entailmen/trree- 1). Utilizing unified label set makes it feasible to evaluate the transferability the AFP method among different types of tasks regardless of the divergence of original labels.
The results are shown in Figure 7, from which we can find that the APET (ISREEE and ADECEN) methods can transfer to the same type of tasks demonstrated by the darker color alongside the diagonal of the matrix and generally perform well both on small-scale PLMs (Figure 7 (a): SMALL and T5MMAL) and large-scale PLMs (Figure 7 (b): BERTLARGE and xx..)) How- ever, the lighter color indicates that APET meth- ods have difficulty performing different types of tasks overall, and both small-scale and large-scale PLMs share this phenomenon. This finding indi- cates that the power of scale does necessarily facilitate the gencralization ability of AFP methods which is in line with the prevalent assumption that fewer parameters often cause underfitting, whereas more parameters tend to cause overfitting. Never- theless, the mechanism behind this phenomenon still arouses our deep concern and is worth expand- ing that we will systematically analyze it in our future work.


Reviews:
[{"review_id": "Rl7lGxqKFF", "reviewer": "Reviewer_8CW5", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\nI sincerely appreciate your insightful comments. \nAfter reviewing the main contribution that you outlined, I have taken note that certain key information may have been inadvertently misunderstood or omitted. Consequently, the Area Chair (AC) has kindly informed me and suggested that I engage in a prior discussion with you through official comments.\n\nI will illustrate the main contributions presented in this paper and provide a more clear explanation of Section 5.3 to address your concerns. Hope the following explanations can make you better understand Section 5.3 and better evaluate the contributions of this work. \n\n\nContribution:\nThe main contribution of this paper is to identify that model scaling (larger models) can (1) mitigate the effects of the positions of tunable parameters on performance, and (2) enables tuning methods to achieve performance comparable to full-parameter fine-tuning by optimizing fewer tunable parameters.\n\nQuestion Response:\nRegarding Section 5.3, this section demonstrates the phenomenon in the aforementioned (2). Specifically, we found that that all parameter-efficient tuning (PET) methods can optimize fewer tunable parameters to achieve the full-parameter fine-tuning performance (upper bound performance). Besides, we can find that various PET methods exhibit two distinct parameter thresholds: a low threshold and a high threshold. When the number of tunable parameters reaches the low threshold, the performance of the PET method surpasses that of random guessing. When reaching the high threshold, the PET method achieves performance comparable to full-parameter tuning. Interestingly, we observed that when different PET methods are applied to the same LLM backbone, their parameter low thresholds consistently lie within the range mentioned in Section 5.3 (lines 484-493).\n\nI trust that this information will assist you in comprehending this study and addressing any concerns you may have. Lastly, kindly hope that you consider re-evaluating this work. If you have any questions or problems further would like to discuss, please let me know. I will reply you as soon as possible here.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The authors present a thorough study of various PET techniques, and also propose a novel one to set the experimental framework on a solid footing.", "reasons_to_reject": "I could not fully understand the case for invariance of low/high parameter thresholds, so I am not ready to accept the resulting conclusions.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper addresses an important topic, particularly parameter efficient tuning (PET) through the use of techniques that add a handful of parameters to (very) large models and optimize only those parameters on domain/task specific data sets in an incremental training stage.\n\nSuch techniques have been shown to be able to achieve the same performance gains as full model fine-tuning/incremental training on the data set of interest and so are very attractive particularly when it comes to serving large models: the base model parameters can be re-used across many tasks, and only the task adaptation \"stub\" needs to be swapped when serving requests of a known, specific type.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Section 5.3 and in particular l484-493 are very hard to follow."}, "extra_text": {}}, {"review_id": "ugkjzfgiSw", "reviewer": "Reviewer_fvk4", "summary": "", "questions": "1. If the author could address the concerns in the weakness section, I will further improve my scores, especially for point 3, 4. \n2. Prefix-tuning (prompt tuning) can leverage reparameterization, and the parameter amount learned by additional MLP layers can give significant impacts to the performance. Have you considered studying this?\n\n---------------------\n### After Rebuttal\n\nI appreciate the response from the author. I am convinced now that the current experiments are solid for delievering the findings. I have increased my score for soundness and keep the excitement score unchanged. I appeciate if the author could include more PEFT results in the camera-ready version.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer, \nI sincerely appreciate your insightful comments.\n\nResponse to Reasons to reject:\n\n- Response [2]: Actually, I have referenced certain papers in my work, such as UniPELT [1], S3PET [3], and so forth. However, some of citations correspond to earlier versions on Arxiv, which might lead to discrepancies. I am committed to rectifying this by updating the citations and incorporating any omitted references.\n\n- Response [3]: Yes, you are right. Actually, you can observe that we have also undertaken experiments concerning this aspect, depicted in Figure 4. Within this figure, APET is presented as a fusion of various insertion methods, with the adjustable parameters corresponding to each PET method expounded in the paper. Therefore, if you have an interest in exploring the efficacy of the amalgamation of different insertion methods, kindly refer to Figure 4. Doing so, you will lead you to the same findings: with the increasing of model scale, the performance variation among APET methods become smaller.\n\n- Response [4]: Actually, although I mentioned Prompt, Adapter , LoRA, and BitFit in the paper, I cover more various and arbitrary PET methods in our implementation with APET. Hence, the tunable modules, can various structures and be distributed in any positions. I believe the continuations of APET must include Compactor, IA3 [6], LayerNorm Tuning [7], Parallel Adapter, and more (you could further refer to our appendix and the code). Since, PET methods are continuously proposed by months, I am hard to list all of them in the paper. Hope this implementation detail can mitigate your concerns and could understand.\n\nActually, although in the paper, I only mentioned Prompt, Adapter, LoRA, and BitFit as examples, it's worthy to note that the implementation of APET encompass a wider array of diverse and arbitrary PET methods (its the tunable modules, can various structures and be distributed in any positions). Consequently, its modules can assume various structures and be positioned flexibly throughout the architecture. I am confident that the configurations of APET encompass methods such as Compactor, IA3 [6], LayerNorm Tuning [7], Parallel Adapter, and more (for further details, you can refer to our appendix and the provided code). \n\nAs the landscape of PET methods continues to evolve over time, add every single method within the implementation becomes a challenging endeavor. But I've already covered all of them as more as possible in implementation. I hope that this level of implementation detail addresses your concerns and aids in your understanding.\n\n\nQuestions For The Authors:\n- Response[1]: Please refer to the aforementioned Response [3] and Response [4] in response to reasons to reject. If I do not reply well or you have further questions, please kindly let me know via official comments.\n\n- Response[2]: Yes, indeed, I have done so. In fact, I have also undertaken Prefix-tuning across three different series of LLMs and 9 distinct tasks. The outcomes can be presented in Figure in the paper as follows:\nBERT_{SMALL}: 73, BERT_{BASE}: 81, BERT_{LARGE}: 82\nBLOOM_{560M}: 55, BLOOM_{1.1B}: 59, BLOOM_{7.1B}: 65\nT5_{SMALL}: 64, T5_{BASE}: 63, T5_{XXL}: 76\nHowever, due to constraints in computational resources, I was able to conduct Prefix-tuning performance evaluations on T5-XLL across 9 tasks (the above T5_{XXL}: 76 are evaluated on only 9 tasks), as opposed to the intended 11 tasks. Regarding the aspect of thoroughness, while I have refrained from presenting the Prefix-tuning outcomes in the main body, I plan to further include these results within the appendix for your reference. Furthermore, concerning the implementation of APET, I would like to highlight that we integrate both prompt and Lora modules. This combination can inherently encompasses the Prefix-tuning methods as well. For a more comprehensive understanding, I encourage you to consult the provided code.\n\nI sincerely appreciate the valuable feedback provided by the reviewer. I am hopeful that these responses will aid in conveying the contributions of this paper more comprehensively and also help in addressing your concerns. Feel free to let me know if you have any further questions via official comments, I will reply you ASAP.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The experimental results are solid and as expected.\n2. The writing and presentation of this work is great.\n3. It gives useful empirical findings for PEFT researchers facing different scales of language models.", "reasons_to_reject": "1. The technical contribution of this paper is limited.\n2. The APET framework is related with other arbitrary PEFT frameworks: UniPELT [1], AutoPEFT [2], S3PET [3], PEFT Design Space [4], NASPEFT [5], but they are not cited nor discussed.\n3. The technical part of this paper can be improved by studying the effectiveness of mixture of different insertion methods and the most recommended PEFT solution for large language models.\n4. As an analysis paper, I will expect more types of PEFT modules to be studied in this paper, such as Compactor, IA3 [6], LayerNorm Tuning [7], Parallel Adapter. It will be interesting to see if the current conclusion applies to wider ranges of PEFT modules.", "questions_to_authors": "1. If the author could address the concerns in the weakness section, I will further improve my scores, especially for point 3, 4. \n2. Prefix-tuning (prompt tuning) can leverage reparameterization, and the parameter amount learned by additional MLP layers can give significant impacts to the performance. Have you considered studying this?\n\n---------------------\n### After Rebuttal\n\nI appreciate the response from the author. I am convinced now that the current experiments are solid for delievering the findings. I have increased my score for soundness and keep the excitement score unchanged. I appeciate if the author could include more PEFT results in the camera-ready version.", "paper_topic_and_main_contributions": "This paper provides a study of the performance of different PEFT methods across different scales of models. It describes interesting findings that model scaling can mitigate the effects of insertion positions of PEFT and the trainable parameter is a critical factor for achieving competitive performance for PEFT in comparison to FFT. It is delightful to read this paper, and the presentation, visuals, of this paper is great. However, the APET framework for studying the parameters is not new. Hence, the technical contribution of this paper is not the strength of this paper, but I appreciate the analysis of PEFT across scales for driving PEFT research further.  I will recommend the author to include the detailed performance of each experiment for each task in appendix for providing additional information rather than only providing an averaged number in the main section. The experiments are solid for the discussed 4 PEFT methods for deriving the corresponding findings and the results are as expected from PEFT researchers, but I expect for more PEFT modules to be studied. This work can give more technical contributions to the field if the author could also study the effectiveness of mixture of different insertion methods in a limited parameter budget, layer positions for inserting PEFTs, and the most recommended/universal PEFT solution for large language models. In conclusion, I am learning positive for this work given its experimental solidness, and will recommend for a findings-level of work given the limited technical contribution.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line 123: 2 dots."}, "extra_text": {"Missing_References": "1. Mao Y, Mathias L, Hou R, et al. UniPELT: A Unified Framework for Parameter-Efficient Language Model Tuning.\n\n2. Zhou H, Wan X, Vulić I, Korhonen A. AutoPEFT: Automatic Configuration Search for Parameter-Efficient Fine-Tuning.\n\n3. Hu S, Zhang Z, Ding N, et al. Sparse Structure Search for Parameter-Efficient Tuning. \n\n4. Chen J, Zhang A, Shi X, Li M, Smola A, Yang D. PARAMETER-EFFICIENT FINE-TUNING DESIGN SPACES. \n\n5. Lawton N, Kumar A, Thattai G, Galstyan A, Steeg GV. Neural Architecture Search for Parameter-Efficient Fine-tuning of Large Pre-trained Language Models. \n\n6. Liu H, Tam D, Muqeeth M, et al. Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning.\n\n7. Giannou A, Rajput S, Papailiopoulos D. The Expressive Power of Tuning Only the Norm Layers."}}, {"review_id": "nYCgOFKgMY", "reviewer": "Reviewer_rQAh", "summary": "", "questions": "1. In Table 1, for prompt method, is it right that $\\Delta h=f(W_{prompt})$? If it is right, then for every input (also can be represented by $h^{in}$), the $\\Delta h=f(W_{prompt})$ is always the same.  I think $f$ should be a function of both $W_{prompt}$ and input.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer, I sincerely appreciate your insightful comments.  After reviewing the main contribution that you outlined, I have taken note that certain key information may have been inadvertently misunderstood or omitted. Consequently, the Area Chair (AC) has kindly informed me and suggested that I engage in a prior discussion with you through official comments.\n\nResponse to Reasons to reject:\n- Response [1]: \nFirstly, I would like to emphasize the primary objective of our work, as clearly highlighted in our title: \"Exploring the Impact of Model Scaling on Parameter-efficient Tuning\". Our primary contribution revolves around the exploration of model scaling, not the introduction of APET as a novel method. \nSecondly, I would like to clarify a crucial aspect: \nOur intention with APET was never to present it as a mere combination of existing PET methods. Instead, APET is utilized as an analytical tool in our work, aiding us in delineating our main observations.\nThe core contribution of our paper lies in the observations we derived using APET:\n\n    - 1.The influence of tunable parameter positions on performance diminishes with **model scaling**.\n\n    - 2.**Larger** models can achieve performance comparable to full-parameter fine-tuning by optimizing a reduced number of tunable parameters, in contrast to smaller models.\n\n- Response [2]: The tunable module (comprising a varying number of tunable parameters) within APET can adopt arbitrary shapes and dimensions. Consequently, it becomes convenient to regulate the overall count of tunable parameters to. I am committed to incorporating this description into the paper. Additionally, for a more comprehensive understanding, I encourage you to refer to the appendix and the provided code, which offer further visualizations and detailed insights.\n\nI sincerely appreciate the valuable feedback provided by the reviewer. I am hopeful that these responses will aid in conveying the contributions of this paper more comprehensively and also help in addressing your concerns. Feel free to let me know if you have any further questions via official comments, I will reply you ASAP.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.  The paper is well written and well structured.\n2. The author concludes the relationship bewteen tunable parameter positions, quantity and model scaling, although they are a bit obvious.", "reasons_to_reject": "1. It seems that the APET they proposed is just a combination of different PET methods, which is also introduced in the paper \"Towards a Unified View of Parameter-Efficient Transfer Learning\". Therefore, it should not be regarded as a contribution. \n2. Some parts of the paper are not described clearly. For example, in Section 5.2, to investigate the position impact,  the number of tunable parameters should be fixed. How do you achieve this with APET? Can you describe it more detailly? Moreover, most of the analysis focuses on the \"within group comparision. I think this section should focus on comparisions between different groups.", "questions_to_authors": "1. In Table 1, for prompt method, is it right that $\\Delta h=f(W_{prompt})$? If it is right, then for every input (also can be represented by $h^{in}$), the $\\Delta h=f(W_{prompt})$ is always the same.  I think $f$ should be a function of both $W_{prompt}$ and input.", "paper_topic_and_main_contributions": "This paper focuses on parameter-efficient tuning. Specifically, they firstly study the impact of model scaling on different parameter-efficient tuning (PET) methods and then propose the arbitrarily PET to investigate the impact of tunable parameters positions and quantity.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work focuses on parameter-efficient tuning (PET) and the impact of the pretrained model size on PET methods and design differences. The authors found that as the pretrained model scale increases, the performance differences among PET methods decrease. They explore the impact of model scale in terms of the position and the number of tunable parameters.   

The authors present a comprehensive study including several tasks and different PET methods/designs in the form of a unified framework (APET) for analysis. As reviewers commented, the paper presents empirical findings based on solid experimental results that are helpful for researchers who study PET methods. Furthermore, I believe authors will incorporate the reviewers suggestions and points together with their adiditonal results to the final version of the paper.
Decision: Accept-Main