Parameter-Efficient Fine-Tuning Design Spaces
Authors: Jiaao Chen
Aston Zhang
Xingjian Shi
Mu Li
Alex Smola
Diyi Yang
Venue: ICLR 2023 poster (2023)
Keywords: parameter-efficient fine-tuning
design spaces

Abstract: Parameter-efficient fine-tuning aims to achieve comparable performances of fine-tuning with much fewer trainable parameters. Recently, various tuning strategies (e.g., Adapters, Prefix Tuning, BitFit, and LoRA) have been proposed. However, their designs are hand-crafted separately, and it remains unclear whether certain design patterns exist for parameter-efficient fine-tuning. Thus, we present a parameter-efficient fine-tuning design paradigm and discover design patterns that are applicable to different experimental settings. Instead of focusing on designing another individual tuning strategy, we introduce parameter-efficient fine-tuning design spaces that parameterize tuning structures and tuning strategies. Specifically, any design space is characterized by four components: layer grouping, trainable parameter allocation, tunable groups, and strategy assignment. Our comprehensive empirical study leads to the discovery of design patterns: (i) grouping layers in a spindle pattern, (ii) uniformly allocating the number of trainable parameters to layers, (ii) tuning all the groups, and (iv) tuning different groups with proper strategies. Our discovered design patterns result in new parameter-efficient fine-tuning methods. Experiments show that these methods consistently outperform investigated parameter-efficient fine-tuning strategies across different backbone models and different tasks in natural language processing.

Published as a conference paper at ICLR 2023
PARAMETER-EFFICIENT FINE-TUNING GINGII DESIGN SPACES
Jiaao Chen'; Aston Zhang', Xingjian Shit, Mu Li,, Alex Smola', Diyi Yang Georgia Institute of Technology, *Amazon Web Services, Stannford University
ABSTRACT
The aim of parameter-efficien fine-tuning is to achieve performance that is com- parable to fine-tuning, but with fewer trainable parameters. Several hand-crafted strategies, such as Adapters, Prefix Tuning, BitFit, and LoRA, have been pro- posed, but it remains unclear whether there underlying design patterns. Thus, we present a parameter-efficieen design paradigm and identify design patterns that are applicable to various experimental settings. Instead of developing another individual tuning strategy, we introduce design spaces that parameterize tuning structures and strategies. These design spaces consist of four components: layer grouping, trainable parameter allocation, tunable groups, and strategy assignment. Our experiments reveal the following design patterns: (i) group layers in spin- dle pattern, (ii) allocate trainable parameters evenly among layers, (iii) tune all groups, and (iv) assign appropriate tuning strategies to each group. These patterns lead to new methods for parameter-efficient fine-tuning, which we show experi- mentally outperform existing strategies across various backbone models and NI.P tasky]]
INTRODUCTION
Large pre- vre-traine models have shown to achieve state-of-the. results in many downstream natural language processing tasks, by fine-tuning on task-specific labeled data (Devlin et al.) [019 [iu al.,
2019 Yang et al. 2019 Joshi et al. 2019, Sun et al., 2019 Clark et al. 2019: Lewis et al. 2020aa Bao et al. 2020 He et al. 2020, Raffel al. 2020] /iems et al.) 2022). However, the cost of fine- tuning all parameters and storing them separately for each task is high in terms of computational and storage resources, e.g., 355 million parameters for RoBERTa (Liu et al., 2019) and 175 billion parameters for GPT-3 (Brown et al. 2020). This makes it challenging to deploy in real-world natural language processing (NLP) systems that handle multiple tasks.
To make pretrained models more efficient for specific downstream tasks, various strategies have been proposed that only learn a small number of extra parameters while keeping the rest frozen (Houlsby et al., 2019b Pfeiffer et al., 2022 Li & Liang] 2021 Brown et al., 2020 Lester et al., Schiik & Schütze 2021 Ziems et al. 2022). One such strategy is adapter tuning (Houlsby al._[001.b.. which adds small neural modules (adapters) each layer of the pretrained network. and only trains the adapters during fine-tuning, Other methods, such as prefix tuning (Li & Liang 2021) and prompt tuning (Lester et al. 2021a). have been inspired by the success of controlling pretrained models through textual prompts (Brown et al. 2020). These methods prepend tunable tokens to the input or hidden layers, and only train these tokens during fine-tuning. BitFit (Zaken et al. 2021) updates the bias terms of pretrained models while freezing the rest, while LoRA (Hu et al. 2021) decomposes attention weight gradients into low-rank matrices to reduce the number of trainable parameters. He et al. (2022) proposed a unified view of these strategies, illustrating their differences and connections, but like its predecessors, the method is still equally applied to different layers of the pretrained network.
Most current fine-tuning strategies to adapt pretrained models to specific tasks are effective, but they often developed through manual design processes without considering potential design patterns
Work done during an internship at Amazon Web Services.
Correspondence to Jiaao Chen
jiaaochen @ gatech.edu>> and Aston Zhang <astonz @ amatoonommmmmoo.ommmm
We will relcase our code at attp:p////gtthib...cm/mmooooo http://awwwnncccceeeeeeeeee--eee ence/paft- gn- spaces
ctt

---

Published as a conference paper at ICLR 2023
Layer Grouping
Trainable Parameter Allocation
Tunable Groups
Strategy Assignment
Figure 1: The design space is characterized by: (i) Grouping of consecutive layers, (ii) The allocation of the number of trainable parameters to each layer, (iii) The selection of groups that will be fine- tuned, and (iv) The assignment appropriate strategies, such as Adapter (A), Prefix (P), BitFit (B), or LoRA (L), to each group.
across these strategies, different backbone models, and downstream tasks. The effectiveness of dif- ferent strategies is also unelear as they are usually applied separately, and it's unknown how they reinforce or complement each other (Mao et al. (022). Our aim is to gain a comprehensive under- standing of the fine-tuning design and uncover interpretable and widely applicable design patterns.
Instead of creating yet another strategy to be applied uniformly to various pretrained layers, we present parameter-effiiint fine-tuning design spaces that allow customization of both tuning struc- tures and strategies. These design spaces are comprised of four main components, as illustrated in layer grouping, trainable parameter ullocation, tunable groups, and strategy assignment.
Figure [
We start our journey towards parameter-efficien fine-tuning design using a relatively unconstrained design space. then narrow this space through successive rounds comparison, using random sampling and while enforcing constraints such equal layer grouping. Through this process, discover several key design patterns, including layer grouping a spindle pattern, uniform allo- cation of trainable parameters, tuning all groups, and appropriate strategy assignments. Our new methods outperform existing parameter-efficient fine-tuning strategies. We demonstrate the effec- tiveness of our approach using T5 (Raffel al., (2020) and classification tasks, but find that the discovered design patterns are applicable to other backbones (such as RoBERTa (Liu et al. 2019). BART (Lewis al., (020b) and XLNet (Yang et al., 2019)), and NLP tasks (e.g., summarization, machine translation, and eight SuperGLUE datasets).
Our contributions are: (i) The introduction of parameter-efficient fine-tuning design spaces. (ii) The discovery of several design patterss in parameter-efficiee fine-tuning through comprehensive experiments. (iii) The creation of parameter-efficient fine-tuning methods based on the discovered design patterns, which outperform existing strategies on various backbone models and NLP tasks.
RELATED WORK
Our work is closely related to and builds on work about network design spaces and parameter- efficient fine-tuning. We discuss the connections and differences below.
Network Design Spaces. Many works designed neural network models via ad-hoc discovery of new design choices that improve performance (Radosavovic et al.] 2019). such as the use of deeper architectures residual connections. Recent work (Radosavovic et al. 2020; You et al.) 2020 Radosavovic et al. 2019) focuses on the design space to discover new design principles for convolutional neural networks (Radosavovic et al.] 2020) and graph neural networks (You et al., 2020 Inspired by this work we focus on the design spaces to rethink parameter-effion fine- tuning, with the goal of discovering design patterns that are applicable to different settings.
Parameter-Efiicn Efficient Fine- Sine-Tuning for NLP. As pretrained models increase in size, storing and fine- tuning them becomes increasingly expensive and unfeasible for those without ample computational
2

---

Published as a conference paper at ICLR 2023
resources. A growing body of research is aimed at finding alternatives to fine-tuning large-scale models that reduce memory and storage costs. Some researchers have proposed using bottleneck layers with Sxip-connections to adapt large models, seen in works such as Houlsby et al. (2019a),
(Stickland & Murray] 2019). (Preiffer et al. 2020), and (Rebuffi et al. 2017). Other works focus on identifying and training only a subset of all model parameters, such as (Zhao et al.; (020) and
(Guo et al.] 20200). More recent research explores low-rank decomposition (Zhang et al. 2021) and the injection of trainable low-rank decomposition matrices into each layer (Hu
2021
Karimi Mahabadi et al.) 2021).
Li & Liang (2021) introduced prefix-tuning, where set of prefixes is added to autoregressive lan- guage models or both encoders and decoders, while [Lester et al. (2021b) proposed adding virtual tokens to the embedding layer. Another approach, side-tuning, was introduced in (Sung et al., 2022). He et al. (2022) and Ding et al. (2022). They proposed unified view of existing parameter-efficien fine-tuning strategies. yet another approach, Mao et al. (2022) introduced a unified framework to combine various methods through mixture-of-experts.
Our research focuses on the general design spuces of parameter-efficeet fine-tuning, providing a more comprehensive view of this method. By experimenting and refining design spaces, we aim to discover design patterns for parameter-efficee fine-tuning.
3 COMPONENTS OF DESIGN SPACES
Our goal is not list all possible design choices, but to show how design spaces can guide parameter-effcient fine-tuning research. As such, we pick a representative subset for each of the fol- lowing four components: (i) layer grouping. (ii) trainable parameter allocation, (iii) tunable groups, and (iv) strategy assignment. provides an example.
Given these choices, we sample from a distribution over them, then pick a subset that performs the best, narrowing down the set of choices. Given that more restrictive set, we repeat the procedure by sampling and picking a now even more restrictive subset until we arrive at a concise description of the design space. Quite understandably, this is very costly when dealing with Large Language Models. Taking a leaf out of (Radosavovic et al. 2020) we perform our experiments using a suffi- ciently cheap model, in this case T5-base and T5-3b, unless stated otherwise. Further details will be discussed in the next section. For now let's review the set of choices available.
Layer Grouping. Different layers in pre-trained models capture varying information and behave differently. For example, the authors of (Jawahar et al., 2019) found that the 3, 4, 5, 6, 7, 9, 12th layers have the most representation power in BERT and each layer captures different type of information, ranging from surface to syntactic to semantic level representation of text. For instance, the 9th layer performs well in semantic tasks such as checking random swaps of coordinated clauses, while the 3rd layer is best suited for surface tasks like predicting sentence length.
When adapting these pre-trained models for downstream tasks, it's crucial to group layers with simi- lar behaviors together. This is critical to the design and proper implementation of parameter-e efficient fine-tuning strategies. In this design component, we study patterns of how to group consecutive lay- ers in pre-trained models (e.g., transformer layers in T5) during the fine-tuning process.
Trainable Parameter Allocation. In parameter-efffiieet
fine-tuning, the total number of trainable parameters is usually set to a small portion of the total parameters in the pretrained model. Our study will explore different ways to allocate the predefined number of trainable parameters to the layers.
Tunable Groups. Not all the parameters of pretrained model need to be updated during fine-tuning for downstream tasks. For example, BitFit (Zaken et al. 2021) only updates the bias parameters while freezing the rest. As result, we explore which groups of parameters need to be learned during parfieter-efffcieent fine-tuning to achieve better performance.
Strategy Assignment. In order to improve the parameter efficiency, different sets of strategies (Li & Liang 2021 Lester et al. 2021b] Houlsby et al. 2019b; Hu et al., (021) were proposed, where only tuned and the remaining parameters in these pretrained models are frozen to adapt their general knowledge to specific down-stream tasks. We hypothesize that different groups might benefit from different proper strategies (or combinations) for capturing
small number of (extra) parameters
3

---

Published as a conference paper at ICLR 2023
different types of information. More formally, given set of individual strategies A for assignment, for any group Gú, assign a subset 24 C A to each layer in Gy,
4 DISCOVERING DESIGN PATTERNS
Each design space, denoted as S,, consists of a set of models (S,-models) that satisfy the con- straints characterizing the space with respect to layer grouping, trainable parameter allocation, tun- able groups, and strategy assignment. To discover design patterns, we start from a relatively uncon- strained parameter-efficien fine-tuning design space Su- We progressively refine it via S4 by comparing the overall quality of models in design spaces enforced with different constraints (e.g., each group has the same number of layers). To quantify the overall quality of models in any design space S; with a low-compute, low-epoch regime (Radosavovic et al. 2020), we randomly sample 100 models from S,, fine-tune with only 3 epochs [] and compute the average of the GLUE average performance. Using such low number of epochs is sufficient to obtain sufficiently representative score draw consistent conclusions (see Table in the Appendix) that extend to a full training run.
We emphasize that our goal is to demonstrate how the perspective of design spaces can help inform parameter-efficien fine-tuning research, rather than to find out the "best" design space or method. For computational efficiency, it is beyond the scope of this work to enumerate all possible constraints with respect to the design space components (Section 3). For efficiency, we use b5-base (pretrained backbone model) as it's s's both representative and also sufficiently small to make experimentation with many options computationally affordable.
In this work, we follow the discovery sequence of "grouping patterns - trainable parameter allocation tunable groups - strategy assignment": (I) To explore and understand the design patterns in all the layers in large pre-trained models in scale, it is necessary and more efficient to study the layers in the unit of groups So we start with the grouping patterns. (2) Once figuring out the optimal grouping patterns, it is then important explore how to allocate the trainable parameters these different groups in order to study more subtle designs with fair comparisons (e.g.. this would allow comparing different patterns of strategy assignments without the impact from different trainable parameters.). (3) Next, it becomes influential to examine which groups need to be learned during fine-tuning before we dig into the strategy assignment patterns. Because it is only meaningful to study assigning strategies to different groups after we figure out which groups need to be learned. (4) Finally, we study the tuning strategy assignment, which the most subtle design.
4.1 S. - THE INITIAL DESIGN SPACE
The initial relatively unconstrained design space So consists of all models without constraints on the design space components. Individual parameter-efficient fine-tuning strategies consist of Adapter, Prefix, BitFit, and LoRA. Specifically, without grouping constraints, each layer of the pretrained layer has probability of 0.5 to be tuned. If tuned, a random strategy, or combinations thereof, with random amount of trainable parameters are assigned to that layer.
Before comparing more subtle design patterns such as to which tuning strategy among Adapter, Prefix, BitFit, and LoRA to pick, we begin by exploring how to group layers and how allocate the total number of trainable parameters to layers.
4.2 S - APPLYING GROUPING CONSTRAINTS
Transformers are quite deep by now. This makes it impractical to pick a different tuning strategy for each layer. As such, the first question to ask is how to assemble the layers into groups that will be tuned using the same strategy. Inspired by RRadosavovic et al. (202)). we consider 4 groups, I Denote by N; the number of layers in G. As illustrated in Figure we compare the following layer grouping patterns:
G....., G4. in the order of forward pass, in the experiments
Increasing (Nz+1 N.): the number of layers in groups gradually increases;
We set the low epach by ohserving whether it enough for models to obtain stable performances to draw consistent conclusions (See Table 7 in the Appendix).
'The experimental results with 8 groups shown in the Table 16 in the Appendix.
4

---

Published as a conference paper at ICLR 2023
Table 1: Average performance (low-compute, low-epoch regime:
100 random models, 3 tuning
epochs) on the GLUE datasets using the T5-base pretrained backbone. We compare adding different layer grouping constraints to the So design space.
Layer Grouping
SST-2
MNLI
QNLI
QQP
RTE
STS-B
MRPC
CoLA
Avg
Sm-models
76.9
70.1
72.5
73.3
63.6
71.7
73.8
24.3
65.7
Increasing
85.3
74.9
77.2
77.5
66,8
76.2
76.0
33.0
70.8
Uniform
84.8
73.7
78.1
78.6
68.5
77.8
79.2
36.1
72.1
Decreasing 81.9
72.1
67.3
75.9
78.6
70.0
Spindle 86.9
75.5
79.8
79.4
69.8
78.3
37.3
73.3
Bottleneck 84.5
74.6
76.9
78.1
69.2
76.2
78.6
32.1
71.3
Table 2: Average performance (low-compute, low-epoch regime: 100 random models, 3 tuning
epochs) on the GLUE datasets using the T5-base pretrained backbone model. We compare adding different parameter allocation constraints to the S₁ design space.
Param Allocation
SST-2
MNL.
QNLI
QQP
RTE
STS-B
MRPC
CoLA
Avg
Increasing 87.2
77.9
79.4
78.7
71.6
77.6
81.4
32.0
73.2
Uniform
87.8
77.4
80.1
80.5
73.9
78.1
80.4
34.3
74.0
Decreasing
86.4
75.8
78.4
77.0
70.4
77.1
78.7
35.8
72.4
Uniform (N++1 N2): the number layers in groups is the same;
Decreasing (N;+1 < N2): the number of layers in groups gradually decreases;
Spindle (N1 N2 N3 > NN): the numbers of layers in groups at both ends are smaller;
Bottleneck (Ni > N2 = Ng < N:): the numbers of layers in groups at both ends are bigger.
Increasing
Uniform
Decreasing
Spindle
Bottleneck
Figure 2: Layer grouping patterns: group ID (GI,...Ga) vs. number of layers per group.
These layer grouping patterns lead to 5 possible design choices. They consist of all models in the So design space that satisfy one of these grouping pattern constraints. To compare the overall model qualities of different design spaces, we (i) randomly sample 100 models from the Sy design space that satisfy each grouping pattern constraint (Figure (ii) fine-tune with 3 epochs; and (iii) compute average performance for each design space. We will follow this procedure as we progressively add new constraints later.
The average performance is shown Table
find that models from the design space with the spindle grouping pattern (Figure 2 consistently outperform those from the other design spaces across all the GLUE tasks. In other words, we find that fine-tuning works better we treat a small number of layers close to the input and close the output special, and furthermore, if we divide up the bulk of the network into two blocks, each with their own design choices.
Applying the spindle grouping partitioning So yields the design space Sj.
4.3
S² - VARYING THE NUMBER OF TRAINABLE PARAMETERS PER LAYER
Now that we know how group the layers need establish how allocate the parameters n, within the layers of each group. particular, we consider the following options:
Increasing (R4+1 2 nz)): number of trainable parameters per layer increases or remains the same.
*The training time for the step is shown in the Table 18 the Appendix.

---

Published as a conference paper at ICLR 2023
Table 3: Average performance (low-compute, low-epoch regime: 100 random models, 3 tuning
epochs) on the GLUE datasets using the T5-base pretrained backbone model. We compare adding different tunable group constraints the S2 design space.
Tunable Groups
SST-2
MNLI
QNLI
QQP
RTE
STS-B
MRPC
CoLA
Avg
Gi
82.6
72.1
77.6
70.6
65.3
71.9
77.6
27.6
68.2
G2
83.3
72.8
77.5
72.8
63.6
72.8
77.5
27.5
68.4
G3
83.6
73.3
78.2
73.3
66.4
71.3
77.9
22.9
68.4
G4
83.2
73.0
77.9
73.7
63.9
72.0
77.9
27.9
68.7
G1, G2
83.5
73.2
78.0
75.4
67.7
73.2
78.0
28.0
69.6
G3. G4
87.8
74.6
78.3
76.9
68.6
74.3
78.3
28.3
70.7
G1, Cz. G3
86.0
79.0
77.8
71.8
79.0
33.0
72.6
G2, Ga. G4
85.2
76.6
79.1
78.6
70.1
77.6
79.1
31.9
72.2
Gı, G2, G3, G4
88.3
77.4
82.1
81.5
74.9
79.4
81.4
34.3
74.9
Uniform (n4+1 n2;): number of trainable parameters in every layer is constant;
Decreasing (f4+1 S ny):: number of trainable parameters per layer decreases or remains the same.
reports the average performance of these 3 design spaces. The uniform allocation design pattern obtains the highest GLUE average performance, making this relatively simple, interpretable design pattern favorable.
As above, we obtain 100 models for each of these new design spaces. Table 2
Allocating the number trainable parameters layers uniformly yields the new design space S2.
4.4 S3 - SELECTING THE GROUPS
Given that we established how to partition layers into groups, and how to allocate parameters per group. the next step is to assess whether all groups actually need tuning. Rather than exploring the 21
combinatorial choices we limit ourselves to the 4(4 + 1)/2 exception of (G2, G3), since focusing interior groups only does not yield good results (this is consistent with our findings in Table
options with the
Based on the GLUE average performance, we find that all the groups need to be tuned obtain the best results. This suggests that all the groups of pretrained layers have captured useful information that should be adapted to the downstream tasks.
Tuning all the groups yields the new design space Sz.
4.5 S4 - SELECTING STRATEGIES PER GROUP
So far the structure we' ve been exploring is fairly trivial: S₁ amounts to a uniform distribution of parameters over the layers of the groups and tuning all groups. This belies the fact that we still have significant freedom of design in picking specific fine-tuning approaches. Specifically, each design space consists of models that assign subset of {Adapter (A), Prefix (P), BitFit (B), and LoRA (L)) to the layers of each group G, for E {1 [1...4) 4). This is quite large space of options. To make some headway, we determine the ideal choice progressively by first reviewing strategies for G1, then G2 to G4. Due to space constraints the details of this procedure are relegated to the appendix(G] in Table g G2 Table G3 in Table[O, and G4 in Table
arrive at the following
strategy assignment for the T5-base pretrained backbone:
G (A, L) Gz: (A, P) Gz: P. Gg:(P.
For example, Adapter is more recommended in groups closer to input, while BitFit is more recom- mended groups closer to the output. The resulting design space will be referred to as S4.
4.6 VERIFICATION OF THE DESIGN CHOICES ON T5-3B
So far our results have led to a competent fine-tuning strategy for T5- base. To assess whether we actually discovered some useful strategies that have validity beyond T5-base, need to apply it to

---

Published as a conference paper at ICLR 2023
other models, too. For convenience pick T5-3b. As before, the detailed results relegated to
the appendix (Tables I2 and I5p. We observe that the following design patterns still apply:
1. grouping layers in a spindle pattern (Table 12)
2. uniformly allocating the number of trainable parameters to layers (Table 13
3. tuning all the groups (Table 14)
4. tuning different groups with proper strategies (Table 15
Note that for T5-3b (with final design space S4-3b), the discovered proper strategy assignment is slightly different
G1: (P, L) - G2: (A, -) G3: G (..
4.7 EXPERIMENTAL SETUP
Datasets. Our process based on the average performance on the widely-used GLUE bench- mark (Wang et al.) 2018). It covers wide range of natural language understanding tasks. First, single-sentence tasks include (i) Stanford Sentiment Treebank (SST-2) and (ii) Corpus of Linguistic Acceptability (CoLA). Second, similarity and paraphrase tasks include (i) Quora Question Pairs (QQP), (ii) Semantic Textual Similarity Benchmark (STS-B), and (iii) Microsoft Research Para- phrase Corpus (MRPC). Third, inference tasks include (i) Multi-Genre Natural Language Inference (MNLI), (ii) Question Natural Language Inference (QNLI), and (iii) Recognizing Textual Entail- ment (RTE). To compare performance, the Matthews correlation is measured for CoLA; the Spear- man correlation is used for STS-B, and accuracy is measured for the remaining GLUE tasks.
Prefrained Backbone Models and Model Settings We use T5-base/3b (Raffel al. 2020) as the main pretrained backbone models for discovering design patterns via our parameter-effciint fine- tuning design spaces. We use HuggingFace Transformers for our implementations and follow the default settings. During the exploration, we fix the total number of trainable parameters (in the percentage that the backbone model) by following He al. (0022).
By limiting ourselves to rather concise parameter space and small number of parameters within that parameter space that we allow fine-tuned we ensure that exploration remains computa- tionally feasible. Obviously, this exploration would be pointless the discovered insights were not portable. Hence, we need to evaluate how well the strategies perform on new models and new architectures.
EVALUATION
The S₁ model (Section 4.5 and Sy-3b model (Section 4.6) adopt the design patterns discovered from TS-base and T5-3b. respectively. We will evaluate their effectiveness when applied different pretrained backbones and different NLP tasks.
5.1 EXPERIMENTAL SETUP
Dataset. Besides the GLUE datasets (Wang et 2018) (Section 4.7 we evaluate our methods on
two generation tasks used by He et al. ): Abstractive Summarization using XSum (Narayan
al. 2018). and Machine Translation using the WMT 2016 en-ro dataset (Bojar al. 2016). We
report ROUGE scores Lin 2004) on the XSum test set, and BLEU scores (Papineni et al. 2002 on the en-ro test set.
Models and Model Settings. We mainly compare our methods with the following baselines: (i) Full Fine-tuning (full): it fine-tunes all the model parameters in the pretrained models; (ii) Adapter
(Houlsby et al.] (211bb): it adds adapter modules to each transformer layer; (iii) Prefix (Li & Liang,
2021]: it optimizes set of small continuous vectors prepended to transformer layers; (iv) BitFit
(Zaken et al. 2021): it only updates the bias terms in pretrained models; (v) LoRA (Hu et al.)
2021): it decomposes the attention weight into low-rank matrices to reduce the number of trainable parameters. Besides T5 (Raffel et al.) 2020), we additionally apply methods to other backbone models including RoBERTa-buse/large (Liu et al. 2019) and BART-base base//arge (Lewis et al., (020a)).
We use the default settings. We set the total number of trainable parameters (in the percentage of
7

---

Published as a conference paper at ICLR 2023
Table 4: Performances of different tuning methods the GLUE datasets using the T5-hase (upper part) and T5-3b (lower part) pretrained backbone models, respectively. The results are averaged over 20 random runs (with standard deviations subscripts). The S4-model and the S4-3b-model perform significantly better than the second-best PEFT methods in all the eight datasets at the sig- nificance level p 0.05(*) or even (*01
Method
SSI-2
QQP
STS-B
MRPC
CoLA
Average
89.4
Adapter
94.6
86.7
75.3
86.7
89.1
59.2
83.3
76.6
BitFit
94,4
84,5
90.6
88,3
74.3
86.6
90.1
57.7
83.3
LoRA
94.8
84.7
91.6
88.5
75.8
86.3
88.7
51.5
82.7
Sy-model
95.51.
87.61.0
92.71.
88.8
80.42.3
87.42.0
91. 91.22.
62.23.2
85.7
full
97.4
91.4
96.3
89.7
91.1
90.6
92.5
67.1
89.5
Adapter
87.8
83.4
90
89,7
65.2
87.1
Prelix
96.3
82.8
88.9
85.5
78.3
83.5
85.4
42.7
80.4
Bitfit
90.7
LoRA
96.2
90.6
94.9
89.1
91.2
91.1
91.1
67.4
88.9
S.-3bmmodel
97.21.
91.67.2
96.67.
89.5
91.55..
91.55.a
91.9.
69.73.4
89.9
Table 5: Performances of different tuning methods on GLUE datasets using the RoBERTa-bas (upper part) and RoBERTa-large (lower part) pretrained backbone models. The results are averaged over 20 random runs (with standard deviations as subseripts). Here we also include two baselines: (i) Sn-model, where all the designs are randomly selected for RoBERTa as in the So design space; (ii) Sz-model, where strategies are randomly assigned different RoBERTa layer groups as in
Sy design space. The Sy-model and Sp-be-mode perform significantly better than the second-best
PEFT methods in all the eight datasets at the significance level < 0.05(*) or even p < 0.01( (**).
Method
SST-2
MNLI
QNLI
QQP
KTE
STS-B
MRPC
CoLA
Average
full
92.8
91.9
80.8
90.3
90.2
63.6
86.5
Adapter
94.2
87.1
93.1
90.2
71.5
89.7
88.5
60.8
84.4
Prefix
86.8
74.5
88.2
61.5
84.6
Bittit
93.7
84.8
91.3
84.5
71.8
90.8
90.0
61.8
84.1
90.0
86.4
So-model
94.2
95.3
90.4
90.6
75.6
89.6
88.0
60.9
85.6
S. model
94.3
87.2
92.8
91.0
81.8
90.3
89.2
63.2
86.2
Sa-model
94.81.6
87.85.'
93.4
91.61.2
85.87.s
90.42
90.0
63.2
87.1
full
96.4
90.2
94.7
92.2
86.6
92.4
90.9
68.0
88.9
Adapter
Prefix
95.7
87.6
92.1
88.7
82.3
89.6
87.4
62.8
85.7
BitFit
90.2
86.2
90.9
92.7
64.2
87.7
LoRA
96.2
90.6
94.7
91.6
87.4
92.0
89.7
68.2
88.8
So So-model
95.5
86.5
92.3
89.8
84.6
89.2
86.3
61.2
85.6
Sa-model
96.3
89.4
93.8
90.2
85.9
90.8
90.9
63.44
87.6
S. Sb--mlll
96.67%
90.87.
95.15% .
92.07%
87.22.8
92.3%.
91,87*
68. 68.45,2
89.3
that in the backbone model) by following He et al. (2022) Specifically, this value is set to 0.5% for Adapter, Prefix, LoRA, and our methods, and 0.1% for BitFit.
For all the experiments, we followed Liu et al. (019) to set the linear decay scheduler with warmup ratio of 0.06 for training. The batch size was 128 for base models and 64 for large models. The maximum learning rate was 5e 5 and the maximum number of training epochs was set to be either or 10, All the experiments were performed using A100 GPUs.
5.2 EFFECTIVENESS ON DIFFERENT BACKBONES
GLUE with T5 Backbone. With our discovered design patterns, fine-tune T5-base mooel) and T5-3b (S.-3b-model on GLUE and compare them with all the baseline methods. The results shown in Table where the key measure is the GLUE average performance (last column). We
8

---

Published as a conference paper at ICLR 2023
Table 6: Performance of different tuning methods on generation tasks (XSUM and en-ro) using the BART-base (left) and BART-large (right) pretrained backbone models.
BART-base XSUM(R-1/2/L)
en-ro (BLEU)
BART-arrge
XSUM(R-1/2/L)
en-ro (BLEU)
full
40.5/19.2/34.88
34.5
full
45.1/22.3/37.2
37,9
Adapter
37.7/17.9/33.1
33.3
Adapter
43.8/20.8/35.7
35.3
Prefix
38.2/18.4/32.4
33.8
Prefix
43.4/20.4/35.5
35.6
BitFit
37.2/17.5/31.4
33.2
BitFit
42.8/18.7/33.2
35.2
LoRA
38.9/18.6/33.5
33.6
LoRA
42.9/19.4/34.8
35,8
PA
39.3/18.7/33.8
33.8
PA
43.9/20.6/35.6
36.4
S4-model
40.2/19.3/34.2
34.1
So-3b-model
44.3/21.7/36.8
37.2
find that our S4-model and S4-3b-model consistently outperform the investigated methods in the key measure. By tuning only 0.5% parameters, our methods even outperform the full fine-tuning baseline where all the parameters are tuned, indicating the effectiveness of our discovered parameter- efficient fine-tuning design patterns.
GLUE with RoBERTa Backbone. We directly apply the S4-model and S4-3b-model (adopting design patterns discovered using T5-base and T5-3b) to fine-tune the RoBERTa-base and RoBERTa- large pretrained backbone models, respectively. We keep all the other settings the same and evaluate them GLUE datasets. We also compare with variant methods randomly sampled from two design spaces: (i) Sn-model, where all the designs are randomly selected for RoBERTa as in So; (ii) Sq- model, where strategies
randomly assigned different RoBERTa layer groups in S3- Table 5 shows that (i) the design patterns (adopted by Sa-model and S4-3b-model) discovered using T5 models are applicable to the RoBERTa backbone models and outperform the investigated methods in GLUE average performance with no extra discovery proces (ii) improved performance from Sy-models, Sy Sy-models, S1-(3b) process (Section 4).
S-oeels models support adding more constraints in the pattern discovery
SuperGLUE with XLNet Backbone. We also directly use the Si-model and S,-3b-model (adopt- ing design patterns discovered using T5-base and T5-3b) to fine-tune the XI.Netbasse and XLNet- large pretrained backbone models without any extra discovery process. We keep all the other settings the same and evaluate them SuperGLUE datasets. Table 17 (In the Appendix) reiterates the fact that our PEFT design patterns discovered from T5 models are generelizable to the XLNet backbone models and outperform the investigated methods in other tasks (SuperGLUE) with no additional discovery process.
Generation Tasks with BART Backbone. We further apply the Są-model and S4-3b-model (adopt- ing design patterns discovered using T5-base and T5-3b) to fine-tune the BART-base and BART- large pretrained backbone models, respectively. We evaluate the models two generation tasks:
summarization (XSUM) and machine translation (en-ro) following He et al. (2022). We also com- pare with PA (parallel adapter) using the same number of trainable parameters (He al. (0022). Table 6 shows that our methods, although adopting design patterns discovered from classification tasks using T5, still outperform investigated parameter-efficcies fine-tuning strategies on generation tasks with different BART backbones.
CONCLUSION
Parameter-efficient fine-tuning adapts knowledge in pretrained models down-stream tasks in more parameter-efficient fashion. Instead of focusing on designing another strategy the first place, introduced parameter-efficient fine-tuning design spaces. We empirically discovered several de- sign patterns in parameter-efficient fine-tuning. These design patterns led to new parameter-efficine fine-tuning methods. Experiments showed that these methods consistently outperform investigated parameter-efficient fine-tuning strategies across different backbone models and different tasks in natural language processing.
Future works might repeat the discovery process using RoBERTa improve performance for this back- hone.

---

Published as a conference paper at ICLR 2023
REFERENCES
Hangbo Bao, Li Dong, Furu Wei, Wenhui Wang, Nan Yang, Xiaodong Liu, Yu Wang, Songhao Piao, Jianfeng Gao, Ming Zhou, al. Unilmv2: Pseudo-masked language models for unified language model pre-training, arXiv preprint arXiv:2002. 12804, 2020.
Ondłej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Aurélie Névéol, Mariana Neves, Martin Popel, Matt Post, Raphael Rubino. Carolina Scarton, Lu- cia Specia, Marco Turchi, Karin Verspoor, and Marcos Zampieri. Findings of the 2016 conference machine translation. In Proceedings of the First Conference Machine Translation: Volume 2. Shared Task Papers, pp. 131-198, Berlin, Germany, August 2016. Association for Compu- tational Linguistics. doi: 10. 10.18653/v1/W166 10WW6662300 2301. URL https:://claatholl.see org/ 116--2011
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020.
Kevin Clark, Minh-Thang Luong, Quoe V Le, and Christopher D Mamning. Electra: Pre-training text encoders as discriminators rather than generators. In International Conference on Learning Representations, 2019.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-IILT, 2019.
Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang. Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, Jing Yi, Weilin Zhao, Xiaozhi Wang. Zhiyuan Liu, Hai- Tao Zheng, Jianfei Chen, Yang Liu, Jie Tang, Juanzi Li, and Maosong Sun. Delta tuning: A comprehensive study of parameter efficient methods for pre-trained language models, 2022. URL https://arxix: hwtg//////////
06904
Demi Guo, Alexander M Rush, and Yoon Kim. Parameter-efficien transfer learning with diff prun- ing. arXiv preprint arXiv:20/2.07463. 2020.
Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kinkpatrick, and Graham Neubig. Towards unified view of parameter-efffcient transfer learning. In International Conference on Learning Representations, 2022.
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Decoding eenhinee- bert with disentangled attention. arXiv preprint arXiv:2006. 03654, 2020.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, An- drea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parfmeter-efficient transfer learning for nlp. In International Conference on Machine Learning, pp. 2790-2799. PMLR, 2019a.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, An- drea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for NLP. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th In- ternational Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research. pp. 2790-2799. PMLR, 09-15 Jun 2019b. URL http::/ press/v99 Ptess/v97/hou hww.hhhuusbyyyyly......oo.yyyyyl 9a html
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021. URL https:
//arxiva wwtp///gggabaaioee
09685
Ganesh Jawahar, Benoît Sagot, and Djamé Seddah. What does BERT learn about the structure of language? In Proceedings of the 57th Annual Meeting of the Association for Computational Lin- guistics, pp. 3651-3657, Florence, Italy, July 2019. Association for Computational Linguistics.
10

---

Published as a conference paper at ICLR 2023
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. Span- bert: Improving pre-training by representing and predicting spans. Transactions of the Association for Computational Linguistics, 8:64-77, 2019.
Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder. Compacter: Efficient low-rank hypercomplex adapter layers. Advances in Neural Information Processing Systems, 34:1022- 1035, 2021.
Brian Lester, Rami AI-Rfou, and Noah Constant. The power of scale for parameter-efficien prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 3045-3059, Online and Punta Cana, Dominican Republic, November 2021a. a. Association for Computational Linguistics. doi: 10.1 1..18865/v1/222 22tp//200eemllmmai.2224.4. ltap:::/aaclathiie Montthooog..... .ow.olllyy............ org. 202g/2021... emn Mimlpp pa-llp-mme ma 243
URL
Brian Lester, Rami AI-Rfou, and Noah Constant. The power of scale for parameter-efftcient prompt tuning, 2021b.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising grquence-to-squeene pre-training for natural language generation, translation, and comprehension. SCL, 2020a.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising sequence-to-scquencc pre- training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. pp. 7871 1-7880, Online, July 2020b. Association for Computational Linguistics. doi: 10. 20.18653/v1/2020..o wwt.0000..lmmmii... madl.mai...33.. URL ptttps://vie a web. www.eb.rrgg/at.ooooggo/oo 2020 acl maacc-maiiie main. 703
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation, 2021.
Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out. pp. 74-81, Barcelona, Spain. July 2004. Association for Computational Linguis- tics. URL htLtttsss: hhttpss//aclaanthll.ooto
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint 1907.1190..11999 2019.
Yuning Mao, Lambert Mathias, Rui Hou. Amjad Almahairi, Hao Ma, Jiawei Han, Scott Yih, and Madian Khabsa. UniPELT: A unified framework for parameter-efffcient language model tun- ing. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pp. 6253-62644 Dublin, Ireland, May 2022. Association for Compu- tational Linguistics. doi: (0.18653/1/20022accllol
URI. htttps://aclanhholls
prg/2022.aar long. 433
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. Don't give me the details, just the sum- mary! topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing, pp. 1797-1807,
Brussels, Belgium, URL hRRL/tttpe://aaaaattooooooooo org/D18 1206 1/D188 1206.
2018. Association for Computational Linguistics. doi:
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In ACL, 2002.
Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. Adapter- fusion: Non-destructive task composition for transfer learning. arXiv preprint 2it::2005000247. 2020.
Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. Adapter-
Fusion: Non-destructive task composition for transfer learning. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main
Volume, pp. 487-503 Online, April 2021. Association for Computational Linguistics. URL
https:/ www. a web. wwwrr//attttteoo ogy eac 1-aamminn 399

---

Published as a conference paper at ICLR 2023
Ilija Radosavovic, Justin Johnson, Saining Xie, Yen Lo., and Piotr Dollár. On network design spaces for visual recognition, 2019.
Iija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. network design spaces, 2020. URL http:::/arxi...or///aaaaaa
Designing
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer, 2020.
Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. Learning multiple visual domains with residual adapters. arXiv preprint arXiv:/705.08040 2017.
Timo Schick and Hinrich Schütze. Exploiting cloze-questions for few-shot text classification and natural language inference. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 255-269, Online, April 2021. Association for Computational Linguistics. doi: 10. 12211186611222.1..om main URL https:
//aclanthology.oo 200p://ooo......./200...... wttp:///222.eee.mmm.a...... eacl-mainn.m....
Asa Cooper Stickland and lain Murray. Bert and pals: Projected attention layers for efficient adap- tation in multi-task learning. In International Conference Machine Learning, pp. 5986-5995 PMLR, 2019.
Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Xuyi Chen, Han Zhang, Xin Tian, Danxiang Zhu, Hao Tian, and Hua Wu. Ernie: Enhanced representation through knowledge integration. arXiv preprint arXiv:1904.09223, 2019.
Yi-Lin Sung, Jaemin Cho. and Mohit Bansal. Lst: Ladder side-tuming for parameter and memory efficient transfer learning, 2022.
http//bbb/.2.../////////// 06522
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. Glue: A multi-task benehmark and analysis platform for natural language understanding. In BlackboxNLP@ EMNLP. 2018.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. In Advances in neural information processing systems, pp. 5754-5764, 2019.
Jiaxuan You, Rex Ying, and Jure Leskovec. Design space for graph neural networks, 2020. URL
https: attps://arxiy: wttp///////////o http:///w/ww....r///////00....00
Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. Bitfit: Simple parameter-efficient fine-
tuning for transformer-basee masked language-models 2021. URL httts:://axxx...rrtt
abs/2106.1013
Aston Zhang, Yi Tay, Shuai Zhang, Alvin Chan, Anh Tuan Luu, Siu Hui, and Jie Fu. Beyond fully- connected layers with quaternions: Parameterization of hypercomplex multiplications with 1/n parameters. In International Conference on Learning Representations, 2021.
Mengjie Zhao, Tao Lin, Fei Mi, Martin Jaggi, and Hinrich Schütze. Masking an efficient alterna- tive to finetuning for pretrained language models. arXiv preprint arXiv:2004. 12406, 2020.
Caleb Ziems, Jiaao Chen, Camille Harris, Jessica Anderson, and Diyi Yang. VALUE: Understanding dialect disparity in NLU. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguisties ( (Volume I: Long Papers), pp. 3701-3720, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. 1111663331/2222..02 acl-long, URL https: 2ow.aatthoooooy.or//2222222 org/2022..cccoo..oooooo.ooz acl acl-long.2555
//aclanthology:
12

---

Published as a conference paper at ICLR 2023
A
MORE EXPERIMENTAL RESULTS
Table 7: Average performances (low-compute, low-epoch regime: 100 random models, tuning epochs 2, 4, 20 for five different blocks) the GLUE datasets using the T5-base pretrained backbone model. We compare adding different grouping constraints to the Sy design space.
Grouping Patterns
SST-2
MNLI
QNLI
QQP
RTE
STS-B
MRPC
CoLA
Avg
epochs
Increasing
73.2
63.3
67.8
68.8
63.8
67.2
64.1
11.0
59.9
Uniform
72.8
63.4
63.4
65.8
12.1
59.2
Decreasing
72.4
63.2
65.1
69.8
59.3
62.7
63.6
18.7
59.4
Spindle
72.6
64.8
66.8
71.1
62.1
62.3
64.8
12.3
59,6
Bottleneck
72.2
63.7
65.3
68.3
61.2
63.2
66.6
12.1
59.0
2 epochs
Increasing
76.2
69.3
73.2
76.5
65.8
72.2
74.0
21.0
66.0
Uniform
74.8
70.9
74.1
75.6
66.5
73.4
71.2
22.1
66.1
Decreasing
71.4
70.1
72.1
76.8
64.3
71.7
73.6
18.7
64.8
Spindle
76.6
71.9
71.8
74.4
67.5
73.5
71.8
22.3
66.2
Bottleneck
74.2
71.1
69.6
73.3
73.3
73.6
65.5
3 epochs
Increasing
85.3
74.9
77.2
77.5
66.8
76.2
76.0
33.0
70.8
Uniform
84.8
73.7
78.1
78.6
68.5
77.8
79.2
36.1
72.1
Decreasing
81.9
72.1
78.3
76.7
67.3
75.9
78.6
28.7
69.9
Spindle
86.9
75.5
79.8
79.4
69.8
78.3
80.1
47.3
74.6
Bottleneck
84.5
74.6
76.9
78.1
69.2
76.2
78.6
32.1
71.3
epochs
Inercasing
88.3
78.5
80.2
80.5
70.8
80.2
80.0
37.0
74.4
Uniform
88.8
78.9
81.9
71.5
81.4
75.4
Decreasing
87.6
74.1
80.8
81.7
79.3
78.9
79.6
38.7
75.1
Spindle
89.6
79.8
83.6
82.8
71.8
81.3
82.1
39.3
76,3
Bottleneck
86.5
77.6
82.7
81.1
70.2
70.9
81.6
36.1
73.3
20 epochs
Inercasing
92.3
83.3
86.2
82.5
71.8
82.2
84.0
51.0
79.1
Uniform
92.8
83.9
86.1
83.6
72.5
83.8
84.2
52.1
79.9
Decreasing
91.4
82.1
85.1
83.1
69.3
81.7
83.6
48.7
78.1
Spindle
93.6
84.8
87.8
84.4
73.5
84.3
85.8
52.3
$0.8
Bottleneck
92.1
82.6
85.6
83.3
71.2
84.6
79.3
B GENERAL EFFECTIVENESS ON SUPERGLUE WITH XLNET BACKBONES
We also directly use the mo-model and S4-3b-modde mooee (adopting design patterns discovered using T5- base and T5-3b) fine-tune the XLNet-base and XLNet-large pretrained backbone models without any extra discovery process. We keep all the other settings the same and evaluate them on Su- perGLUE datasets. Table 17 reiterates the fact that our PEFT design patterns discovered from T5 models are generelizable to the XLNet backbone models and outperform the investigated methods other tasks (SuperGLUE) with additional discovery process.
13

---

Published as conference paper at ICLR 2023
Table 8: Average performances (low-compute, low-epoch regime: 100 random models, 3 tuning
epochs) on the GLUE datasets using the T5-base pretrained backbone model. We compare adding different G1 strategy assignment constraints to the Sy design space.
Strategy Assignment
SST-2
MNLI
QNLI
QQP
RTE
STS-B
MRPC
CoLA
Avg
-1-Appeter (A)
89.8
83.5
84.9
80.8
72.5
80.8
78.5
37.7
76.1
G1 Gr-Prex (P)
89.3
83.1
84.4
80.1
70.1
80.0
77.6
33.0
74.7
GitBitFit (B)
89.0
82.9
81.4
72.0
81.1
77.0
30.8
74.8
Gr-LoRA (L)
89.9
83.6
85.0
81.1
71.8
81.0
78.8
35.3
75.8
G1-(P, L)
89.1
82.8
85.1
81.2
71.9
81.5
79.1
35.0
75.7
G1-(A,
89.8
82.8
84.8
81.1
72.2
81.3
79.2
36.4
75.9
Gi-(A,
89.6
83.8
85.6
81.3
72.9
81.7
79.5
36.8
76.4
P,)
83.5
81.5
72.2
81.4
79.2
35.2
75.9
G1-(A, L,L)
89.3
83.6
85.5
81.6
72.3
81.0
78.8
35.7
76.0
89.2
83.3
84.8
81.8
72.5
81.1
78.6
35.6
75.8
Gr-(A, 89.8
83.4
84.8
81.1
72.6
81.6
79.4
34.8
75.9
Gr
90.0
83.1
85.3
81.6
72.6
81.4
79.2
36.5
76.1
Table 9: Average performances (low-compute, low-epoch regime: 100 random models, 3 tuning epochs) on the GLUE datasets using the T5-hase pretrained backbone model. We compare adding different G2 strategy assignment constraints with G1-(L, A) the S3 design space.
Strategy Assignment
SST-2
MNLI
QNLI
QQP
RTE
STS-B
MRPC
CoLA
Avg
G2 Gg-Adapter (A)
91.6
84.3
82.3
73.5
82.8
81.3
38.8
77.5
G2-Prefix (P)
89.6
84.0
86.5
81.5
73.3
82.5
80.5
36.2
76.7
Gz-BiuFi (B)
91.2
83.6
85.7
82.9
72.6
82.6
80.8
33.1
76.5
Gz-LoRA (L)
91.4
84.4
86.1
82.0
72.8
81.8
81.6
39.8
77.4
G2-(P, L)
84.6
73.8
82.8
82.0
38.5
77.7
G2-(A,
92.2
84.2
87.1
82.2
74.4
83.0
82.5
40.8
78.3
G2)(A,
92.0
84.4
86.5
81.8
73.6
82.6
82.2
40.1
77.9
G2
91.8
84.8
86.8
81.8
74.1
83.0
82.1
37.9
77.7
G2-(P, L,
91.6
84.1
87.1
82.0
74.0
82.9
82.4
35.8
77.4
G2-(A, P. B)
91.8
84.2
82.1
73.7
83.3
82.2
41.2
78.1
G2-(A, B,
92.2
84.3
86.1
82.0
74.1
83.2
82.0
37.6
77.6
G2-(A, L)
92.0
84.1
87.0
81.9
74.2
83.1
81.3
42.4
78.1
Table 10: Average performances (low-compute, low-epoch regime: 100 random models, 3 tuning
epochs) on the GLUE datasets using the T5-base pretrained backbone model. We compare adding different G3 strategy assignment constraints with G1-(L. A) G2-(P, A) to the S3 design space.
Strategy Assignment
SST-2
MNLI
QNLI
QQP
RTE
STS-B
MRPC
CoLA
Avg
Gg-Adapter (A)
92.5
85.3
87.5
83.3
73.9
84.0
83.8
44.9
79.4
Gy-Prefix (P)
84.7
82.6
74.2
83.8
82.9
40.5
78.4
Ga-BitFit (B)
91.9
84.3
87.0
82.0
73.6
84.1
83.3
36.1
77.8
Ga-LoRA (L)
92.8
85.4
87.8
83.5
74.7
82.4
84.0
44.0
79.3
Gg-(P, L)
93.0
85.2
88.3
83.8
75.2
84.4
84.2
37.9
79.0
Ga-(A,
92.4
85.6
88.1
83.6
75.0
84.2
84.0
41.8
79.3
G3-(A,
85.9
83.1
75.3
84.3
83.9
42.2
79.4
G3-(A,P,
92.6
86.0
87.5
83.4
75.6
84.6
83.5
43.9
79.6
Ga-(P. B.
92.7
85.8
87.2
83.7
75.2
84.5
83.8
40.8
79.2
Gy-(A, P,
93.3
85.8
88.6
84.0
75.5
84.9
84.1
42.1
79.8
Gs L)
86.5
88.0
83.2
75.8
84.2
84.2
39.7
79.4
Gy L)
93.3
85.6
87.7
83.8
75.2
84.3
84.4
41.6
79.4
14

---

Published as a conference paper at ICLR 2023
Table 11: Average performances (low-compute, low-epoch regime: 100 random models, 3 tuning epochs) on the GLUE datasets using the T5-base pretrained backbone model. We compare adding different G4 strategy assignment constraints with G1-(A, L) G2-(A, P) G3-(A, P, B) to the Sy design space.
Strategy Assignment
SST-2
MNLI
QNLI
QQP
RTE
STS-B
MRPC
CoLA
Avg
Gd-Aeer (A)
93.8
85,8
88.6
84.8
76.3
85,8
86.0
48.5
81.2
Gr-Prefix (P)
93.5
85.2
88.3
83.6
76.8
85.3
85.6
44.8
80.3
Gu-BitFit (B)
85.3
88.9
84.4
77.1
85.4
86.2
46.1
80.9
G--LoRA (L)
94.0
86.0
89.2
85.0
77.2
85.5
85.8
47.7
81.3
Ga-(P, L)
94.3
86.2
89.3
85.8
78.0
86.0
88.2
47.2
81.8
G4-(A,
94.1
86.2
89.6
85.4
77.9
86.2
86.9
45.3
81.4
G4-(A,
94.2
85.9
89.2
85.5
77.8
86.2
88.0
46.8
81.7
G,
85.8
85.7
77.4
86.5
87.9
44.8
C+-(P, L)
94.6
86.4
90.4
86.1
78.2
86.8
88.5
47.2
82.3
P,
94.5
86.0
89.6
86.0
78.0
86.2
88.1
44.8
81.6
Gr B.L)
94.3
86.4
89.2
85.6
78.2
86.4
88.3
46.6
81.9
GA.AP
86.2
85.9
78.5
86.1
88.0
45.3
81.6
Table 12: Average performances (low-compute, low-epoch regime: 100 random models, 3 tuning
epochs) on the GLUE datasets using the T5-3b pretrained backbone model. We compare adding different layer grouping constraints to the So design space.
Grouping Patterns
SST-2
MNLI
QNLI
QQP
RTE
STS-B
MRPC
CoLA
Avg
So-models
80.3
72.1
74.7
72.8
76.9
75.2
71.0
32.2
69.4
Increasing
84.4
75.7
83.0
78.3
82.7
80.3
76.3
42.1
75.3
Uniform
86.8
77.1
82.6
76.2
83.8
81.6
77.3
48.9
76,8
Decreasing
83.2
74.3
81.8
77.3
82.8
79.9
76.5
40.8
74.5
Spindle
88.6
78.8
83.7
77.7
84.2
80.9
78.3
44.6
77.1
Bottleneck
86.3
77.0
82.2
75.6
83.3
80.2
77.1
41.5
75.4
Table 13: Average performances (low-compute, low-epoch regime: 100 random models, tuning epochs) on the GLUE datasets using the T5-3b pretrained backbone model. We compare adding different layer parameter constraints to the S₁ design space.
Parameter Allocation
SST-2
MNLI
QNLI
QQP
RTE
STS-B
MRPC
CoLA
Avg
Increasing
90.3
79.3
84,9
79.3
85.2
82.8
79.2
50.1
78.9
Uniform
90.6
80.8
84.6
79.7
85.5
82,4
78.9
50.8
79.1
Decreasing
88.6
78.2
83.5
78.1
84.4
81.5
78.1
49.6
77.7
15

---

Published as a conference paper at ICLR 2023
Table 14: Average performances (low-compute, low-epoch regime: 100 random models, tuning epochs) on the GLUE datasets using the T5-3b pretrained backbone model. We compare adding different tuning groups constraints the S₂ design space.
Tunable Groups
SST-2
MNLI
QNLI
QQP
STS-B
MRPC
CoLA
Avg
G1
88.3
78.3
82.2
77.4
82.1
80.7
76.1
49.4
76.8
G2
89.1
78.8
82.1
77.2
82.3
81.2
76.4
49.6
77.1
G3
89.6
78.5
82.6
78.1
81.9
77.4
48.7
77.5
G4
89.8
79.3
82.7
77.9
83.5
81.9
77.9
48.5
77.1
G1, G2
90.1
80.2
83.4
78.5
84.3
82.4
78.5
51.1
78.5
G3, G4
90.5
80.6
83.8
78.7
84.2
83
78.2
50.3
78.6
G1, G2, Gs
90.6
84.9
79.3
82.9
79.3
79.0
G2, Ga. G4
90.8
80,9
84.6
79.1
85.1
83.1
79.1
49.2
78.9
G1. G2. Gg. G4
91.1
81.4
85.2
80.4
85.9
83.5
80.0
51.6
79.9
16

---

Published as a conference paper at ICLR 2023
Table 13: Average performances (low-compute, low-epoch regime: 100 random models, 3 tuning
epochs) the GLUE datasets using the T5-3b pretrained backbone model. We compare adding different strategy assignment constraints following the process in Section 4.5
Strategy Assignment
SST-2
MNLI
QNLI
QQP
RTE
STS-B
MRPC
CoLA
Avg
G--Apeeer (A)
91.1
81.4
86.1
80.5
86.7
83.3
80.1
50.8
80.0
Gi-Prefix (P)
90.8
81.1
85.5
80.2
86.2
83.1
79.8
50.2
79.6
Gi-BitFit (B)
81.3
85.1
79.6
85.8
82.8
79.6
49.5
79.2
G--LoRA (L)
91.4
81.9
86.2
80.8
86.4
83.9
80.8
49.6
80.0
Gr-(P, L)
91.8
82.9
86.8
81.3
87.1
84.2
81.6
52.3
81.0
G1-(A, P)
91.3
81.9
86.4
81.1
85.6
83.7
80.7
52.8
80.1
G1-(A,
82.3
81.5
85.8
84.9
81.5
51.8
80.6
P,)
81.7
81.2
86.4
84.2
80.9
52.3
Gr-P, L,
91.5
82.8
86.3
81.4
86.1
83.6
81.2
51.5
80.5
Cr,(A,P,B)
91.3
82.3
86.7
80.8
86.8
84.3
80.7
51.8
80.5
(A
91.7
82.5
86.2
81.3
86.3
84.6
81.3
51.7
80.7
(A.PP.
82.3
81.1
86.6
84.2
81.1
51.1
80.5
Ad-pter (A)
92.1
82.5
86.4
81.8
87.2
84.8
81.8
53.8
81.3
Gg-Prefix (P)
91.8
83.1
87.2
81.6
86.2
84.4
81.1
52.8
81.0
G2-BitFit (B)
91.2
82.1
81.1
86.3
84.6
80.3
53.1
Ge-LoRA (L)
92.6
82.9
87.5
81.3
87.4
85.1
81.9
52.2
81.4
G2-(P. L)
91.6
82.7
87.6
81.6
87.8
85.3
82.1
52.8
81.4
G2-(A.
92.1
83.3
87.5
81.9
87.4
85.5
81.8
53.1
81.5
G2-(A,
92.5
83.7
88.1
82.2
87.4
85.7
82.9
53.6
82.1
P,
92.3
83.4
81.6
87.1
85.3
81.4
53.2
81.4
G2-(P, L)
91.8
83.1
87.4
81.5
87.2
85.1
82.7
53.8
81.5
G2-(A,P,
91.5
82.6
87.8
81.3
86.5
85.2
82.1
54.2
81.4
02,(A.B,L)
92.6
83.5
87.2
82
87.3
86.5
82.5
52.8
81.8
Gr,AA,
83.2
81.6
87.5
85.5
82.4
51.2
81.5
Gg-Aderter (A)
92.6
84.1
88.3
81.8
87.8
85.4
82.8
55.2
82.2
Gr-Prefix (P)
92.1
83.3
87.6
81.4
87.1
85.4
82.6
53.5
81.6
G3-BitFit (B)
92.4
83.9
82.1
87.2
85.8
82.4
53.3
81.9
La-LoRA (L)
93.1
84.3
87.7
82.4
87.8
86.2
83.1
54.3
82.3
G3-(P. L)
92.8
84.1
88.7
82.6
88.2
86.2
83.3
54.7
82.6
Ga-(A, P)
93.1
83.8
89.1
82.3
88.1
85.8
82.6
55.1
82.5
Gy-(A,
84.5
88.4
82.8
88.2
86.1
83.5
54.6
82.6
Ga-(A.
92.8
84.6
88.1
82.5
87.7
85.5
83.2
53.8
82.3
Gs-(P, L)
93.6
84,9
89.3
83.1
88.2
86.5
83.9
55.8
83.2
G3-(A, P,
93.3
83.9
88.5
82.2
88.4
86.2
83.5
55.3
82.6
G3-(A, L.
93.4
84.2
88.9
82.6
87.8
85.8
84.2
54.9
82.7
S,B,A,
84.4
82.3
88.5
86.2
84.2
54.2
82.5
G4-Adapter (A)
92.8
85.2
89.1
83.5
87.8
86.5
84.2
56.3
83.2
GrePrefix
92.8
84.6
89.5
82.6
87.4
86.5
83.8
55.8
82.8
BitFit (B)
93.8
84.9
89.5
83.3
88.7
86.8
84.4
55.2
83.3
G (L)
93.3
84.7
89.3
82.7
88.3
86.2
82.7
54.7
82.7
G4-(P, L)
93.8
85.3
89.6
83.6
88.6
86.8
84.6
56.3
83.5
G+-A,
93.8
84.9
89.8
84.3
88.5
86.6
84.8
56.7
83.6
G)
85.6
84.1
88.2
86.6
85.2
55.4
83.5
L.)
94.2
85.2
89.6
83.9
88.2
86.4
84.9
55,9
83.5
G4-(P, B.
93.8
85.9
89.8
83.6
88.6
86.9
85.2
56.3
83.7
C,-(,,P
94.4
85.7
90.1
84.8
88.9
87.2
85.3
57.3
84.2
G L)
93.8
85.3
89.5
84.1
88.8
86.7
85.5
56.6
83.7
Ga P,
94.1
85.4
89.7
84.4
88.5
86.5
85.2
56.8
83.8
17

---

Published as a conference paper at ICLR 2023
Table 16: Average performances (low-compute, low-epoch regime: 100 random models, 3 tuning epochs) on the GLUE datasets using the T5-base pretrained backbone model. We compare adding different layer grouping constraints to the So design space. Layer grouping is based on 8 groups.
Layer Grouping
SST-2
MNLI
QNLI
QQP
RTE
STS-B
MRPC
CoLA
Avg
Sn-models
76.9
70.1
72.5
73.3
63.6
71.7
73.8
24.3
65.7
Increasing
83.2
74.1
76.6
77.1
67.7
76.8
74.7
30.0
70.0
Uniform
83.6
73.4
78.0
77.9
68.2
76.4
78.6
34.2
71.3
Decreasing
80.3
71.6
77.4
75.5
67.0
75.3
77.2
26.4
68.9
Spindle
86.2
74.3
79.1
78.6
68.5
77.4
79.5
35.1
72.3
Bottleneck
83.2
73.1
75.8
77.6
67.9
75.3
78.2
70.3
Table 17: Performances of different tuning methods on the SuperGLUE datasets using the XLNet- base (upper part) and XLNet-large (lower part) pretrained backbone models, respectively. The re- sults are averaged over 10 random runs. The Sa-model and Sa-3b-model perform significantly better than the second-bor best PEFT methods in all the eight datasets at the significance level p 0.05 (*) or even p < 0.01 (**).
Method
BoolQ
CB
COPA
MultkC
ReCoRD
WIC
WSC
Adgger
72.8
71.3/78.0
64.0
67,0/24.5
7L07718
76,2
65.0
60.8
66.2
Prefix
72.0
70.5/77.0
63.3
66.4/23.8
69.9/1.0
60.8
BidFit
70.0/76.2
LoRA
72.2
71./7788
64.3
67.4/24.8
708//133
76.8
65.1
61.1
66.4
S, mmndel
73.8**
71.7/78.4"
65.9"*
68 2/22/22..0 22.55
71.1/72.0"
78.4
65.8"
62.6"
67.5
Adageer
74.4
71,4/81.1
71.3/72.4
64.6
Prefix
72.4
70.0/78.3
66.9
68.8/25
70.9/1.2
73.8
66.9
64.0
67.7
BidFit
71.1
70.7/79 8
65.0
6/25
71,1/71.
$0.4
67.2
64.3
68.1
LoRA
72.0/72.8
3b-bm-odel
76.8
74.6/81.9
68.6
69.5/27.1*
72.4/73.3"
81.2"
68.2
64.8"
69.7
Table 18: Total training time (low-compute, low-epoch regime: 100 random models, 3 tuning epochs) on the GLUE datasets using the T5-base pretrained backbone model with A100 GPUs from S. to S..
SST-2
MNLI
QNLI
QQP
RTE
STS-B
MRPC
CoLA
18 mins
22 mins
20 mins 40 mins mins
12 mins
mins
mins
18


Reviews:
[{"review_id": "tojbvdviyop", "reviewer": "Reviewer_CSJR", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper empirically evaluates the best practice of parameter-efficient fine-tuning to achieve comparable performance with as few trainable parameters as possible. To achieve this goal, the authors design and implement different technique combinations and find the best solution via downstream experiments. Specifically, the pipeline contains four steps: 1) grouping layers into different sets; 2) adding trainable parameters towards each group; 3) deciding which group should be trained; 4) assigning groups with different training strategies.\n\nExperiments show that spindle-style grouping, uniformly allocating trainable parameters to each layer, and tuning all layers achieve the best practice performance.  Step 4 requires the evaluation of different strategy combinations from Adapter (A), Prefix (P), BitFit (B), and LoRA (L).   Different architectures show different best combination settings. \n\n", "summary_of_the_review": "The proposed combination method requires massive experiments and different architectures report different best settings. It is still unclear whether the proposed method is applicable to diverse architectures. Furthermore, the improvements over baselines are marginal in Table 5, and an ablation study is required to evaluate the contribution of different settings in the final performance. ", "strength_and_weaknesses": "Strength:\n\n1. The proposed architecture beats traditional widely-used approaches, like LoRA.\n2. The authors empirically explore the unified search space of parameter-efficient training. \n\n\nWeaknesses:\n\nMy main concern comes from the application of the proposed methods: If a simple parameter-efficient training method (e.g., LoRA) can achieve comparable results, why do we need to design a heavy search algorithm to search for the best solution with only marginal improvements? \n\n\n**Experiment Results**\n\n1. Different architectures have different best settings. It is time-consuming if we need to do a grid search given a new architecture. \n\n2. Table 5 shows that the improvements over LoRA mainly come from step 4. The authors should report more solid performance improvements to show the cost of searching is valuable. \n\n3. GLUE contains several small-size datasets, like RTE, and CoLA. Different random seeds can contribute to over 5+ accuracy variance. As we can see, the improvements over baselines mainly come from these two datasets. Mean and variance are required for convincing results. \n\n\n**Experiment Settings**\n\n\"we (i) randomly sample 100 models from the S0 design space that satisfy each grouping pattern constraint (Figure 2); (ii) fine-tune with 3 epochs; and (iii) compute the average performances for each design space. We will follow this procedure as we progressively add new constraints later.\"\n\nThe authors should report the costs of pre-training 100 models. \n\nFurthermore, the authors should report the mean and variance across 100 models. If the gap between mean performance and the best performance is small, do we really need such search space? \n\n\n\n**Experiment Conclusions**\n\nThe step2 and step3 seem to be artificial considering that the final decision is tuning all groups and uniformly allocating the number of trainable parameters. \n\nIf the authors can address these concerns, I'd like to raise the score.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "laSKti9b4o", "reviewer": "Reviewer_8TXz", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper systematically looks for the best way to combine parameter-efficient fine-tuning methods. Authors consider 4 design decisions: (1) how to group layers together (that layers within one group are treated equally), (2) how to allocate the \"budget\" of trainable parameters between groups, (3) whether to tune each of the groups and (4) the combination of PEFT algorithms to use in each group. In (2) and (3), authors find that naive solution is optimal. In (1) and (4), they find a combination of methods that outperforms all individual strategies. Finally, authors demonstrate that this combination can generalize to different models (e.g. from T5 to RoBERTa, BART) -- and that it is fairly consistent among tasks.", "summary_of_the_review": "To reiterate: I believe that this is paper is simply a systematic hyperparameter search -- in an area that could use just such a search. In its current form, the paper lacks several important details: justification of some choices (e.g. number of groups), verification of the search protocol (e.g. is it robust to the order of design spaces?) and reproducibility details (code, setups, config).", "strength_and_weaknesses": "### Strengths\n\nTo the best of my understanding, this paper is a glorified hyperparameter search. [If authors disagree, I will eagerly hear their argument] However, this is not a bad thing: parameter-efficient finetuning (arguably) badly needs a systematic hyperarameter search. The area is flooding with various algorithms and they do not always compare in the same setting. To that end,\n\n- the paper finds specific optimal combinations of algorithms for popular models. Even if we ignore generalization, these combinations are immediately useful to practitioners.\n- if authors truly found a combination of layers that generalizes across models (see a few concerns below), this combination could have an even greater practical impact, since it would be a great default combination for new transformer-based models\n- the paper reports several negative results in Section 4.2.3, 4.2.4. These are (again, arguably) not surprising, but still useful to save the time of researchers that tune their PEFT algorithms.\n\n\n### Weaknesses\n\nAll my concerns are about unjustified or under-explored design decisions:\n\n1. Using 4 groups: while authors mention that it is justified by some vision model, it is unclear why this choice is justified for the NLP tasks in question. One way to validate this choice would be to run an ablation analysis: if doubling the number of groups does not change results, then 4 groups are likely enough.\n2. The proposed search protocol answers one question at a time, e.g. group sizes first, then algorithms. To the best of my understanding, the order of these design spaces is arbitrary. Do the results hold if authors change the order in which they apply design constraints? e.g. select methods first, then determine group sizes.\n3. Authors explore how to allocate parameters between layer groups. However, when a single parameter group uses multiple methods (e.g. lora and soft prompts), how do they allocate parameters between groups? E.g. more prompts or higher adapter rank?\n\n\n### Questions\n\n1. To the best of my knowledge, T5 and BART are encoder-decoder models, while RoBERTa is encoder-only. How do authors apply grouping in encoder-decoder models? e.g. does the spindle grouping apply encoder and decoder separately, or pretend that they are stacked together? Does LoRA apply to T5/BART's cross-attention, or only self attention? Does any of this affect model quality?\n\n2. Concerning the en-ro machine translation experiments: to the best of my understanding, the pre-trained models were not trained on the destination language. Furthermore, its tokenizer / embeddings might be unsuitable for this language. How exactly do you run these experiments?\n\n3. the search protocol for finding optimal design spaces is based on random search. The same problem can be solved with [A] hyperparameter optimization techniques, e.g. TPE or gaussian process optimization (see Ray-tune or Optuna) or [B] neural architecture search. Why did authors opt for random search?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "WYcf3HathX", "reviewer": "Reviewer_3ina", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Parameter-efficient training (PET) is a very popular technique for domain and/or task adaptation of large pre-trained models. As opposed to in-context learning, it does perform weight updates, but much less so than full fine-tuning (0.5% of the total parameters in this case).\nMany different techniques and architectures have been proposed over the last few years, and this paper proposes to:\n1. model that design space in a more structured way\n2. go over that design space empirically and discover possible new designs for PET\n\nAs a result they do not only show that this is possible, but also that the obtained design have some type of generalization, carrying over to different architectures and tasks", "summary_of_the_review": "A good and rather exhaustive exploration of different design choices, addressing the issue that current PET strategies are rather rigid. The obtained design performs better than previous strategies.", "strength_and_weaknesses": "STRENGTHS\n1. A proposal to structure the different PET proposals into a unifying design space. In this sense, it is similar to He et al. (2022) (on this, a more thorough explanation of the main difference with that paper would be welcome)\n2. A pragmatic way of searching over that space\n3. Empirical results showing that the discovered design is better than previously proposed architectures, and generalize to new architecture and tasks. Specificailly, the obtained fine-tuning strategy carries over from T5 to Roberta, indicating that the findings are not ad-hoc for the T5 architecture. Similarly, the discovered design carries over from GLUE to a very different tasks (summarisation on XSUM, and translation for en-ro).\n\nWEAKNESS\n1. From the claims, I was expecting to read a \"Evolved Transformer for parameter-efficient training\" paper. This is not it, in that the search space is tree-based, and the design selection is greedy over each of the four stages. This is still interesting, as current proposals always felt wasteful in that they run the same architecture on all layers, but not necessarily what was expected from the abstract\n2. Clarity. The paper is easy to read, but hard to understand in its details. For instance\n - what are the four groups? A very quick search of the provided reference (Dosovitskiy et al. (2020) does not answer that \n - the paper (specially the appendices) contain tons of tables and numbers, but it is not clear - even for a non-casual reader - what all those tables are saying. As an example, 4.3 seems to make the claim that the discovered design space carries over from T5-base to T5-3b, but it is not clear what table needs to be compared with what table. I compared Table 15 with all of Table 8-11 and from those it seems that this claim is actually not true (this is, the best strategy assignment per group for T5-3b is different than the best strategy assignment per group for T5-base)\n ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "DKcZLFJx2e", "reviewer": "Reviewer_CmEs", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a strategy to find a parameter-efficient tuneable architecture from a given pre-trained backbone neural network. In particular, the strategy consits of four phases exploring 1) how to group layers in the backbone neural network; 2) how to allocate the tuneable parameters within each group; 3) how to decide which groups to tune and which groups to keep frozen; 4) assign the best strategy to increase the parameters in each group. The paper uses this strategy to find an architecture based on BERT, RoBERTa and BART that tunes only 0.5% of the original parameters, and achieves better results than full fine-tuning on GLUE for BERT and RoBERTa backbones.", "summary_of_the_review": "The paper proposes a simple-to-implement strategy to find a good way of tune a small set of parameters instead of fine-tuning the entire network, which gives excellent results in the GLUE benchmarks for different Transformer-based backbone networks. The work, however, has several and important drawbacks mentioned above, thus I think it's marginally below the acceptance threshold. If the authors are able to satisfactorily address some of my concerns, I'll be happy to increase my score.\n\nUPDATE AFTER DISCUSSION:\nThe authors have addressed some of my concerns and clarified my questions. Thus, I'm increasing the score from 5 to 6. I thank the authors for their detailed response.", "strength_and_weaknesses": "**Strengths**\n- The paper is excellently written, it's easy to follow, and enough details are provided for practitioners and other researchers to implement the method.\n- The proposed approach seems to provide quite consistent improvements (2 out of the 3 backbones improve results w.r.t. full fine-tuning). \n- Although there are no experiments with runs using multiple random seeds, there are extensive experiments on multiple datasets (GLUE benchmark), and the best options are quite consistent across all datasets.\n\n**Weaknesses**\n- The main criticism is that the paper ignores other important baselines to choose where/how to tune the 0.5% of parameters during fine-tuning. For instance, the search space could be encoded in one of the many AutoML algorithms and let it find the optimal strategy, rather than relying on the fixed and hardcoded steps suggested by the paper, and then comparing the final results under a certain compute budget.\n- The second shortcomming is that the three backbone architectures that have been tested are Transformer-based. The method should be ideally also tested with other backbones and/or domains (e.g. ResNets for image classification).\n- The goal of having only 0.5% of tuneable parameters should be estated. For instance, if the goal is to reduce the fine-tuning cost, notice that the computational savings of tuning only 0.5% of parameters are nothing compared to the large amount of tuning needed to find the optimal strategy.\n- It's not clear how the \"Prefix\" strategy can be applied independently on each group of layers. After all, it adds a new trainable token to the sequence. So, when \"Prefix\" it's added to a given layer (or group), all subsequent layers will have an additional token. No? Could you clarify this?\n- The paper states that due to computational limitations, not all combinations were explored in this work. That's fine, but then the paper should be explicitly mention the cost of applying the proposed strategy w.r.t. the number of layers in the network and strategies considered.\nOverall, the subspace explored in each design step by the paper isn't clear.\n- The overall cost of finding a good strategy to find a small subset of parameters to fine-tune is very expensive, though. Thus, this reviewer has doubts that this will be applied in practice, in comparison to simply fine-tuning the entire network or just using Adapters or LoRa in all layers, which give competitive results with a much simpler (and cheaper) process.\n- For example: when exploring the space $S_1$, depending on the number of layers, there are multiple options for the Increasing, Decreasing, Spindle, and Bottleneck options (e.g. for a network with 12 layers, with increasing strategy, valid configurations are $N_1 = 1, N_2 = 2, N_3 = 4, N_4 = 5$ or $N_1 = 1, N_2 = 2, N_3 = 3, N_4 = 6$), but it's not indicated which ones are recommended or were explored in the paper.\n- In Table 6 (BART results), the bolded results should be those of \"full\" and not the proposed method, since \"full\" is better than the proposed method.\n- Clarification: when multiple strategies are applied on the same group, it means that all the strategies are applied in all layers of the group, right? (e.g. $G_1-(A, L)$ means that in every layer of $G_1$ adapters are added and LoRa is applied).\n- Suggestion: In table 7 (appendix A), add an extra (multirow) column indicating the number of epochs on each group, for better readability.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster