Sparse Structure Search for Delta Tuning
Authors: Shengding Hu
Zhen Zhang
Ning Ding
Yadao Wang
Yasheng Wang
Zhiyuan Liu
Maosong Sun
Venue: NeurIPS 2022 Accept (2022)
Keywords: Parameter Efficient Tuning
Pre-trained Models
Neural Architecture Search
Delta Tuning

Abstract: Adapting large pre-trained models (PTMs) through fine-tuning imposes prohibitive computational and storage burdens. Recent studies of delta tuning (DT), i.e., parameter-efficient tuning,  find that only optimizing a small portion of parameters conditioned on PTMs could yield on-par performance compared to conventional fine-tuning. Generally, DT methods exquisitely design delta modules (DT modules) which could be applied to arbitrary fine-grained positions inside PTMs. However, the effectiveness of these fine-grained positions largely relies on sophisticated manual designation, thereby usually producing sub-optimal results. In contrast to the manual designation, we explore constructing DT modules in an automatic manner. We automatically \textbf{S}earch for the \textbf{S}parse \textbf{S}tructure of \textbf{Delta} Tuning (S$^3$Delta).   Based on a unified framework of various DT methods, S$^3$Delta conducts the differentiable DT structure search through bi-level optimization and proposes shifted global sigmoid method to explicitly control the number of trainable parameters.  Extensive experiments show that S$^3$Delta surpasses manual and random structures with less trainable parameters. The searched structures preserve more than 99\% fine-tuning performance with 0.01\% trainable parameters. Moreover, the advantage of S$^3$Delta is amplified with extremely low trainable parameters budgets (0.0009\%$\sim$0.01\%). The searched structures are transferable and explainable, providing suggestions and guidance for the future design of DT methods. Our codes are publicly available at \url{https://github.com/thunlp/S3Delta}.

Sparse Structure Search for Delta Tuning
Shengding Hu Zhen Zhang' Ninn Ding', Yadao Wang", Yasheng Wang", Zhiyuan Liu Maosong Sun
Dept. of Comp. Sci. & Tech., Institute for AI, Tsinghua University, Beijing, China Beijing National Research Center for Information Science and Technology Institute Guo Qiang, Tsinghua University, Beijing, China 'Noah's ' Ark Lab, Huawei Inteerattinall Innovation Center of Tsinghua University, Shanghai, China {hsd20, zhen-zhal Cmails. tsinghua. edu.
Abstract
Adapting large pre-trained models (PTMs) through fine-tuning imposes prohibitive computational and storage burdens. Recent studies of delta tuning (DT), i.e., parameter-efficiemt tuning, find that only optimizing small portion of parameters conditioned on PTMs could yield on-par performance compared to conventional fine-tuning. Generally, DT methods exquisitely design delta modules (DT modules) which could be applied to arbitrary fine-grained positions inside PTMs. However, the effectiveness of these fin-grained positions largely relies sophisticated manual designation, thereby usually producing sub-optimal results. In contrast the manual designation,
explore constructing DT modules in automatic manner. We automatically Search for the Sparse Structure of Delta Tuning De*tell)). Based on a unified framework of various DT methods, De conducts the differentiable DT structure search through bi-level optimization and proposes shifted global sigmoid method to explicitly control the number of trainable pa- rameters. Extensive experiments show that Delta surpasses manual and random structures with less trainable parameters. The searched structures preserve more than 99% fine-tuning performance with 0.01 % trainable parameters. Moreover, the advantage of D is amplified with extremely low trainable parameters budgets (0. (0.0009%~0l) 1.001% %).. The searched structures are transferable and explainable, providing suggestions and guidance for the future design of DT methods. Our codes are publicly available at htttps:: : http//gttttt//t/// wttcchhnnn////eele
Introduction
building upon Transformers (36 have been emerging and achieving state-of-the-art results on variety of downstream tasks. Despite the blessing effectiveness, these big models also bring the curse of prohibitive costs on computation and storage during the adaptation because of the gradient computation of the whole model and the giant size of the fine-tuned checkpoint.
Increasingly large pre-trained models (PTMs) 6 27 30 31
[2]
To alleviate such costs, studies of delta tuning (DT) [7]. also known as parameter-efficcen tuning [15] 28
20 have been developed, which only train a small portion of PTMs and keep the vast majority of parameters frozen. Studies have verified that delta tuning could achieve competitive performance compared conventional fine-tuning with very few trainable parameters, resulting in considerable savings in model adaptation costs. Generally, these approaches manually design delta modules (DT modules) to complete model adaptation. For example, adapter-based
Equal contribution, ordered alphahetically. t Corresponding authors: Z.Liu (liuzy @tsinggua.edd.ccm
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

layers to Transformers and only fine- tune 0.5%-8% parameters to yield promising results: BitFit 42 only fine-tunes the bias terms (0.04% -.1 %% parameters) within Transformers; LoRA [6] inserts trainable rank decomposition matrices to each layer of Transformers and is successfully adopted on GPT-3 [ with 175 billion parameters.
methods [15] 28 25
inject two newly-intne inwyo-intoodueed
While early research focused on how to design practically effective DT modules, more recent research has advanced the understanding of delta tuning more deeply. He al. [3] bridge connections among different approaches to form a unified framework. And Ding et al. 7 indicate that the combination of different trainable modules could bring different levels of gain on downstream tasks. The above empirical evidence implies that there may exist an optimal mixture of DT modules that is more effective than manually designed structures. In fact, considering the fine-grained structure inside PTMs, the positions where the DT modules could be applied are numerous, but not all DT modules at all positions contribute equally to the task performance. How to find the optimal structure of DT modules and remove the redundancy in the trainable parameters is essential for a more efficient adaption method. Predictably, such optimal structure is difficult to construct artificially and may vary with specific tasks and models. Therefore, we propose to automatically search the optimal structure that contains a mixture of DT modules at diverse positions inside PTMs. Also importantly, the structure should be sparse to ensure the parameter efficiency.
We present Sparse Structure Search for Delta Tuning Delta) to automatically search such optimal trainable structure, which could flexibly control the number of trainable parameters according to practical requirements. The searching process and the optimization of S Delta is guided by performance to ensure the effectiveness on specific tasks. Moreover, the structures change automatically to suit the preset limitation of the number of trainable parameters. In contrast, heuristically designed structures are usually coarse-graines and independent of performance and budget, making them neither optimal nor flexible to adjust the number of trainable parameters.
In terms of the specific methodology, we firstly construct a unified search space by applying probabilistic gating controlled by structural parameters to all potential DT modules. Then, we develop a framework of differentiable DT structure search by treating the problem as a constrained neural architecture search problem. In our framework, the structural parameters are updated via bi-level optimization [22]. Unlike the traditional neural architecture search that learns from scratch, we implement the first neural structure search based on a pre-defined backbone and under the delta tuning scenario. To search under a pre-defined budget of trainable parameters, we develop shifted global sigmoid to explicitly control the number of activated DT modules in the searching phase.
We conducted extensive experiments to study the effectiveness of Deelta. Firstly, the experiments show that with 0.01 % parameters, we are able to recover 99% and 98% fine-tuning performance on GLUE (38 and SuperGLUE (37), respectively. Secondly, the searched structure surpasses the human- designed structures considerably while consuming less (~ 1/5) trainable parameters. Moreover, the advantage enlarges when the number of trainable parameters is minimal (0.0009%~~ 0.01% 1)).. Furthermore, the searched structures are transferable across tasks, which significantly strengthens the usefulness of the searched structures. Apart from the performance boost, we visualize and explain the searched structures, which is beneficial to the future design of new DT methods.
2 Related Work
Delta Tuning (DT), Our work is related to the studies of delta tuning (DT) for pre-trained models [7]. Generally, DT only optimizes a small portion of parameters and leaves the vast majority of parameters untouched for the adaptation to downstream tasks. Pioneer work select parts of the PTMs to be trainable (35] 10]. Adapter [15] is one of the earliest methods that apply the concept of parameter-wis. efficiency to pre-trained language models, which inserts linear neural modules to every Transformer layer and achieves on-par results to full fine-tuning. As the PTMs scaling in recent years, DT is valued for its efficiency in computing and storage. This has spawned not only empirical studies (28 [4] and variants on the adapter [28] 25 34). but also a range of other approaches. Prefix tuning 20 prepends embeddings to the hidden states of the Transformer model, and prompt tuning [18] further simplifies the strategy and only prepends such embeddings to the input layer. There are also approaches which specify some of the parameters inside PTMs that can be trainable to achieve good results, such as Masking (3). BitFit [42). DiffPruning 19] etc. LoRA [16) assumes that the change in model weights intrinsically low-rank after fine-tuning, and uses trainable rank cemmpoooopoiition matrices for model adaptation. In addition to specific methods, some

---

studies have comprchensively investigated DT methods. He et al. [33 models multiple methods unified manner, Ding al. Z provides theoretical discussion and comprehensive empirical study of these methods. Our work proposes to automatically search for trainable structures in the context of DT, which a different perspective from all the aforementioned work. In terms of the structure of DT, AdapterDrop [3] explores dropping fraction of Adapter modules based on manual trials. A concurrent work [26] learns switches on Adapter modules to select the beneficial adapter modules. However, it is not optimized under preset number of trainable parameters. They also both limited to adapter-basee -based methods. On the contrary, our proposed method can search within mixture of almost all DT modules under constrained trainable parameter budget.
Neural Architecture Search (NAS). Our work conducts structure search in the scope of DT, which is related to the Neural Architecture Search algorithms. A line of NAS algorithms uses Reinforcement Learning or Evolutionary Algorithms to explore the best structure with reward from training the structure from scratch 44
[2)]. which usually consumes prohibitive computation 21 approaches the problem with gradient-based optimization. DARTS [22] relaxes the discrete structure using continuous structural parameters, which are optimized with gradient-baser optimization. DARTS achieves competitive performances with much fewer computational resources. We take inspirations from DARTS in optimizing the structural parameters of Delta. We are also the first to conduct NAS conditioned on a pre-trained backbone model. We also take inspiration from the NAS algorithms with binary gates 3 40].
45 32
resources. Another line of NAS algorithms [2
Method
In this section, we firstly introduce the preliminaries of pre-trained model adaptation, transformer architecture, and the delta tuning. Then we introduce method Delta in detail.
3.1 Preliminaries
Pretrained Model Adaptation. The recent prevalent pre-train then fine-tune paradigm in deep learning takes advantage of a pre-trained model M with parameters θ and continues to optimize θ on downstream task D = (Duiin D Dres under objective function L. In fine-tuning, all the parameters of the pre-trained model are optimized using the train split to minimize L, i.e.
mine (O), D Praiin.
(I)
Transformer Architecture. The pre-trained models typically adopt the Transformer model [36] as their backbone. The Transformer model is composed of multiple stacked Transformer layers that processes the hidden state sequentially through different computation modules, such as Self-Attention module (SelfAtt)). Cross-Attention module (CrossAttm), Feed-Forward module (FFN), and Layer Normalization module (LN), etc., and details of each module are in Appendix [A] The computation process the Transformer can be abstracted by sequence of transformations of hidden representation. In each computation step, the input hidden representation H E I2x R is transformed into an output hidden representation got E IR= where is the sequence length of the input and d1, d2 are the hidden dimensions,
Delta Tuning (DT), DT methods only train a small portion of parameters conditioned on the backbone PTMs to improve the adaptation efficiency [15
28
42
16
25
9
20
18] Although the specific
forms of the various DT modules are substantially different, He et al. 13 unify them as modifications
A of the hidden state?
Hot mm A.
The formulas of some DT methods under the unified view are listed in Table [ The DT modules can be applied to extensive positions the backbone PTMs, which listed in the rightmost column in Table I In training, we freeze all the parameters in the backbone module m. i..., θ and set
We use little bit more flexible notation than 13 which takes into account the frozen backbone module and thus can distinguish the DT modules that take either Hd or H" as their input.
3

---

Table 1: Different DT methods are the specializations of the unified view (Equation and can be
applied to extensive positions on the PTMs.
Method
Transformation
A
Potential Positions
LoRA [6
[ AB)
H&AB
Weight matrices
Adapter 15
H
( + [[m[H"")) W down Wap H
(mm(H") dowz Wop
After any modules
Purallel Adapter 13
m + f(H W Wip
JHH" Wamm! Wap
Between mny modules
BitFit 42
H ml m(H"") by
by
Linear layers
LNFi
H Var. H. 86)
Layer Normalization modules
Unifeed Scorch Space
Differemiable Delta Structure Search
Searched Sparse Siuueture
with Probabilisiic Gating
① Explicit guanity commot
O Diffecesttaaee sampling
ENFI
FENCIR
FENCRO
D
FFNU
BetTi
FPNUSS
THTTI
① Bileed apimisciun
OFF
ON
AMBLO
(
Am3
Amriq
ARB N
Ami
Amti
AuR V
Figure 1: The framework of Delta. We propose unified search space with probabilistic gating to enable search among a mixture of DT methods. We find the optimal sparse structure using the differentiable DT structure search and explicit sparsity control.
parameters introduced in computing A, denoted by ô, as the only trainable parameters. Therefore, the adaptation objective in DT is
mins ((M(O, 6), Duain).
For the convenience of notation, simplify Equation (4 into
ming Laain (8).
(5)
3.2 Sparse Structure Search for Delta Tuning
Our target is to search for the optimal structure of DT constrained by pre-defined and limited trainable parameter budget B. To achieve this, we design Sparse Structure Search for Delta Tuning Deeta,,, which is driven by three essential components: a unified search space with probabilistic gating, an efficient differentiable DT structure search algorithm, and an explicit sparsity control algorithm using shifted global sigmoid.
Unified Search Space with Probabilistic Gating. The potential positions in the backbone models to which DT modules could be applied are extensive, especially when we consider a mixture of different types of DT modules through the unified view (Equation (3). However, not all positions contribute to the task performance equally, and only fraction of positions should be activated avoid redundancy of the trainable parameters. To this end, we design probabilistic gating mechanism over all possible DT modules' positions. Specifically, for each DT module that computes A, for the hidden representation, we activate the modification A, with probability p; [0, 1].
H m(H 2,i,,
(6)
where 2; E [0,1) ~ B(1,p:) is a random sample from Bernoulli distribution.
Differentiable DT Structure Search. Finding the optimal structure from search space with abun- dant potential positions is challenging due to the compositionality of activated positions. Furthermore,
'LNFit only trains the variance vector in the Layer Normalization module of the PTMs, which is inspired by Frankle et al. [8] who only train the Batch Normalization module in Convolutional Neural Networks.

---

directly comparing the fully trained model with each DT structure is intractable. In our work, we propose optimize the gating probability P. with madient-basee optimization. To make the sampling process differentiable, we use the Binary Concrete Distribution 24 approximation to the Bernoulli distribution,
17
as soft and differentiable
upp
2, = log
(7)
(1 - u1- p.)
where u ~ (0, 1) a random sample from the uniform distribution in [0, 1] and (3 is the temperature to control the sharpness of the distribution & distribution O Similar distributions
used in learning sparse networks [23] or pruning dense networks [39]. By replacing the hard sample 2, with the soft approximation 2, we can mack-propagate through p; in training. However, directly optimizing p; in the probability space [0. 1] may lead to numerical instability, therefore, we parameterize it with structural parameters a, R. i.e., P: g(o;). And we denote all the structure parameters
We optimize a through bi-level optimization [ 22 i.e., optimizing a conditioned on the optimized parameters 8 of the DT modules. The inner and outer level of optimization are conducted on separate splits of training data, denoted by D&; Da. which is analogous to validating structures trained on Dş using a different split Da to avoid over-fitting to Dz. Thus, the optimization objective is
mina (M 8", a)),
(8)
s.l. 8" argming (M
Following DARTS [22]. we make approximations to the gradient of the structural parameters by applying chain rule and taking finite difference approximations m
VoLa Va (VoLs(0,)))
(10)
(11)
v..L (8) - Vo (8),
(12)
~ Vafa(68 - vaLs(O a) - VaLs(o
(13)
2e
where the optimal 5* is approximated by the parameters of one-step update = 5 ax). is the learning rate of parameters 5, and is small scalar used in the finite difference approximation.
Explicit Sparsity Control with Shifted Global Sigmoid. Most the DT modules in the search space are redundant and contribute little to the performance. However, the search algorithm may not be aware of the sparsity target and degenerate to greedily adding more DT modules. As opposed to previous sparse network learning methods [23 9 which punish the dense structures with Lo regularization, we explicitly control the sparsity of structure at the target level during the search through shifted global sigmoid parameterization. (See Section 4.9 for comparing the two methods),
E, Detach(fi))
P: D0
(14)
Eipr P::
where
p. = Sigmoid
(15)
The Detach(-) operator turns a parameter that requires gradient into scalar that is free from gradient computation. Equation (14) doesn't change the value of p.. but enforces the competition among different positions and DT modules, which is similar to Softmax operation (See Appendix C.2 for
details).
'The distribution of & has property that P(s, > 0.5) pt. and when 9 approximate O, distribution 2, converges to B11 p.) (See Appendix C.1 which makes it a suitable surrogate for Bemoulli distribution. We use the same sample compute Cot a)) VaLo(8" ,a), VoLa(8, a),, and VaLa (8', a).
5

---

In Equation 15 ( is scalar. Increasing Ç's value will monotonically reduce to 0 while keeping Pj in [0, 11. So the expected number of trainable parameters E[N] monotonic function w.r.t. Ç.
N = E Σ T( 1218,1 R E Σ T(2 0.5)|o:l
Σ pilo.l.
(16)
where the 6. is the number of parameters introduced in computing A, Thus, we can dynamically adjust to make B[N] approach B via monotonie optimization,
5* = argmin Σ Ps p: - B3), where Σ Ep.16,1 S B.
(17)
Evaluation of the Searched Structure. To determine the final structure DT, instead of sampling from Pt. we choose the set of positions where the sum of is the highest while still being within the budget B. This deterministic algorithm reduces the variance of the final structures. After obtaining the final structure, we re- re-initialize and re-train the parameters in the DT modules to converge on D
Algorithm 1 Algorithm of Delta
Initialize all DT modules in the search space, and initialize a. while not converged do
1. Calculate Ç. Pi. and sample 24.
Compute the each loss terms by forward and backward propagation.
3. Update a according to Equation (13).
Update ö using VoLo(ô, a)).
end while
Determine and evaluate the final structure.
4 Experiments
4.1 Datasets and PTMs
We apply De to multitask benchmarks GLUE 33 and SuperGLUE [37 following previous works. All datasets are downloaded from the HuggingFace Datasets [9]. Since the test splits of these datasets are held officially and invisible to the researchers, we conduct random splits from either train set or validation set to make the new train, validation, and test splits, which is critical to ensure fair evaluations according to Chen et al. [4] We repeat 4 times using different random seeds for experiments in Table 2 and times for experiments in Figure 2] The details are in Appendix [ We use the Taage model (703M parameters) as the backbone PTMs.
4.2 Baselines
We compare & with several widely used baselines (See Appendix for details). Fine-tune.
Traditional fine-tuning trains all parameters the PTMs.
LoRA. We apply LoRA linear layer to the Self-Attention's 's query modules and value modules as Hu al. [16] suggest. We include two rank levels (r and r 1) in our experiments.
Adapter. We adopt the first adapter method proposed by Houlsby et al. [15]. Their method requires more parameters than the other methods but achieves good empirical results.
Low Rank Adapter (Adapter-L.R). We adopt the Low Rank Adapter as an efficient variant of the adapter-based based method. It is proposed in 25
simple but effective baseline. The rank is set to 1.
BitFit. BitFit proposes only adapt the bias layer in the model. We adopt the same setting as Zaken et al. that tunes the bias inside all linear modules, and the Layer Normalization layer?
LNFit. We train the variance vector of all Layer Normalization layers', including the Layer Normal- ization after the whole transformer encoder.
Although T5 has bias in linear modules, treat it bias vectors with initialization.

---

Table 2: Results on GLUE 38 benchmark (above) and SuperGLUE [37] benchmark (below). Green and blue represent the best and second best scores, respectively, among the methods in our search space. The first three rows represent the results of fine-tuning and other DT methods, which are not used in our search space due to high trainable parameter ratios. On SuperGLUE tasks, since the results on COPA vary dramatically (=26.00), the average results of SuperGLUE become easily dominated by the results on COPA. Therefore we also report the average results that exclude COPA (AVG -COA)) The widths of the yellow rectangles are proportional to the trainable parameter ratios.
GLUE
Parameter Rotios
Method
CoLA
SST2
MRPC
QQP
STSB
MNLI
QNLI
AYG
1000056%
Fine-tune
62.25 25 3.96
95.87
0.42
91.86 + 1.19
89.50
0.22
36
$9.61
0.30
94.22
0.35
87.88
65.339/%
59.03
0.28
88.39
0.06
0.25
89.53
94.17
0.19
87.40
21,325/%
LoRA(e-8)
58.43
4.16
95.79
0.27
92.21
0.88
88.35
0.25
91.78 0.31
89.38
0.32
94.14
0.12
87.15
Methods in the Search Space
8.13%%
BitFit
56.98 3.89
96.24
0.33
92.16
+: 0.68
88.12
a: 0.07
91.59
0.08
89.10 2: 0.09
94,07 :: 0.21
36.90
4.121%%
Adapter- LR
56.78 1. 4.80
95.90
0.14
92.76
1 0.67
88.08
0.13
91 26
0.31
89:30
0.34
93:94 1 0.07
36.86
2.67%%%
LoRAGr=
56.77
2.29
1.00
91.54
0.33
0.05
1,201%%
LNFi
56.15
4: 4.06
95.81
A: 0.20
91.71
-: 0.39
88.17
0.10
91.37 0.24
89.11 2: 0.09
93.99 1: 0.20
56.62
1.39%%%
S Delta M
39.34 +: 4.75
95.8
0.14
92.13
+: 2.09
83.04
=: 0.23
91.58
0.25
89.14 2: 0.13
94.12 4: 0.12
87.17
1.39%%
8 Delta S De
36.71 1. 3.03
95.93
0.15
93.27
1.39
88.14
0.08
91.58
0.19 91.38
88.81
0.44
93.95
1 0.11
$6.91
0.35%
54.56 + 3.66
95.93
0:24
92.14
1.10
88.02
0.20
89.04
93.93
0.14
SuperGLUR
Metheal
Booil
COPA
MultRC
ReCORD
10000/5/
tune Adapter
36.67
13.50 5.26
76.65
1.01
85.03
0.67
154.41
65 06.333664
35.98 à 0.68
94.64
63,00 7.75
77.60
0.84
85.96
0.37
89.21
2.94
84.17
31.32%6% %
LoRAir33
35.06 0.70
91.96
51,00
76.94
1.16
85.84
87.05
72.00
78.56
Medbods
Search Space
8.1318%
BiiFit
85.00 x: 0:48
89.29 :: 2.92
15:00 1: 8.08
75.79 1.15
85.85 :: 0.32
86 15
:: 1.46
12:34
1,61
81:35
82:41
4.125/%
Adupter
96.04 0.09
85.61
2.679%-%
LoRA(-1)
67.50
76.71 1.05
85.05
1.08
1,709/%
LNFI
1,309%%
SPDelMoMM De
34.02
76.35 0.92
86
1.90
83,10
1
6.10
76.17 1.41
96.02
0
S"Deltu-
83.56
87.50 =: 54,00
76.09
0.97
86.10 0.26
35.79 1.89
68.42
77.35
X1.24
We do not include Prompt Tuning I8] as our baseline because it takes much longer steps to converge and doesn't achieve competitive performance on Tr 18
4.3
S Delta Search Spaces and Budgets
The search space has a noteworthy influence on the performance of Delta. In our experiment, we define two kinds of search space.
Mix. The first search space considers mixture of LoRA, Adapter-LR, Bitfit, and LNFIT modules. LoRA can be applied to any linear modules in the transformer block, including the query(Q). key(K), value(V), output(O) sub-modules of the attention module(ATTN), and the two sub layer W1 and W2 in Feed Forward modules(FFN). The Adapter-LR can theoretically be applied to any position in the computational graph. However, avoid overcomplicating the search space, we limit the outputs of the ATTN module and the FFN modules. For BitFit, the potential applied positions are all the linear modules (Q, K, V, O, WI, W2) and the Layer Normalization modules (LN). For LNFit, the potential positions are all the LN modules. In this search space, there are 916 potential DT modules to be selected and the total number of candidate structures is 2916 if we do not consider the budget constraint. We denote the structures searched on this search space as $ Delta-M.
LoRA. We narrow down the search space into single type of DT module. We choose LoRA as an example. The potential positions are the same as the LoRA modules in Mix search space. There are 288 potential positions in total. We denote the structures searched this search space Delta-L.
We also explore different numbers of trainable parameters. Experiments in Table 2 are conducted on 1.39% % and 0.35% % trainable parameters ratios. More sparsity levels are tested in section 4.5
4.4 Results on GLUE and SuperGLUEE
Table 2 shows the performance of different methods on GLUE and SuperGLUE tasks. Comparing De with the manual structures within the search space, we find that DeDelta-M (1.39%%) %) achieves the highest average score on GLUE and SuperGI.UF (without COPA) despite using the least number of trainable parameters (~ 1/5 compared to BitFit). D'Detta-M (0.35%% %) also
7

---

RTE
MaluRC
MRFC
91.0
76.
194
73.
Na
NB
0.055
1,50
0.722 5025
3.02
1.503
Spundhy(S
Spunsiyy(%
(a) RTE
(b) MultiCC
(c) MRPC
Figure 2: Performances under different trainable parameters ratios. The x-axis represents the ratio of the number of trainable parameters to the backbone PTM's parameters. The scaled y-axis represents the scores. The accuracy of fine-tuning is in gray horizontal line. The result of LoRA (r=1) and Low Rank Adapter are plotted in grey dot.
surpasses Adapter-LR, LoRA (r=1), LNFit using approximately 1/2, 1/5 trainable parameters, respectively. Narrowing the search space from Mix to LoRA leads to a moderate decrease in performance, which justifies the need to search among mixture of DT modules. It may also hint that the combination of different DT modules could lead to stronger performance. However, even though the performance of Delta-L is not optimal, compares favorably to LoRA(r=1), which also shows that the human designed structures, though benefit from applying DT modules uniformly on the PTMs, is sub-optimal. Compared with fine-tuning, Deeeta (1.39%%) preserves 99.2% and 98.1 % performance on GLUE and SuperGLUE, respectively. In fact, we must emphasize that Delta orthogonal to specific DT modules. The performance of Delta can benefit from the future invention of better DT modules, thus potentially achieving comparable or even superior performance to fine-tuning with extremely limited trainable parameters.
4.5 Performance under Different Sparsity Levels
To explore the limit of trainable parameter reduction, we train different methods with decreasing sparsity levels from 5.6%% to 0.086%% %. To apply baseline methods target numbers of trainable parameters, we randomly sample a set of potential positions in their corresponding search space to reach the target sparsity level. In Figure 2 demonstrate the results three datasets, RTE, MultiRC and MRPC. We can see that Deltta- and Delett--L have considerable advantages in extremely low trainable parameter budgets. For example, DelttaM trains only 0.086%% parameters whereas recovering 96.8%,98.77 7%.97.5%%% of the FT performances on RTE, MultiRC, MRPC, respectively. With 5.6%% trainable parameters on MultiRC and MPRC, all the methods saturate to FT performance, proving the feasibility of removing redundant parameters with Delta.
4.6 Transferability of the Searched Structures
Another essential characteristic of S Delta is the transferability of the searched structure. In Table 4 we split the GLUE benchmark into source datasets and target datasets. We search on the source dataset (Mix search space) and train the searched structure the target datasets. We can that the searched structures are highly transferable, even surpassing the structures direct searched on the target datasets sometimes. The transferability guarantees the reusability of the searched structures.
4.7 Efficiency of the Search Process
Although Delta focuses on the parameter-efficieng of the searched structures. we also analyze the searching efficiency in Table Generally speaking, the search for an optimal structure consumes 5-8 times training time and times GPÜ memory (Due to bi-level optimization). However, affordable compared to manually designing different structures and running numerous evaluations.
4.8 Visualization and Explanations of the Search Structures
To understand the searched structures, we draw the heat maps of the p: on different datasets in Appendix [D.2 We find obvious patterns and similarities most datasets. Therefore, we average the p; across datasets to see the overall pattern of the searched structure. Figure 3 shows the heatmap of p. of

---

Table 3: The computational resources in the searching phase and re- re-tranning phase, Computation time, memory consumption are listed.
kow (011
Dataset
Search
Re-train
Ratio
10 ar1
Time/min
Fed DXLAT.
RTE
148.6
30.0
5.0
STSB
139.3
26.3
5.3
Figure 3: Visualization of p: of S Delta-M. The numbers on the squares are the average of p; across all datasets and all seeds. The deeper the color is, the more activated is the DT module. The x-axis represents different layers of PTMs (E denotes Encoder, and D denotes Decoder), and the y-axis represents different positions (modules) of PTMs.
CoLA
145.6
17.0
8.6
Memory/GB
RTE
27.7
16.6
1.7
STSB
28.9
10.6
2.7
CoLA
16.1
8.9
1.8
S'Delta vs Lo Regularization
90
S'aelta MERC
Score
84
MRRE
052
8'Delta
Figure 5: Visualization of p, of DT modules in Delta-L.
SITE
Table 4: Structure transfer from source datasets and target datasets. The target datasets are in the row name, and the source datasets are in the column names. "No transfer" means the structure is searched on the target dataset.
74
5.555
1,388
0.348
0.172 n. 8.086
Sparstty(%%)
Figure 4: Comparing shifted global sigmoid to To regularization. Perfor- mance on different datasets is in differ- ent colors. Shifted global sigmoid and Lo regularization are in solid lines and dotted lines, respectively.
Source
Target Datasets
STSB
QQP
QNLI
CoLAA
No transfer
91.58
0.25
88.03
0.23
94.11 1. 0.12
59.34 1 4.75
MRPC
91.63 + 0.31
86.16
0.08
93.96 + 0.06
56.41 + 3.81
MNLI SST2
91.39 0.67
88.06
0.08
94.13 0.10
56.38
= 3.98
91.37 = 0.23
88.02 = 0.10
94.14 2 0.16
55.58 = 4.13
Delta-M. We can see that (1) The BitFit modules in the Self-Attention modules and Cross-Attention modules in the higher decoder layers are highly preferred, proving that the BitFit modules are simple and effective. This observation is also beyond the intuition of human experts, as most previous work ignores the contribution of training or applying DT methods to last layers of the encoder and decoder are emphasized, which is close to the traditional use PTMs as feature extractors by training only the last layer; (3) We also observe that BitFit modules tend to be distributed approximately evenly across the higher layers (See Appendix D.2
modules; (2) The
(e2 for details). Figure shows the p; of Delta-LL The trend of choosing higher layers still exists. Interestingly. the query sub-modules are prioritized in the encoder, while the value sub-modules are stressed in the decoder.
4.9 Ablation Study
To explicitly control sparsity, we propose shifted global sigmoid, which differs from the Lo regulariza- tion used in previous work (23). We compare the results of regularization using shifted global sigmoid and Lo on three datasets. From Figure 4 is clear that shifted global sigmoid has an advantage over Lo regularization at almost all sparsity, and the advantage increases with increasing sparsity.
5 Conclusion
In this paper, we propose Sparse Structure Search for Delta Tuning Deetta, which conducts differentiable DT structure search with explicit sparsity control in a unified search space of a mixture of various DT modules. Experiments demonstrate the effectiveness of &elta find the optimal structure of DT modules and push the limit of trainable parameter reduction. For future works, there are open questions that are worth investigating.
(1) Better search spaces or better DT modules could be designed to further explore the potential of structure search. (2) The current NAS algorithms are not tailored for the scenario where a pre-trained backbone model exists. Therefore, more specialized search algorithms could be developed for DT structure search.

---

6 Acknowledgements
This work is supported by National Key R&D Program of China (No. 2020AAAA0005002,, Institute Guo Qiang at Tsinghua University, Beijing Academy of Artificial Intelligence (BAAI), International Innovation Center of Tsinghua University, Shanghai, China.
Shengding Hu proposed the idea and framework. Shengding Hu and Zhen Zhang designed the methods and experiments. Zhen Zhang conducted the experiments. Shengding Hu and Ning Ding wrote the paper. Zhiyuan Liu and Maosong Sun advised the project and participated in the discussion. Yadao Wang and Yasheng Wang participated in the discussion and provided computational resources.
References
[1] G Anandalingam and Terry L Friesz. Hierarchical optimization: An introduction. Annals of Operations Research, 1992.
[2] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Proceedings of NeurlPS, 2020.
[3] Han Cai, Ligeng Zhu, and Song Han. Proxylessnas: Direct neural architecture search on target task and hardware. In Proceedings of ICLR, 2019.
[4] Guanzheng Chen, Fangyu Liu, Zaiqiao Meng, and Shangsong Liang. Revisiting parameter- efficient tuning: Are we really there yet? ArXiv preprint, 2022.
[5] Xin Chen, Lingxi Xie, Jun Wu, and Qi Tian. Progressive differentiable architecture search: Bridging the depth gap between search and evaluation. In Proceedings of ICCV, 2019.
[6] Jacob Devlin, Ming-Weo Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding, In Proceedings of ACL, 2019.
[7] Ning Ding, Yujia Qin, Guang Yang. Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, et al. Delta tningg A comprehensive study of parameter efficient methods for pre-trained language models. ArXiv preprint, 2022.
[8] Jonathan Frankle, David J. Schwab, and Ari S. Morcos. Training batchnorm and only batchnorm: On the expressive power of random features in cnns. Proceedings of ICLR, 2021.
[9] Demi Guo, Alexander Rush, and Yoon Kim. Parameter-efficient transfer learning with diff' pruning. In Proceedings of ACL, 2021.
[10] Yunhui Guo, Honghui Shi, Abhishek Kumar, Kristen Grauman, Tajana Rosing, and Rogério Schmidt Feris. Spottune: Transfer learning through adaptive fine-tuning. In Pro- ceedings of CVPR, 2019.
[11] Yunhui Guo, Yandong Li, Liqiang Wang, and Tajana Rosing. Adafilter: Adaptive filter fine- tuning for deep transfer learning. In Proceedings of AAAI, 2020.
[12] Xu Han, Zhengyan Zhang, Ning Ding, Yuxian Gu, Xiao Liu, Yuqi Huo, Jiezhong Qiu, Liang Zhang, Wentao Han, Minlie Huang, Qin Jin, Yanyan Lan, Yang Liu, Zhiyuan Liu, Zhiwu Lu, Xipeng Qiu, Ruihua Song, Jie Tang, Ji-Rong Wen, Jinhui Yuan, Wayne Xin Zhao, and Jun Zhu. Pre-trained models: Past, present and future. Al Open, 2021.
[13] Junxian He, Chunting Zhou, Xuezhe Ma. Taylor Berg-Kirkpatrice and Graham Neubig. To- wards unified view of efffmeter-efffcient transfer learning. In Proceedings of ICLR, 2022.
[14] Ruidan He, Linlin Liu, Hai Ye, Qingyu Tan, Bosheng Ding, Liying Cheng, Jiawei Low, Lidong Bing, and Luo Si. On the effectiveness of adapter-based tuning for pretrained language model adaptation. In Proceedings of ACL, 2021.
10

---

[15] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly, Parameter-efficient transfer learning for NLP. In Proceedings of ICMI, 2019.
[16] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. 2021.
[17] Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. In Proceedings of ICLR, 2017.
[18] Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In Proceedings of EMNLP, 2021.
[19]
Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario Saško, Gunjan Chhablani, Bhavitvya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh, Canwen Xu, Nicolas Patry, Angelina McMillan-Major Philipp Schmid, Sylvain Gugger, Clément Delangue, Théo Matussière, Lysandre Debut, Stas Bekman, Pierric Cistac, Thibault Goehringer, Victor Mustar, François I Lagunas, Alexander Rush, and Thomas Wolf. Datasets: A community library for natural language processing. In Proceedings of EMNLP, 2021.
[20] Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation.
In Proceedings of ACL, 2021.
[21] Hanwen Liang, Shifeng Zhang, Jiacheng Sun, Xingqiu He, Weiran Huang, Kechen Zhuang, and Zhenguo Li. Darts+: Improved differentiable architecture search with early stopping. ArXiv preprint, 2019.
[22] Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: differentiable architecture search. In Proceedings of ICLR, 2019.
[23] Christos Louizos, Max Welling, and Diederik P. Kingma. Learning sparse neural networks through 1 0 regularization. In Proceedings ICLR, 2018.
[24] Chris J. Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: continuous relaxation of discrete random variables. In Proceedings of ICLR, 2017.
[25] Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder. low-rank hypercomplex adapter layers. 2021.
Compacter: Efficient
[26] Nafise Sadat Moosavi, Quentin Delfosse, Kristian Kersting, and Iryna Gurevych. Adaptable adapters. ArXiv preprint, 2022.
[27] Matthew E. Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In Proceedings of NAACL, 2018.
[28] Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. AdapterFusion: Non-destructive task composition for transfer learning. In Proceedings of ACL, 2021.
[29] Hieu Pham, Melody Guan, Barret Zoph, Quoe Le, and Jeff Dean. Efficient neural architecture search via parameters sharing. In Proceedings of ICML, 2018.
[30] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training, 2018.
[31] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang. Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 2020.
[32] Esteban Real, Alok Aggarwal, Yanping Huang, and Quoe Le. Regularized evolution for image classifier architecture search. In Proceedings of AAAI, 2019.
11

---

[33] Andreas Rücklé, Gregor Geigle, Max Glockner, Tilman Beck, Jonas Pfeiffer, Nils Reimers, and Iryna Gurevych. DropterDrop: On the efficiency of adapters in transformers. In Proceedings of EMNLP, 2021.
[34] Yi-Lin Sung, Jaemin Cho, and Mohit Bansal. Vl-adapter: Parameter-efficien transfer learning for and-languaae tasks. 2021.
[35] Nima Tajbakhsh, Jac Y Shin, Suryakanth R Gurudu, R Todd Hurst, Christopher Kendall, Michael B Gotway, and Jianming Liang. Convolutional neural networks for medical image analysis: Full training or fine tuning? IEEE transactions on medical imaging, 2016.
[36] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez. Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proceedings of NeurlPS, 2017.
[37] Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. Superglue: A stickier benchmark for general-purpont language understanding systems. In Proceedings of NeurlPS, 2019.
[38] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A multi-tassk benchmark and analysis platform for natural language understanding, In Proceedings of ICLR, 2019.
[39] Ziheng Wang, Jeremy Wohlwend, and Tao Lei. Structured pruning of large language models. In Proceedings of EMNLP, 2020.
[40] Bichen Wu, Xiaoliang Dai, Peizhao Zhang, Yanghan Wang, Fei Sun, Yiming Wu, Yuandong Tian, Peter Vajda, Yangqing Jia, and Kurt Keutzer. Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search. In Proceedings of CVPR, 2019.
[41] Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tie-Yan Liu, On layer normalization in the transformer architecture. In Proceedings of ICML, 2020.
[42] Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg, Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models. 2022.
[43] Mengjie Zhao, Tao Lin, Fei Mi, Martin Jaggi, and Hinrich Schütze. Masking efficient alternative to finetuning for pretrained language models. In Proceedings of EMNLP, 2020. 2020.
[44] Barret Zoph and Quoc V. Le. Neural architecture search with reinforcement learning. In Proceedings of ICLR, 2017.
[45] Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoe V. Le. Learning transferable architectures for scalable image recognition. In Proceedings of CVPR, 2018.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] In the conclusion section 5 we describe two future works which can also be seen as the limitations of our current work.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See
Appendix [E]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] Our theorectical results are based on Transformer Architecure and Delta Tuning, which are stated in Section 3.1 and Appendix A
12

---

(b) Did you include complete proofs of all theoretical results? [Yes] Yes, the set of proofs and illustrations are in Section 3.2 and Appendix
3. If you experiments...
(a) Did you include the code, data, and instructions needed reproduce the main ex- perimental results (either in the supplemental material or as URL)? [Yes] In the supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] In Section 4.1 and Appendix B
(c) Did you report error bars (e.g., with respect to the random seed after running exper- iments multiple times)? [Yes] We report the standard error of each method in the T
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] We include these information in Appendix B.3
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? datasets, which are listed Section 4.1
[Yes] We use existing
(b) Did you mention the license of the assets? [Yes] In Section 4.1 we use the dataset
from Huggingface Library [19].
(c) Did you include any assets either in the supplemental material We add our own code and model to supplemental material. We do not generate new data
URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A] The data is publicly available.
(c) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
13


Reviews:
[{"review_id": "dvTWcc5jY0_", "reviewer": "Reviewer_mV46", "summary": "This paper proposes to design an automatic scheme to search the PET models for parameter-efficient finetuning.  Extensive experiments are given, which demonstrates that the auto-searched method surpasses the manual and random ones with less trainable parameters.\n", "questions": "See weakness", "limitations": "yes", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n---The ideal of this work is interesting, while combines the architecture search and PET models tuning.\nThe given experiments compared with seveal pior arts, which show the superior of the this proposes method.\n\nWeakness\n-- Although this work claims that the trainable parameters are reduced obviously, the runtime of the finetuning should be reported to further verify the effectness of this work. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "VnQq8vw1jf0", "reviewer": "Reviewer_qZC4", "summary": "The paper presents S3PET an approach to automatically search optimal trainable parameter efficient structure within language models. The proposed approach consists of three main components: A unified search space with probabilistic gating, a differentiable parameter efficient training search algorithm and an explicit sparsity control algorithm. Results on GLUE and SuperGLUE benchmarks show competitive performance with SOTA while using less number of trainable parameters. ", "questions": "The authors do not compare with Prompt Tuning baselines and they mention that it takes much longer to converge [L208].\nIf there are some initial experiments about this, it would be good to report numbers on how long it would actually take to converge. ", "limitations": "The authors did not provide any explicit discussions on limitations. For example in terms of efficiency of training, the approach seems to be much slower than pre-defined sparse structures.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Unlike predefined trainable substructures in previous works, the proposed approach searches the optimal trainable structure and is guided by downstream performance.\n- One important advantage of the approach seems to be the strong results even when the number of training parameters is small (~0.01%).\nThis is significantly less than prior methods. \n\nWeaknesses:\n- The experiments only focus on GLUE/SuperGLUE classification tasks. There are no results on generation tasks like translation or summarization. \n- It would have been interesting to see some analysis on training data requirements for the proposed approach. E.g., would it still work if there are limited data. \n- The only pretrained model experimented with is T5-large. Performance under larger models or decoder-only models is unexplored.\n- It would have been interesting to see some discussion on how expensive the training of the proposed approach is. There is some discussion in the appendix but I would encourage authors to move those to main text. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "F60naYfFKIT", "reviewer": "Reviewer_LCyP", "summary": "This paper develops a neural architecture search method for the adapter module (or called PET modules) in parameter-efficient tuning schemes.  The authors construct a unified search space for parameter-efficient tuning, and adopted binary differentiable NAS method for architecture search. A shifted global sigmoid is developed to control the sparsity of the PET structure. The searched PET structure achieves competitive results with about 1/5 trainable parameters compared to previous manually designed methods.", "questions": "1. How does PET methods save model adaptation cost? Optimizing an PET module with gradient decent requires all the subsequent network layers to calculate their gradient in the backward pass. All the parameters (after the first PET module) still need to compute and save gradient. Thus, the computational cost and memory cost is actually not reduced.\n\n\n2. Minor suggestions:\n\n   (1) It would be better to use \\eqref in latex for reference of equations (add parentheses for equation numbers).\n\n   (2) Avoid using \"Moreover\" constantly in consecutive sentences (in Line 66-68). Some words like \"Firstly... Secondly...\" might be better.", "limitations": "Yes.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. This paper is generally in good form and easy to follow. Figures and tables are professional.\n\n2. The studied problem, automatic design of PET structure, is novel and interesting.\n\n3. The unified search space may be useful for future research; the proposed Shifted Global Sigmoid for sparsity control is sound and effective.\n\n4. The experimental evaluation is extensive, covering transferability of the searched structure, compare of search space, compare of sparsity method, etc. \n\nWeakness:\n1.  My main concern is the contradiction between the search cost and the application scenario of S3PET. The motivation of PET method is to reduce the model adaptation cost. However, performing the proposed PET search would cost 5 to 8 times training time (as in line 263), which may contradict the motivation of PET methods. I understand the authors claim of the search cost is comparable to (or cheaper than) manual design. However, similar effort is needed for the proposed method, e.g. on search space design or hyper-parameter tuning. Overall, this issue of search cost could severely limit the application scenarios and the significance of S3PET. \n\n2. Some important literatures are not introduced. (1) The proposed method is highly relevant to differentiable NAS methods with discrete binary gate, e.g. ProxylessNAS [*1], FBNet [*2], etc. In DARTS, the options of gate are activated together in each step, while in these methods, only one option is activated and the structure parameters are optimized by replacing the hard option with soft approximation in backward calculation, same to this paper. (2) Some automatic model adaptation method should be introduced: [*3] [*4].\n\n\n[*1] Han Cai, et al. ProxylessNAS: Direct Neural Architecture Search On Target Task And Hardware. ICLR, 2019.\n\n[*2] Bichen Wu, et al. FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Search. CVPR, 2019.\n\n[*3] Guo, Y., et al. Spottune: transfer learning through adaptive fine-tuning. CVPR, 2019.\n\n[*4] Guo, Y., et al. Adafilter: Adaptive filter fine-tuning for deep transfer learning. AAAI, 2020.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "3D_Ii5N_y5y", "reviewer": "Reviewer_SumM", "summary": "This paper explores constructing PET modules in an automatic manner, and S 3PET with NAS is proposed, which conducts the differentiable PET structure search through bi-level optimization and proposes shifted global sigmoid method to explicitly control the number of trainable parameters. Extensive experiments show superior performance with low trainable parameters budgets. ", "questions": "As listed in Weaknesses, the following questions need to be solved. \n\n(1)\tEvaluation details and oracle.\n\n(2)\tExperiments details and more explanations. \n", "limitations": "There is not serious negative societal impact of this work currently. ", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n\n(1)\tAutomated search for parameter-efficient tuning.\n\n(2)\tState-of-the-art performance with low trainable parameters. \n\nCons:\n\nAlthough the authors employ NAS for parameter-efficient tuning, some descriptions are not clear. \n\n(1)\tThe authors state that they choose the set of positions where the sum of p_i. However, how to determine the number of positions? If all positions are used, does the performance improve? It shows the oracle.\n\n(2)\tIn Table1, the authors show the results on GLUE and SuperGLUE. However, which dataset is employed for NAS? Is it searched on GLUE and SuperGLUE separately? If so, I still think it lacks generalization ability. Adding NAS step is also time-consuming, the authors should also report the time cost of NAS compared to the training step. In Table2, it seems that the performance improvement is not significant. It is obvious that NAS can improve performance, so it lacks insight. In Table 3, the authors conduct the transferability experiments, but it is not discussed why such a method can guarantee transferability. Maybe it is because the dataset for searching must be very large. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept