Bi-Drop: Enhancing Fine-tuning Generalization via Synchronous sub-net Estimation and Optimization
Authors: Shoujie Tong
Heming Xia
Damai Dai
Runxin Xu
Tianyu Liu
Binghuai Lin
Yunbo Cao
Zhifang Sui
Venue: EMNLP 2023 Findings (2023)
Keywords: pretrained language model
adaptive sub-net Optimization

Abstract: Pretrained language models have achieved remarkable success in natural language understanding. However, fine-tuning pretrained models on limited training data tends to overfit and thus diminish performance. This paper presents Bi-Drop, a fine-tuning strategy that selectively updates model parameters using gradients from various sub-nets dynamically generated by dropout. The sub-net estimation of Bi-Drop is performed in an in-batch manner, so it overcomes the problem of hysteresis in sub-net updating, which is possessed by previous methods that perform asynchronous sub-net estimation. Also, Bi-Drop needs only one mini-batch to estimate the sub-net so it achieves higher utility of training data. Experiments on the GLUE benchmark demonstrate that Bi-Drop consistently outperforms previous fine-tuning methods. Furthermore, empirical results also show that Bi-Drop exhibits excellent generalization ability and robustness for domain transfer, data imbalance, and low-resource scenarios.

Bi-Drop: Enhancing Fine-tuning Generalization via Synchronous sub-net
Estimation and Optimization
Shoujie Tong! Heming Xia
Damai Dail.4
Runxin Xu
Tianyu Liu
Binghuai Lin Yunbo Cao Zhifang Sui
National Key Laboratory for Multimedia Information Processing, Peking University
2 School of Software & Microelectronicc, Peking University Allibaba Group
4 School of Computer Science, Peking University Tencent Cloud Al
edu. cn {xiaheming daidamai tianyu0421, sekk. edu. {runxinxu)@gmaiil, com (binghuailin, yunbocao (ebboccao)ttencentt. com
Abstract
Provituss
Pretrained language models have achieved re- markable success in natural language under- standing. However, fine-tuning pretrained mod- els limited training data tends overfit and thus diminish performance. This paper presents Bi- Drop, a fine-tum tine-tuning strategy that se- lectively updates model parameters using gradi- ents from various sub-nets dynamically gener- ated by dropout. The sub-net estimation of Bi- Drop is performed in in-batch manner, overcomes the problem of hysteresis in sub-net updating, which is possessed by previous meth- ods that perform asynchronous sub-net estima- tion. Also, Bi-Drop needs only one mini-batch estimate the sub-net achieves higher util- ity of training data. Experiments on the GLUE benchmark demonstrate that Bi-Drop consis- tently outperforms previous fine-tuning meth- ods. Furthermore, empirical results also show Bi-Drop exhibits excellent generalization ability and robustness for domain transfer, data imbalance, and low-resource scenarios.
Methods
Compatee Gracken Naak
Zelemtety Sysber Peronzior
Bi-Dmpp
Ebta
Herward Nuttoe Despatt
Figure 1: Comparisons between Bi-Drop and previous methods. Unlike previous methods that require multiple mini-batches of data to asynchronously determine the sub-net to optimize, Bi-Drop has a synchronous sub-net selection strategy with higher data utility.
et al., 2021; Wu et al., 2021; Xu et al., 2021; Yuan et al., 2022), such as sub-net optimization strate- gies like Child-Tuning (Xu et al.. 2021) and DPS (Zhang et al., 2022). Child-TuningJD selects static sub-net for updating based parameter impor- tance estimated by Fisher Information (FI). As an improved variant of Child-Tuninge DPS dy- namically decides the sub-net to be updated by estimating FI with multiple mini-batches of data. Although these Fl-based methods achieve better generalization ability than vanilla fine-tuning, they still have two limitations: (1) hysteresis in sub-net updating: the sub-net preference is estimated with the model parameters in previous iterations and may be incompatible with the current update step; and (2) insufficient utility of training data: estimation requires cumulative gradients through multiple nini-batches, so these methods cannot fit in situations with data scarcity.
1 Introduction
recent years, Natural Language Processing (NLP) has achieved significant progress due to the emergence of large-scale Pretrained Language Models (PLMs) (Devlin et al., 2018; Liu et al., 2019; Raffel et al., 2019; Clark et al., 2020). For downstream tasks, compared with training from scratch, fine-tuning pretrained models can usually achieve efficient adaptation and result in better per- formance. Despite the great success, fine-tuning methods still face challenges in maintaining gener- alization performance on downstream tasks they tend to run into the overfitting issue when the train- ing data is limited (Phang et al., 2018; Devlin et al.. 2018; Lee al., 2020).
In this paper, we delve deeper into adaptive sub- net optimization strategies and propose Bi-Drop, Fl-free strategy for fine-tuning pretrained lan- guage models. Unlike Fisher information estima- tion, which requires cumulative gradients of mini- batches, Bi-Drop only relies on information in a single mini-batch to select the parameters to update. Specifically, Bi-Drop utilizes gradient information
To improve the generalization ability of fine- tuning methods, many regularization techniques have been proposed (Chen et al., 2020; Aghajanyan
"quul contributions

---

from different sub-nets dynamically generated by dropout in each mini-batch. As illustrated in Figure I, within a single training step of Bi-Drop, a mini- batch will go through the forward pass multiple times and, due to the randomness introduced by dropout, yield various distinct sub-nets. We then apply a parameter selection algorithm with pertur- bation and scaling factors to stabilize the gradient updates. With this synchronous parameter selec- tion strategy. Bi-Drop can selectively update model parameters according to the information from only the current mini-batch, and thus mitigate overfitting with a high utility of training data.
Sub-net Cranularity
Hyste- resis
Data
Models
Depend.
Child-TuningD DPS
task-wise
Yes
Yes
cycle-wise
Yes
Yes
Bi-Drop
step-wise
No
No
Table Comparison of Bi-Drop with prior FI-based sub-net optimization methods. "cycle-wise" refers to the granularity that includes multiple mini-batches of data.
et al., 2018) sparked a continuous emergence of various pre-trained models, including RoBERTa (Liu et al., 2019), ELECTRA (Clark et al., 2020), XLNet (Yang et al., 2019), GPT-2 (Radford et al.. 2019), and GPT-3 (Brown al., 2020), which have brought remarkable improvements in model struc- tures and scales. Until now, fine-tuning still one of the most popular approaches to adapting large pretrained language models to downstream tasks.
Extensive experiments on the GLUE benchmark demonstrate that Bi-Drop shows remarkable superi- ority over the state-of-the-an art fine-tuning regulariza- tion methods, with a considerable margin of 0.53 1.50 average score. Moreover, Bi-Drop con- sistently outperforms vanilla fine-tuning by 0.83 ~ 1.58 average score across various PLMs. Fur- ther analysis indicates that Bi-Drop attains superb generalization ability for domain transfer and task transfer, and is robust for data imbalance and low- resource scenarios.
Regularization Methods for Fine-tuning Large- scale PLMs are prone to over-fitting (Phang et al.. 2018; Devlin et al., 2018) and exhibit inadequate generalization ability when fine-tuned with limited training data (Aghajanyan et al., 2021; Mahabadi et al., 2021), resulting in degraded performance. To tackle this issue, various regularization techniques have been suggested to enhance the generalization capacity models, including advanced dropout alternatives (Wan et al., 2013; Wu et al., 2021), ap- plying adversarial perturbations (Aghajanyan et al., 2021; Wu et al., 2022; Yuan et al., 2022) and con- strained regularization methods (DauméllI, 2007; Chen et al., 2020). In recent years, Child-tuning (Xu al., 2021) and DPS (Zhang al., 2022) pro- pose to estimate parameter importance based on Fisher Information (FI) and selectively optimize a sub-net during fine-tuning to mitigate overfitting.
To sum up, our contributions are three-fold:
We propose Bi-Drop, a step-wise sub-net op- timization strategy that adaptively selects the updated sub-net based on the current mini- batch data. Compared with prior Fl-based methods, Bi-Drop derives more stable and robust training trajectory with simultaneous sub-net update and high utility of training data.'
With extensive experiments on various PLMs, we demonstrate that Bi-Drop achieves con- sistent and remarkable superiority over prior fine-tuning regularization methods.
Further analysis shows that Bi-drop attains superb generalization ability for domain trans- fer and task transfer, and is robust for data imbalance and low-resource scenarios.
FI-based methods have a strong dependence on the training data and exhibit hysteresis in sub-net updating. As shown in Table 1, compared with prior FI-based methods, Bi-Drop introduces step- wise sub-net optimization strategy that adaptively selects the sub-net to be updated based on the cur- rent mini-batch. is worth noting that, model- agnostic technique, Bi-Drop is orthogonal most previous fine-tunin methods, which could further boost the model's performance.
2 Related Work
Pretrained Language Models In recent years, the field of natural language processing (NLP) has witnessed significant advancements due to the de- velopment of large-scale pretrained language mod- els (PLMs). The introduction of BERT (Devlin
'Our cnde is available at tongshoujie/Bi-Do

---

Average Gradients
neuron units
dropped units
Carain
80;
Backocard Gradients ofaubnntt
Backward Gradients of
Gradient
Masked Average Gradients
de2
(1) Forward Multiple Times with Dropout
(2) Colculate the Gradients Mask
(3) Selectively Update Parameters
Figure 2: An overall illustration of Bi-Drop. Bi-Drop splits each training step into three sub-steps: (1) Multiple forward propagations: each mini-batch sample goes through the forward pass multiple times (denoted dropout; (2) sub-net selection: an advanced strategy is adopted to select the sub-net net to be updated based on the gradients of distinct sub-nets generated by dropout; (3) Parameter updating: only the parameters of the selected sub-net updated mitigate overfitting.
with
3 Methodology
(e.) McTo 80.
3.1 Background
0.+1 0.
We first introduce the paradigm of sub-net opti- mization by giving general formulations of the backpropagation during the vanilla fine-tuning and CHILD-TUNING.- We denote the parameters of the model at (-th iteration as 0, - (0 where d.. represent the i-th element of 0g at the t-th training iteration. Op denotes the parameter matrix of the pre-trained model. The vanilla fine-tuning adopts Stochastic Gradient Descent (SGD) to all the model parameters, formally:
3.2 Bi-Drop
As introduced in Section 3.1, CHILD-TUNING only optimizes an unchanged sub-net during fine- tuming and ignores the update of other parameters, which may degrade the model's performance downstream tasks. In this section, we offer de- tailed introduction our proposed method, Bi- Drop. which selects the updated parameters adap- tively at each fine-tuning step. Specifically, Bi- Drop splits each training step into three sub-steps: (1) multiple forward propagations, (2) sub-net se- lection, and (3) parameter updating, We provide pseudo-code of Bi- Bi-Drop in Algorithm
0 (0:)
(1)
where L represents the training loss within a batch; 1) is the learning rate. Instead of fine-tuning the entire network, CHILD-TUNING, proposes to only optimize subset of parameters (i.e., the sub-net). It first adopts the Fisher Information (FT) to esti- mate the relative importance of the parameters for specific downstream task, which can be formulated as:
3.2.1 Multiple Forward Propagations
Instead of prior Fl-based methods that require accu- mulated gradients to measure the parameter impor- tance, Bi-Drop leverages distinct sub-nets gener- ated by dropout to select the sub-net to be updated. Inspired by Wu et al. (2021), given the training data D = {(I,. 3)) ) at each training step, we feed 2; to the model multiple times in the forward pass with different dropouts, and obtain their gradients correspondingly:
[D]
(2 (9o)
F(O,)= = Σ 1-1
(2)
00g
McTD F(Oo) > sort (F(O
where D is the training data, F(Og) denotes the fisher information matrix of the pretrained param- eters; s-rt()), represents the highest value of p percentile in F(O,) after sorting in ascending or- der; MCTD is a mask matrix that is the same-sized as 0g. During fine-tuning, CHILD- CHILD-TUNINGD only optimizes the selected sub-net in MeTs:
OC
1, 1,2,..., ..... k
(5)
where 0.00 and g represents the parameters of the j-th forward pass and its corresponding gradients. denotes the number forward passes, i.c., the number of distinct sub-nets with different dropouts.
(

---

Algorithm Bi-Droo for Adam Optimizer
consistently larger gradients and smaller variances are more favorable by this factor.
Require: 0o: initial pretrained weights; (()): stochastic objective function with parameters 0: 2,,3,,8 32 E [0, 1): exponential decay rates for the moment estimates; ry: learning rate; I: initialize timestep <- O, first moment vector
Scaling Factor We further propose the scaling factor as regularization term. This factor mea- sures the ratio of the average parameter gradients the original parameters. Parameters whose gradient scale is much smaller than the original parameters will not be updated, which is similar in spirit to gradient clipping.
mo + O, second moment vector t-
2: while not converged do
3: ++
11 Multiple forward propagations
4: g: ())
j = 1, ...,.
86
F Fsca (0,)
|4 00l
(8)
1/ Sub-net selection
5: M, + SelectSubNetwork(g))
3.2.3 Parameter Updating
1/ Gradients Updating
6: ge 8t ② M,
Following prior work (Xu al., 2021; Zhang al., 2022), we derive a step-wise mask matrix M, fil- tered by selecting the highest value of p percentile measured by the aforementioned two estimation factors.
7:
m, 31-m; 1m-1111 + (1- B1) gt
8:
Vi s B2 B2
82) gr
9:
m + m 111--99 B4)
10:
Vi < vs/(1 - (32)
11 Update weights
11:
<- W1-1 ma/ /c/ 0h +()
F Final (0.) Fper((:) ) (Oc)(O:)
(9)
12: end while
13: return w,
M. F Final (0:) > sort (Fiina(O)))) (0.))),
(10)
Then, we utilize M, update the sub-net which consists of important parameters at each training step. We denote the formulation by simply replac- ing Eq.4 with our step-wise mask matrix Mc:
3.2.2
Sub-net Selection
this subsection, we introduce our sub-net se- lection strategy, which estimates the relevant im- portance of parameters based on the gradients of distinct sub-nets generated by different dropouts. Concretely, our strategy is based on two estima- tion factors: the perturbation factor and the scaling factor.
O.t1 = O. mpc M.
(11)
Experiments
Perturbation Factor We propose the perturba- tion factor, which estimates the importance of pa- rameters according to their stability with different dropouts in the forward pass. We point out that var- ious sub-nets generated by dropout can viewed as adversarial perturbations to vanilla model. The perturbation factor is formalized as follows:
4.1 Datasets
GLUE Benchmark Following previous work (Lee et al., 2020; Dodge et al., 2020; Zhang et al.. 2021), conduct extensive experiments on the GLUE benchmark, including natural language in- ference (RTE, QNLI, MNLI), paraphrase and simi- larity (MRPC, STS-B, QQP), linguistic acceptabil- ity (CoLA), and sentiment classification (SST-2).
8
(2)
(6) We include the detailed statistics and metrics of the datasets in Appendix A. Since the test results are only accessible online with limited submission times, follow prior studies (Phang et al., 2018; Lee et al., 2020; Aghajanyan et al., 2021; Dodge et al., 2020: Xu al., 2021; Zhang et al., 2022) that fine-tune the pretrained model on the training set and report the results on the development sets using the last checkpoint.
-
Fper((.) (.:)= |pcl
Σ
(5)
m.)
(7)
where 14 is the average gradients of parameters. Fprr measures the stability of parameters by both considering the mean and variance of gradients with adversarial perturbations, iee. sub-nets with

---

Method
CoLA
MRPC
RTE
STS-B
Avg
Vanilla
63.76(65.55)
90.41(91.91)
70.97(74.01)
89.70(90.58)
78.71(80.51)
0.00
Weight Decay
63.70(65.53)
90.89(91.98)
72.24(74.37)
89.66(90.22)
79.12(80.53)
+0.41(0.02)
Top-K Tuning
63.89(65.377
91.09(92.05)
72.84(75.17)
89.64(90.83)
79.37(80.86)
+0.66(0.35)
Mixout
64.05(65.41)
91.19(22.05)
72.32(75.52)
89,89(90.42)
79.36(80.85)
+0.65(0.34)
RecAdam
64.05(65.35)
90.88(91.88 89)
72.48(74.30)
89.6700.62)
79,28(80.54)
+0.57(0.03)
ChildTuningo
64
91.19(9..20)
72.35(76.53)
90.18(90.88)
(9.49)))336
+0.78(0.85)
R3F
64. 64.88366.33)
91.55(92.28)
72.42(74.37)
89,70(0..58)
79.64(80.89)
+0.9300338)
DPS
91.05(92.17)
72.88(75.81)
90,47(91. .2)
79.70(81.30)
+0.99(0.79)
R-Drop
(4687)66.79)
91.76(92.17)
73.12(76.53)
90.14(90.42)
79.96(81
+1.25(0.97)
Bi-Drop
64.94(66.69)
91.79(92.68)
73.79(77.61)
90.59(91.04)
80.26(82.01)
+1.55(1.50)
Table 2: Comparison between Bi- Bi-Drop with prior fine-tuning methods. We report the mean (max) results of 10 random seeds. The best results are bold. Note that since R3F is not applicable to regression, the result on STS-B (marked with ") remains the same as vanilla. Bi-Drop achieves the best performance compared with other methods.
NLI Datasets We also evaluate the generaliza- tion ability of Bi-Drop on several Natural Language Inference (NLI) tasks, including SNLI (Bowman et al., 2015), MNLI (Williams et al., 2018), MNLI- M (Williams et al., 2018) and SICK (Marelli et al.. 2014). We report all results by Accuracy on the development sets consistent with GLUE
4.3 Experiments Setup
We conduct experiments based on the Hugging- Face transformers library (Wolf al., 2020) and follow the default hyper-parameters and settings unless noted otherwise. We report the averaged results over 10 random seeds. Other detailed exper- imental setups are presented in Appendix B.
4.4 Results on GLUE
4.2 Baselines
Comparison with Prior Methods We compare Bi-Drop with various prior fine-tuning methods based on BERTaarge and report the mean (and max) scores GLUE benchmark in Table 2, following Lee et al. (2020) and Xu et al. (2021). The re- sults indicate that Bi-Drop yields the best average performance across all tasks, showing its effec- tiveness. Moreover, the average of the maximum scores attained by Bi-Drop is superior to that of other methods, providing further evidence of the effectiveness of Bi-Drop. We also conducted the same experiment Roberta and the details can be found in Appendix E.
Besides the vanilla fine-tuning method, we mainly compare Bi-Drop with the following baselines: Mixout (Lee et al., 2020) is a fine-tuning tech- nique that stochastically replaces the parameters with their pretrained weight based on the Bernoulli distribution. R3F (Aghajanyan et al., 2021) is fine-tuning strategy motivated by trust-region the- ory, which injects noise sampled from either a nor- mal or uniform distribution into the pre-trained rep- resentations. R-Drop (Wu et al., 2021) minimizes the bidirectional KL-divergence to force the output distributions of two sub-nets sampled by dropout to be consistent with each other. Child-Tuningo (Xu et al., 2021) selects the task-relevant parame- ters as the sub-net based the Fisher information and only updates the sub-net during fine-tuning. DPS (Zhang et al., 2022) is a dynamic sub-net op- timization algorithm based on Child-TuningD.. estimates Fisher information with multiple mini- batches of data and selects the sub-net adaptively during fine-tuning.
Comparison with Vanilla Fine-tuning We show the experimental results of six widely used large- scale PLMs on the GLUE Benchmark in Table 3. The results show that Bi-Drop outperforms vanilla fine-tuning consistently and significantly across all tasks performed on various PLMs. For instance, Bi-Drop achieves improvement of up to 1.58 av- erage score on BERT base and 1.35 average score on Robertabase- The results highlight the universal ef- feetiveness of Bi-Drop in enhancing the fine-tuning performance of PLMs. Additionally, because Bi- Drop lowward-popageer twice, we present an ad- ditional study of the baseline with doubled batch
For reference, we also show other prior fine- tuning techniques in our main experimental results, such as Weight Decay (DauméllI, 2007), Top-K Tuning (Houlsby et al., 2019) and RecAdam (Chen et al., 2020).
2https: thub. hwwp:h/hgiinnaaaaeerrraafrrroraa

---

BERT base
Robertaunse
Method
CoLA
MRPC
RTE
STS-B
CoLA
MRPC
RTE
STS-B
Avg
Vanilla Bi-Drop
57.67
90.38
69.57
89.35
76.74
59,45
91.94 92.40
76.28
90.60
79.57
60.76
91.01
71.73
89.78
78.32
61.28
78.99
91.00
80.92
BERT
Robertatage
Method
CoLA
MRPC
RTE
STS-B B
Avg
CoLA
MRPC
RTE
STS-B
Avg
Vanilla Bi-Drop
63.76 64.94
90.41 91.79
70.97 73.79
89.70 90.50
78.71 80.26
66.01 68.03
92.56 92.95
84.51
92.05
83.78
86.10
92.36
84.86
DeBERTatage
ELECTRAuag
Method
CoLA
MRPC
RTE
STS-B
Avg
CoLA
MRPC
RTE
STS-B
Avg
Vanilla 65.18
92.32
85,60
91.6 91.64
83.69
70.02
92,94
88.37
91.91
85.81
Bi-Drop
66.91
92.88
86.72
92.33
84.71
71.29
93.68
88.98
92.61
86.64
Table 3: Comparison between Bi- Bi-Drop and vanilla tinettunin applied six widely-used large-scale PLMs. We report the mean results of 10 random seeds. Average scores all tasks underlined. The best results are bold. It shows that Bi-Drop yields consistent improvements across all tasks among different PLMs.
SNLI
MNLI
Datasets
Vanilla
CTp
R- R-Drop
DPS
Bi-Drop
Vanilla
CTp
R- R-Drop
DPS
Bi-Drop
MNLI
64.53
64.16
64.51
64.70
66.88
75.78
76.27
76.85
75.48
76.63
MNI.I-m
66.11
66.30
66.59
67.29
68.49
77.31
77.30
77.93
77.45
77.98
SNLI
83.37
83.53
83.59
82.95
83.62
70.87
70.76
71.48
71.56
71.63
SICK
52.59
53.73
53.59
55.89
54.28
53.27
55.35
54.21
54.81
54.93
Avg
61.08
61.40
61.56
62.63
63.22
67.15
67.80
67.87
67.94
68.18
Aueg
t 0.32
t 0.48
1.55
f 2.14
1 0.65
t 0.72
0.79
f 1.03
Table 4: Evaluation for out-of-domain generalization The models are trained on MNLI/SNLI and tested on out-of-domain data, Average scores computed excluding in-domain results (underlined). The best results are bold. Bi-Drop can better maintain the out-of-domain generalization ability of the model.
size in Appendix D.
indicate that Bi- Bi-Drop encourages the model learn deeper and more generic semantic features and al- leviate superficial biases of specific tasks, which improves the model's generalization ability.
4.5
Out-of-Dom Generalization
We further evaluate the generalization ability of Bi-Drop on a widely used experimental setting in prior research (Aghajanyan et al., 2021; Xu et al.. 2021; Zhang et al., 2022): out-of-domain gener- alization. In detail, we finetune BERT large with different strategies 5k subsampled MNLI and SNLI datasets respectively, and directly evaluate the fine-tuned model on other NLI datasets. The ex- perimental results Table 4 illustrate that Bi-Drop outperforms vanilla fine-tuning and prior methods across various datasets. Specifically,, compared with other fine-tuning methods on SNLI, Bi-Drop demonstrates consistent and substantial improve- ment, with average score increase of 2.. In particular, Bi-Drop achieves improvement of 2.35 on MNLI task and 2.38 on MNLI-m task. For models trained on MNLI, Bi-Drop also consistently outperforms prior methods, with an average im- provement of 1.03 score. The experimental results
4.6 Task Generalization
We also evaluate the generalization ability of fine- tuned models following the experimental setting of Aghajanyan et al. (2021) and Xu et al. (2021), which freezes the representations of the model fine- tuned on one task and only trains the linear clas- sifier on the other task. Specifically, we finetune BERT large among one task selected among MRPC, CoLA, and RTE and then transfer the model to the other two tasks. Figure 3 shows that Bi-Drop consistently outperforms vanilla fine-tuning when the fine-tuned model is transferred to other tasks. In particular, Bi-Drop improves by 3.50 and 3.28, when models trained on MRPC and RTE respec- tively are evaluated on CoLAA The results fur- ther verify the conclusion that Bi-Drop helps mod- els learn more generalizable representations, com-

---

Vanilla Bi-Drop
58.24
88.35 89.35
89
35
67.16
57
Eval on CoLA
Eval on RTE
Eval MRPC
86
30
29.19
53
83
$1.74
81.51
20
80
Train
on CoLA Train
on on CoLA
Figure 3: Evaluation for task generalization. The model is fine-tuned on specific task among MRPC, CoLA, RTE and transferred to the other two tasks. Bi-Drop can be more generalizable.
Vanila
R3F
CHLD-TUNIKGU
Drop
Drpp
0.93
0.68
0.92
0.92
0.66
Moe
0.64
0.90
0.90
0.62
0.62
0.89
0.60
0.60
(8): MRPC (r-3e-5)
(b): MRPO (Im4e-5)
(o): (OLA((-2--5)
(d): (0-A30)=3-55)
Figure 4: Results of Bi-Drop across four experimental settings. Each method includes a violin plot for random runs. Compared with other methods, the shorter and thicker violin plot of Bi-Drop proves its better stability.
pared with the vanilla fine-tuning approach.
investigate the robustness of Bi-Drop to label noise, we conduct simple simulation experiments on RTE, MRPC, and CoLA by randomly corrupting pre- determined fraction of labels with erroneous val- ues. We evaluate the robustness of various fine- tuning methods trained on noisy data. The results shown in the left panel of Table demonstrate that Bi-Drop achieves consistent superiority other fine-tuning methods on noisy data. Furthermore, we conducted a comparison of model performance degradation across varying noise ratios. We calcu- lated the degradation and presented it in brackets, as compared with Table 2. It indicates that Bi- Drop has the smallest performance drop compared with other fine-tuni sunittuning methods. These results col- lectively demonstrate that Bi-Drop is more robust to label noise than prior methods.
5 Analysis
5.1 Stability to Random Seeds
We further investigate the stability properties of fine-tuned models. Figure shows the output dis- tributions models with four experimental settings and across 10 random seeds. The results demon- strate that Bi-Drop outperforms other strategies in terms of average performance, and also exhibits greater stability by achieving more consistent re- sults across 10 random seeds with lower variance.
5.2 Robustness Analysis
Recent research has brought to light that the vanilla fine-tuning approach is prone deception and vul- nerability in many aspects. In this study, assess the robustness of Bi-Drop by designing evaluation tasks that focus on two common scenarios, aiming to examine its ability withstand various forms of perturbations while maintaining its robustness.
Robustness to Data Imbalance Minority class refers to the class that owns insufficient instances in the training set. In this section, we strive to explore the robustness of diverse fine-tuning approaches for the minority class by carrying experiments synthetic RTE. MRPC, and CoLA datasets. The ex- perimental results are illustrated the right panel of Table 5, which shows that Bi-Drop significantly outperforms other fine-tuning methods. Bi-Drop
Robustness to Label Noise Due the inherent limitations of human annotation, widely-used large- scale datasets inevitably contain a certain amount incorrect labels (Vasudevan et al., 2022). To

---

Method
CoLA
MRPC
RTE
Avg
Method
CoLA
MRPC
RTE
Avg
Noise Ratio:
5%
Reduction Ratio: 70%
vanilla
61.39
90.24
69.78
73.80 (J 1.25)
Vanilla
81.12
78.88
27.71
62.57 (
CTp
61.66
90.39
71.47
74.51
() 1.77)
CTр
83.10
79.50
28.14
63.58 (1 1.01)
R-Drop
62.33
91.21
73.16
75.57
(J 1.00)
R-Drop
81.37
76.46
32.61
63,48 (1 0.91)
Bi-Drop
62.29
91.85
73.57
75.90
(1 0.94)
Bi-Drop
81.88
80.17
37.66
66.57 (1 4.00)
Noise Ratio:
10%
Reduction Ratio: 60%
vanilla
59.35
89.78
68.51
72.55
2.50)
Vanilla
83,30
83.98
39.32
68.87 (
CTp
61.56
89.88
69.78
73.72
(L) 2.56)
CTp
85.78
84.55
37.65
69.33 (1 0.46)
R-Drop
61.12
91.10
71.36
74.53
(L 2.04)
R-Drop
83.55
85.07
43.51
70.71 (f 1.84)
Bi-Drop
61.65
91.12
71.96
74.91
( 1,93)
Bi-Drop
84.33
85.35
49.62
73.10 (1 4.23)
Noise Ratio: 15%
Reduction Ratio: 50%
vanilla 58.96
88.10
68,
71.73
3.32)
Vanilla
86.07
87.40
45.27
72.91
CTp 59.99
88.10
68.88
72.32
3.96)
CTp
87.98
87.64
50.78
75.47
2.56)
R-Drop 60.80
89.26
69.55
73.20
(J 3.36)
R-Drop
86.04
88.89
55.73
76.89
3,98)
Bi-Drop
60.08
89.73
70.88
73.56
3.28)
Bi-Drop 86.73
89.61
58.27
78.20 5.29)
Table Left: Robustness to label noise. noise ratio is the percentage training instances whose labels are transferred to incorrect labels. Right: Robustness to data imbalance. We reduce the number of instances labeled by 70%/60%/55%% in the training set and test the accuracy of instances labeled I (as the minority class) in the validation set. Bi-Drop can maintain more robust representations compared with other fine-tuni tine-tuning methods.
IK
Datasets
Vanilla
CTp
DPS
Bi-Drop
Vanilla
CTp
DPS
Bi-Drop
CoLA
36.23
40.77
41.87
44.78
48.92
51.63
52.89
54.76
MRPC
81.33
81.95
83.29
83.18
83.90
84,62
85.03
85.44
RTE
58.87
59.27
59.16
60.04
62.17
63.85
64.92
66.96
STS-B
82.60
82.79
83.41
85.19
85.91
86.92
87.26
87.53
SST-2
86.
88.19
89.08
89.38
89.95
89.88
90.44
90.41
QNLI
78.76
79.32
79.27
80.48
82.49
83.65
83.86
84.35
QQP
71.88
73.43
74,22
74.61
77.65
78.57
78.79
79.01
MNLI
46.44
45.74
47.63
50.79
56.55
60.96
59.47
61.25
Avg 67.78
68.93
69.74
71.06
73.44
75.01
75.33
76.21
Anng
1 1.15
1.96
3.28
1 1.57
1.89
1 2.77
Table 6: Comparison between Bi- Bi-Drop and prior sub-net optimization strategies with varying low-resource scenarios (0.5K. 1K). We report the results of 10 random sceds and the best results are bold. Bi-Drop performs better than other methods in low-resource scenarios.
achieves up to 4.00, 4.23, and 5.29 average score improvements on 30%, 40%, and 50% reduction ratios respectively, outperforming other fine-tuni fine-tuning methods lower reduction ratios and showcasing its robustness towards the minority class.
BERT large with a limited amount of training data. The results are illustrated in Table 6. As the data amount decreases from 1.0K to 0.5K, the average improvement score of Child-Tuning over vanilla fine-tuning decreases from 1.57 1.15, while its improved variant DPS maintains a relatively stable improvement. But Bi-Drop improves the average improvement score from 2.77 to 3.28. The results indicate the superiority of Bi-Drop over prior meth- ods in low-resource scenarios.
5.3 Performance in Low-Resource Scenarios
As illustrated in Section 1 and 2, compared with prior FI-based sub-net optimization methods that have a strong dependence on the training data, Bi- Drop proposes step-wid wise sub-net selection strat- egy, which chooses the optimized parameters with the current mini-batch. In this section, we conduct extensive experiments to analyze how this depen- dency affects the performance of models. Con- cretely, we adopt various fine-tuning methods on
5.4 Ablation Study
To evaluate the effectiveness of our proposed fine- tuning strategy, we conduct an ablation study Table 7. The results show that both our sub-net selection strategy and gradient averaging strategy

---

Method
CoLA
MRPC
RTE
STS-B
Avg
A
Bi-Drop (gan + ESS)
64.94
91.79
73.79
90.59
80.26
+1.55
Savg Perturbation Factor
64.71
91.67
73.21
90.36
79.99
+1.28
Bang Scaling Factor
64.38
91.63
72.96
90.41
79.85
+1.14
Sang
RSS
64.24
90.96
71.73
90.25
79.30
+0.59
Eany
63.82
91.26
71.16
89.81
79.01
+0.30
Vanilla
63.76
90.41
70.97
89.70
78.71
0.00
Table Ablation results. ESS represents Effective Sub-net Selection strategy using both factors Perturbation and Scaling. RSS stands for Random Sub-net Selection strategy. Both our sub-net selection strategy and gradient averaging strategy are effective.
contribute to the performance improvement of Bi- Drop.
2021. Better fine-tuning by reducing representational collapse. In ICLR.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. large an- notated corpus for learning natural language infer- ence. In Proceedings of the 2015 Conference Empirical Methods Natural Language Processing, EMNLP 2015, Lisbon, Portugal, September 17-21, 2015, pages 632-642. The Association for Computa- tional Linguistics.
6 Conclusion
In this work, we propose new sub-net optimiza- tion technique for large-scale PLMs, named Bi- Drop. which leverages the gradients of multiple sub-nets generated by dropout select the up- dated parameters. Extensive experiments on var- ious downstream tasks demonstrate that Bi-Drop achieves consistent and remarkable improvements over vanilla fine-tuning and prior excellent ap- proaches by considerable margin, across various model architectures. Further analysis indicates the generalizability and robustness of Bi-Drop over transferring, data imbalance and low-resource periments.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. 2020. Language models are few-shot learners. NewrIPS.
Sanyuan Chen, Yutai Hou, Yiming Cui, Wanxiang Che, Ting Liu, and Xiangzhan 2020. Recall and learn: Fine-tuning deep pretrained language models with less forgetting. In Proceedings of the 2020 Confer- ence on Empirical Methods Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 7870-7881. Association for Computa- tional Linguistics.
7 Limitations
We propose novel and effective fine-tuning method, Bi-Drop, which achieves considerable performance improvement in downstream tasks. However, similar to some previous studies(Jiang et al., 2020; Aghajanyan et al., 2021; Wu et al., 2021), Bi-Drop requires multiple forward propaga- tions, which makes its training time efficiency not good enough compared with the vanilla fine-tuning method.
Kevin Clark, Minh-Thang Luong, Quoe Le, and Christopher D. Manning. 2020. ELECTRA: pre- training text encoders as discriminators rather than generators. In 8th International Conference Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Hal DauméllI. 2007, Frustratingly easy domain adapta- tion. In ACI 2007, Proceedings of the 45th Annual Meeting of the Association for Computational Lin- guistics, June 23-30, 2007, Prague, Czech Republic. The Association for Computational Linguistics.
Acknowledgements
This paper is supported by the National Key Research and Development Program of China 2020AAA0106700 and NSFC project U19A2065.
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understand- ing. In NAACL.
Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, and Noah A. Smith. 2020. Fine-tuning pretrained language models: Weight initializations, data orders, and early stop- ping. CoRR, abs/2000.06305..
References
Armen Aghajanyan, Akshat Shrivastava, Anchit Gupta, Naman Goyal, Luke Zettlemoyer, and Sonal Gupta.

---

Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski. Bruna Morrone, Quentin de Laroussilhe, Andrea Ges- mundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for NLP. In Pro- ceedings of the 36th International Conference on Ma- chine Learning, ICML. 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 2790-2799. PMLR.
Li Wan, Matthew D. Zeiler, Sixin Zhang, Yann Le- Cun, and Rob Fergus. 2013. Regularization of neural networks using dropconnect. In Proceedings of the 30th International Conference on Machine Learning, ICML 2013, Atlanta, GA. USA, 16-21 June 2013, volume 28 of JMLR Workshop and Conference Pro- ceedings, pages 1058-1066. JML.R.org.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2019. Glue: A multi-task benchmark and analysis platform for natural language understanding. In ICLR.
Haoming Jiang, Pengcheng He, Weizhu Chen, Xi- aodong Liu, Jianfeng Gao, and Tuo Zhao. 2020. SMART: Robust and efficient fine-tuning for pre- trained natural language models through principled regularized optimization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACI.).
Adina Williams, Nikita Nangia, and Samuel R. Bow- man. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Pro- ceedings of the 2018 Conference of the North Amer- ican Chapter of the Association for Computational Linguisties: Human Language Technologies, NAACL- HLT 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 1 (Long Pupers), pages 1112-1122 Association for Computational Linguistics.
Cheolhyoung Lee, Kyunghyun Cho, and Wanmo Kang- 2020. Mixout: Effective regularization to finetune large-scale pretrained language models. In ICLR.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: robustly optimized bert pretraining ap- proach. arXiv preprint arXiv: 1907. 1692.
Thomas Wolf, Lysandre Dehut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi. Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma. Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transformers: thate-tf natural language processing. In Pro- ceedings of the 2020 Conference on Empirical Meth- ods in Natural Language Processing: System Demon- strations, EMNLP 2020 Demos, Online, November 16-20, 2020, pages 38 45, Association for Computa- tional Linguistics.
Rabeeh Karimi Mahabadi, Yonatan Belinkov, and James Henderson. 2021. Variational information bottlneck for effective low-resource fine-tuning. In 9th Inter- national Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zam- parelli. 2014. SICK cure for the evaluation of compositional distributional semantic models. In Proceedings of the Ninth International Conference on Language Resources and Evaluation, LREC 2014, Reykjavik, Iceland, May 26-31, 2014, pages 216 223. European Language Resources Association (ELRA).
Chuhan Wu, Fangzhao Wu, Tao Qi, and Yongfeng Huang. 2022. Noisytune: A little noise help you finetune pretrained language models better. In Proceedings of the 60th Annual Meeting of the As- sociation for Computational Linguistics (Volume 2: Short Papers), ACL 2022, Dublin, Ireland, May 22- 27, 2022, pages 680-685. Association for Computa- tional Linguistics.
Jason Phang, Thibault Févry, and Samuel R. Bowman. 2018. Sentence encoders on stilts: Supplementary training on intermediate laheled-data tasks. CoRR, abs/1811.01088.
Lijun Wu, Juntao Li, Yue Wang, Qi Meng, Tao Qin, Wei Chen, Min Zhang, Tie-Yan Liu, et al. 2021. R-drop: regularized dropout for neural networks. In NewriPS.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Tlya Sutskever: 2019. Language models are unsupervised multitask learners. OpenAl blog.
Runxin Xu, Fuli Luo, Zhiyuan Zhang, Chuanqi Tan, Baobao Chang, Songfang Huang, and Fei Huang. 2021. Raise a child in large language model: To- wards effective and generalizable fine-tuning. In EMNLP.
Colin Raffel, Noam Shazeer. Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu, 2019. Exploring the limits of transfer learning with a unified text-to-text trans- former. arxiv preprint 1910 10683.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Car- bonell, Russ R Salakhutdinov, and Quoc V Le. 2019, Xlnet: Generalized autoregressive pretraining for lan- guage understanding. In NeurIPS.
Vijay Vasudevan, Benjamin Caine, Raphael Gontijo Lopes, Sara Fridovich-Keil, and Rebecca Roclofs. 2022. When does dough become bagel? analyzing the remaining mistakes on imagenet. In NewrIPS.
Hongyi Yuan, Zheng Yuan, Chuanqi Tan, Fei Huang, and Songfang Huang. 2022. Hype: Better pre-trained language model fine-tuning with hidden representa- tion perturbation. CoRR, abs/2212.000 12.08533.

---

Haojie Zhang, Ge Li. Jia Li, Zhongjin Zhang, Yuqi Zhu, and Zhi Jin. 2022. linettunng pre-trained language models effectively by optimizing subnetworks adap- tively. In NeurIPS.
Tianyi Zhang, Felix Wu, Arzoo Katiyar, Kilian Wein- berger, and Yoav Artzi. 2021. Revisiting few-sample BERT fine-tuning. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.

---

Appendix
C Hyper-Parameter Analysis
GLUE Benchmark Datasets
Bi-Drop uses two dropout techniques. In order to analyze the impact of the dropout rate on the ex- perimental results, simple analysis experiment was done here. In order make the comparison fair, all the parameters except the dropout rate are kept the same in the experiment. For simplicity, the dropout values are the same twice. The ex- perimental results are shown in 5. It can be seen that different datasets have different preferences for dropout values. CoLA and RTE achieve the best re- sults when the dropout value is 0.05; while MRPC achieves the best results when the dropout value is 0.1; STSB is insensitive the dropout value until the dropout value less than 0.1.
In this paper, we conduct experiments on datasets in GLUE benchmark (Wang et al., 2019). The statistical information of the GLUE benchmark is shown in Table 8.
Dataset
#Train
#Dev
Metrics
Sinleesentence Tasks
CoLA
8.5k
1.0k
Matthews Corr Accuracy
SST-2
67k
872
Inference
RTE
2.5k
277
Accuracy Accuracy Accuracy
QNLI
105k
5.5k
MNLI
393k
9.8k
67.6
Similarity and Paraphrase
67.2
MRPC
3.7k
408
Fl
68.8
STS-B
5.7k
1.5k
Spearman Cor
0.025 0.050
0.075
0.100 0.125
0.025 0.050 0,075 0.100 0,125 MRPC
QQP
364k
40k
Fl
CoLA
85.1
84.9
Table 8: Statistics and metrics of eight datasets used in this paper form GLUE benchmark.
84.7
84.5
0.025 0.050 0.075 0.100 0.125
0.025 0.050 0,075 0.100 0.125
RTE
stab
B Experimental Details
Figure 5: The effect of dropout rate experimental results
In this paper, we fine-tune different large pre- trained language models with Bi-Drop, including BERTBASE", BERTLARCES ROBERTABASE ROBERTALARCE'S DeDERTaLARGE'. ELECTRA,,NCE The training epochs/steps, batch size, and warmup steps are listed in Table 9. For the glue dataset, our maximum length is set as 128. We use grid search for learning rate from (le-5, 2e-5, le-1}. For Bi-Drop, we use grid search for dropout rate from (0.05, 0.1). The number of forward passes is fixed to two(k 2).
D Batch Size Doubled Training
and
We implement Bi-Drop by repeating the input data twice and forward-propagating twice. This is sim- ilar to doubling the batch size at each step. The difference is that half of the data is the same the other half, and directly doubling the batch size, the data in the same mini-batch all different. a fair comparison, we experimented with directly doubling the batch size. So for a fair compari- son, we experimented with directly doubling the batch size. The experimental results are shown in Table 10, results show that directly doubling the batch size has basically no improvement, and Bi- Drop significantly better than directly doubling the batch size.
for
We conduct all the experiments on single A40 GPU.
tree/main "https://huggingface http:://hggnnnnece...ooeeertbbes
main main hhtpess/haggiinhheeeoooorrbbbbbr
E
Comparison with Prior Methods on Roberta
main htttppps//haggiigaacc.../////cce
Sebertta-largg//tre//mlll co/google/
We compare Bi-Drop with various prior fine-tuning methods based on BERT large and report the mean
Seectra-largediiseene ttrpeaaaneeeeeeeainnnnneeeee

---

Model
Dataset
Batch Size
Epochs/Steps
Warmup Ratio/Steps
BERT
all
3 epochs
10%
CoLA
16
5336 steps
320 steps
RTE
16
2036 steps
122 steps
Roberta
MRPC
16
2296 steps
137 steps
STS-B
16
3598 steps
214steps
CoI.A
32
3 epochs
10%
RTE
32
10 epochs
10%
ELECTRA
MRPC
32
3 epochs
10%
STS-B
32
10 epochs
10%
CoLA
epochs
100 steps
RTE
32
6 epochs
50 steps
DeBERTa
MRPC
32
6 epochs
steps
STS-B
32
4 epochs
100 steps
Table 9: Hyperparameters settings for different pretrained models on variant tasks. These settings are reported in their official repository for best practice.
Method
CoLA
MRPC
RTE
STS-B
Avg
Vanilla
57.67
90.38
69.57
89.35
76.74
Vanilla(double bsz)
57.69
69.85
89.55
76.79
Bi-Drop
60.76
91.01
71.73
89.78
78.32
Table 10: Comparison of Bi-Drop and directly doubling the batch size. Bi-Drop is significantly better than directly doubling the batch size.
(and max) scores on GLUE benchmark in Table 11, following Lee et al. (2020) and Xu et al. (2021).

---

Method
CoL.A
MRPC
RTF
STS-B
Avg
A
Vanilla
66.01
92.56(93.66)
84.51
92.05(92.22)
83.78(85.05)
0.00
ChildTuningo
66.82(68.21)
92.67(93.58)
85.89(87.72)
92.36(92.53)
84.44(85.51)
+0.66(0.46)
DPS.mix
66.86(68.53)
92.51(93.89)
85.37(87.72) 72)
92.47(92.73)
84.30(85.72)
+0.52(0.67)
R-Drop
67.26(69.61)
92.47(93362)
85.44(87.63) (
92.42(92..58)
84,40(85.87)
+0.62(0.82)
Bi-Drop
68.03(70.89)
92.95(94.66)
86.10(88.09)
92.36(92.58)
84.86(86.56)
+1.08(1.51)
Table 11: Comparison between Bi-Drop with prior tunitu methods. We report the mean (max) results of 10
random seeds. The best results bold. Note that since R3F applicable regression, the result STS-B (marked with ") remains the same vanilla. Bi-Drop achieves the best performance compared with other methods.


Reviews:
[{"review_id": "z6HtZVDMmQ", "reviewer": "Reviewer_TJ6o", "summary": "", "questions": "See above.", "limitations": "", "rating": 3, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "***Q1: The idea of looking for important parameters and doing selective optimization is very similar to [1]. However, [1] uses soft-masking instead of hard-masking. What are the pros and cons compared to soft-masking methods?***\n\nR1: Based on your statement, we carefully reviewed the paper [1]. Although our paper shares some similarities with the ideas presented in [1], there are significant differences between the two approaches. **Firstly**, the motivations behind [1] and our research are different. While [1] focuses on identifying important heads that capture pre-trained knowledge and applies a soft-mask to those heads, our motivation is to update parameters that are most useful for the current task while conservatively masking the remaining parameters. **Secondly**, in [1], a masking strategy is applied to each head of the multi-head attention, while in our method, we selectively mask individual parameters across the entire network. **Thirdly**, [1] utilizes a soft-mask, which requires introducing a learnable gate vector for each head of the attention mechanism. The gate vector's average gradient for each parameter is computed using the Kullback-Leibler (KL) divergence loss, and this average gradient is used as weights during backpropagation to achieve the soft-mask effect. On the other hand, our approach employs **Perturbation Factor** to select parameters that exhibit more stable gradients across multiple forward passes under different dropout configurations. This allows us to update these selected parameters without introducing additional learnable parameters, making our method simpler and more efficient. **Based on the three points mentioned above**, [1] only applies the masking strategy to a few heads of the multi-head attention, which enables them to introduce a relatively small number of learnable parameters for the soft-mask implementation. However, our method is designed for the entire model by selectively choosing sub-net based on rule-based strategies, without introducing any learnable parameters. If we were to implement a soft-mask approach, it would require introducing a learnable parameter for every parameter, which could lead to a significant increase in memory and computational resources. Therefore, the soft-mask approach is not suitable for our method.\n\n***Q2: Using fisher information is very straightforward when it comes to important computation. There is a lot of randomness in the dropout, how can you guarantee you are protecting what is important?***\n\nR2: As mentioned in the related work section of this paper, FI-based methods indeed rely heavily on the training data and can exhibit hysteresis in sub-net updating. Furthermore, as mentioned in the response to question 1, the motivation behind this work is different from the motivation in paper [1]. Reference [1] focuses on identifying and preserving parameters that are crucial for the knowledge acquired during pre-training. In contrast, our approach involves selecting parameters that are important for the current task. We only update these important parameters, while the rest of the parameters remain unchanged. Therefore, from this perspective, our method is more conservative, as we protect parameters that are deemed less important for the current task.\n\n***Q3: Can this method apply to parameter-efficient methods, like LoRA? Since this method physically changes the LM, it seems to me it is not applicable in this case. Also the authors may want to compare with those parameter-efficient method as well.***\n\nR3: Thank you very much for your advice. In fact, our approach and the parameter-efficient methods are orthogonal, so our approach can be applied on top of methods like LoRA. In the future, we can verify the effectiveness of applying our approach on top of methods like LoRA.\n\n[1]: Adapting a Language Model While Preserving its General Knowledge, Ke et al., EMNLP 2022", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Generalization in fine-tuning is an important problem  \n2. Masking part of the parameters makes intuitively sense because the LLM is known to be over-parameterization  \n3. The writing is clear\n", "reasons_to_reject": "1. The idea of looking for important parameters and doing selective optimization is very similar to [1]. However, [1] uses soft-masking instead of hard-masking. What are the pros and cons compared to soft-masking methods?  \n2. Using fisher information is very straightforward when it comes to important computation. There is a lot of randomness in the dropout, how can you guarantee you are protecting what is important?  \n3. Can this method apply to parameter-efficient methods, like LoRA? Since this method physically changes the LM, it seems to me it is not applicable in this case. Also the authors may want to compare with those parameter-efficient method as well\n\n[1]: Adapting a Language Model While Preserving its General Knowledge, Ke et al., EMNLP 2022", "questions_to_authors": "See above.", "paper_topic_and_main_contributions": "This paper improves the fine-tuning by updating partial parameters. This is achieved by learning a gradient mask based on dropout. It lies in the family of regularization methods. Experiments results on GLUE demonstrate its effectiveness.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "See above."}, "extra_text": {}}, {"review_id": "ouZDVbvFaM", "reviewer": "Reviewer_uYzG", "summary": "", "questions": "1. Can the authors explain some details about the ablation experiments? What is the impact of the two factors in 3.2.2?\n2. Would the authors be willing to add more experimental results, such as experiments comparing with other methods on Roberta-Large?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "***Q1: Can the authors explain some details about the ablation experiments? What is the impact of the two factors in 3.2.2?***\n\nR1: Thank you very much for your careful reading of our partial ablation experiment. As you mentioned, in Table 7 of the ablation experiment results, ESS represents our Effective Sub-net Selection strategy, which involves using the two factors described in section 3.2.2. In fact, we also conducted separate ablation experiments for these two factors, but due to space limitations, they were not included in the table. Now, we present a more detailed ablation result in the table below. Once again, thank you for your question.\n\n| **Method**                                | **CoLA**  | **MRPC**  | **RTE**   | **STS-B** | **Avg**   | **∆**     |\n| ----------------------------------------- | --------- | --------- | --------- | --------- | --------- | --------- |\n| **Bi-Drop** ( **g**$_{avg}$ + **ESS**)   | **64.94** | **91.79** | **73.79** | **90.59** | **80.26** | **+1.55** |\n| **g**$_{avg}$ **+ Perturbation Factor** | 64.71     | 91.67     | 73.21     | 90.36     | 79.99     | +1.28     |\n| **g**$_{avg}$ **+ Scaling Factor**      | 64.38     | 91.63     | 72.96     | 90.41     | 79.85     | +1.14     |\n| **g**$_{avg}$ **+** **RSS**             | 64.24     | 90.96     | 71.73     | 90.25     | 79.30     | +0.59     |\n| **g**$_{avg}$                           | 63.82     | 91.26     | 71.16     | 89.81     | 79.01     | +0.30     |\n| **Vanilla**                               | 63.76     | 90.41     | 70.97     | 89.70     | 78.71     | 0.00      |\n\nCaption: Ablation results. ESS represents our Effective Sub-net Selection strategy using both factors Perturbation and Scaling. RSS stands for Random Sub-net Selection strategy. Both our sub-net selection strategy and gradient averaging strategy are effective.\n\n***Q2: Would the authors be willing to add more experimental results, such as experiments comparing with other methods on Roberta-Large?***\n\nR2: Based on your comments, we conducted experiments using the RoBERTa-large model. The specific results are shown in the table below.\n\n| **Method**            | **CoLA**         | **MRPC**         | **RTE**          | **STS-B**        | **Avg**          | **∆**           |\n| --------------------- | ---------------- | ---------------- | ---------------- | ---------------- | ---------------- | --------------- |\n| **Vanilla**           | 66.01(68.03)     | 92.56(93.66)     | 84.51(86.28)     | 92.05(92.22)     | 83.78(85.05)     | 0.00            |\n| **ChildTuning**$_D$ | 66.82(68.21)     | 92.67(93.58)     | 85.89(87.72)     | 92.36(92.53)     | 84.44(85.51)     | +0.66(0.46)     |\n| **DPS**$_{mix}$     | 66.86(68.53)     | 92.51(93.89)     | 85.37(87.72)     | 92.47(92.73)     | **84.30(85.72)** | +0.52(0.67)     |\n| **R-Drop**            | 67.26(69.63)     | 92.47(93.62)     | 85.44(87.63)     | 92.42(92.58)     | 84.40(85.87)     | +0.62(0.82)     |\n| **Bi-Drop**           | **68.03(70.89)** | **92.95(94.66)** | **86.10(88.09)** | **92.36(92.58)** | 84.86(86.56)     | **+1.08(1.51)** |\n\nCapiton: Comparison between Bi-Drop with prior fine-tuning methods. We report the mean (max) results of 10 random seeds. The best results are **bold**. Bi-Drop achieves the best performance compared with other methods.\n\n***Q3: The approach is not novel enough. Bi-Drop contains three parts, which are multiple dropout, parameter importance measure and dynamic update sub-net mask. In which all of them are used in the previous work. At the same time, the authors does not prove why to combine them through the experiments or theoretical analysis .***\n\nR3:Thank you for your comment. Our work, similar to the CHILD-TUING and DPS mentioned in this paper, focuses on addressing how to effectively update sub-net in pre-trained models. Therefore, measuring parameter importance serves the purpose of facilitating sub-net updates, meaning that these two aspects are essentially the same. The key difference among different works lies in how the importance of different parameters is measured, as mentioned in the related work section of this paper. CHILD-TUING and DPS employ FI-based methods for this purpose. However, FI-based methods indeed rely heavily on the training data and can exhibit hysteresis in sub-net updating. In contrast, our method introduces an innovative approach by using different dropout settings to perform multiple forward propagations, resulting in multiple gradient values for each parameter. These gradient values are then utilized in the Perturbation Factor and Scaling Factor mentioned in the paper to calculate the importance of each parameter. Our method overcomes the limitations of the FI-based methods mentioned above, and we have demonstrated its effectiveness through extensive experimentation compared to FI-based methods. In summary, our innovation lies in proposing a new and effective strategy for measuring parameter importance (or, in other words, an effective subnetwork selection strategy), overcoming the limitations of FI-based methods. The paper provides a clearer demonstration of our method's specific workings, which is why it is divided into three parts in Section 3.2. However, in practice, it all boils down to performing effective subnetwork selection.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. According to the results presented by the authors, their method achieved good performance in different experiments.\n2. In the robustness and generalization experiments, the proposed method shows more effective results.", "reasons_to_reject": "1. The approach is not novel enough. Bi-Drop contains three parts, which are multiple dropout, parameter importance measure and dynamic update sub-net mask. In which all of them are used in the previous work. At the same time, the authors does not prove why to combine them through the experiments or theoretical analysis .\n2. The experiment is not enough to prove the effectiveness of the method. The main experiments are only conducted on BERT-large, with little difference in performance. Meanwhile, the authors only compare with Vanilla method on other PLMs (e.g. Roberta).", "questions_to_authors": "1. Can the authors explain some details about the ablation experiments? What is the impact of the two factors in 3.2.2?\n2. Would the authors be willing to add more experimental results, such as experiments comparing with other methods on Roberta-Large?", "paper_topic_and_main_contributions": "This paper proposes a fine-tuning method that updates the sub-net based on the parameter importance measured by the gradient of multiple dropout on the parameter. The effectiveness of the proposed method is verified on different datasets and different PLMs.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "kt0KwqlxRK", "reviewer": "Reviewer_NTTU", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "***Q1: What is the significant motivation of leveraging stepwise sub-network detection and parameter update? The line of 187 seems the real motivation of the proposed method, compared with previous method of Child-tuningD, but there is no empirical or theoretical proof to support the claim. Please clearly state the strong point and motivation, uncovering the novelty of this work.***\n\nR1: We appreciate your inquiry of the empirical proof of Bi-Drop. As we illustrate in the paper, Child-tuningD [1] chooses an unchanged subnetwork to optimize and ignores the update of other parameters during the whole finetuning process, which may affect the model’s perception of downstream tasks. In other words, **since there is a misalignment between the original pretrained model and the fine-tuned one, the importance of model parameters may vary from the estimation before** **tuning**. To handle this problem, DPS [2] proposes to dynamically select the optimized sub-network from multiple mini-batches of data. However, these FI-based methods still have two limitations: (1) **hysteresis in sub-net updating**: the sub-net preference is estimated with the model parameters in previous iterations and may be incompatible with the current update step; and (2) **insufficient utility of** **training data**: FI estimation requires cumulative gradients through multiple mini-batches, so these methods cannot fit in situations with data scarcity. \n\nIn this work, we propose Bi-Drop, a **novel** ***FI-free*** **method** to estimate the importance of model parameters. Compared with previous methods that choose the optimized sub-net based on the accumulated gradients of multiple mini-batches, Bi-Drop derives a more stable and robust training trajectory with **simultaneous sub-net update** and **high utility of** **training data**. Experimental results illustrated in the paper demonstrate the superiority over prior methods as well as its generalization ability and robustness. \n\n***Q2: The line of 76 is ambiguous and can not explicitly show why reusing training data is helpful for mitigating overfitting. Intuitively reusing training data can accelerate convergence but it is an unanswered question for better generalization. More references or experiments are required for this claim.***\n\nR2: Here we would like to make clarifications on the claims of the 76 line in our paper. We emphasize the strengths of Bi-Drop with two aspects: Firstly, Bi-Drop proposes a novel training trajectory to selectively update model parameters during fine-tuning, which contributes to better generalization capabilities of models; Secondly, compared with previous methods (such as [1] and [2]) that have high dependences on the data amount, Bi-Drop can easily fit in situations with data scarcity, with a high utility of training data.\n\n***Q3: Also there should be a structural issue in terms of identifying subnetworks. Are the attention weights and weights from other layers considered equally?***\n\nR3: Following [1] and [2], we treat all parameters in the Transformer Block equally to estimate their importance to the downstream task. However, we appreciate your inquiry of the structural difference in identifying subnetworks, which is an interesting point to explore. We are committed to further exploring this in our ongoing and future research endeavors.\n\n***Q4: With the K times feedforward processes, we can estimate the importance of each parameter. Ablation studies w.r.t K should be available. And the batch size and the highest value of p percentile. We need detailed analysis of why stepwise version of subnetwork detection & update, apparently the proposed method is very sensitive to these three factors.***\n\nR4: In our main experiments, we set the same batch size as [1] and [2] for fair comparison, which is illustrated in Appendix B. Besides, we have also implemented the baseline with doubling the batch size, to fairly illustrate the superiorty of Bi-Drop. We will include more ablation studies with various k and p percentile in the revised version.\n\n[1] Xu et al. Raise a Child in Large Language Model: Towards Effective and Generalizable Fine-tuning. In EMNLP 2021.\n\n[2] Zhang et al. Fine-Tuning Pre-Trained Language Models Effectively by Optimizing Subnetworks Adaptively. In NIPS 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.Good and enough experiments, the expiration to OoD generalization and task generalization is interesting. In the scenario of continual pre-training, weights enjoying task generalization would be an indicator for avoiding forgetting. Further exploration in continual pretraining would be a good consequence of the proposed method.\n2. The proposed Perturbation Factor and Scaling Factor are effective and straightforward.", "reasons_to_reject": "1. What is the significant motivation of leveraging stepwise sub-network detection and parameter update? The line of 187 seems the real motivation of the proposed method, compared with previous method of Child-tuningD, but there is no empirical or theoretical proof to support the claim. Intuitively if the static sub-network is estimated with all training data, then it would be good enough. In contrast, the subnetwork is detected and updated in each mini-batch optimization in a dynamic manner, this means the subnetwork can only fit the provided minibatch of data, but never know its generalization to other data in the training set. How randomness introduces benefits should be the strong point of this paper. One potential method is to check if there is strong overlapping among training trajectories in terms of the selected parameters. Compared to child-tuning, the selected sub-network is much smaller for whatever promising reasons. A case is SGD introduces an approximated regularization to the model, contributing to a better solution. This is the theoretical benefits of SGD than GD. I will definitely raise my score if empirical proof is available for why this method is better than child-tuning, speaking of stepwise subnetwork update. Please clearly state the strong point and motivation, uncovering the novelty of this work.\n2. The line of 76 is ambiguous and can not explicitly show why reusing training data is helpful for mitigating overfitting. Intuitively reusing training data can accelerate convergence but it is an unanswered question for better generalization. More references or experiments are required for this claim.\n3. Also there should be a structural issue in terms of identifying subnetworks. Are the attention weights and weights from other layers considered equally? \n4. With the K times feedforward processes, we can estimate the importance of each parameter. Ablation studies w.r.t K should be available. And the batch size and the highest value of p percentile. We need detailed analysis of why stepwise version of subnetwork detection & update, apparently the proposed method is very sensitive to these three factors.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper introduces a batch-wise subnetwork detection and parameter update method for the purpose of avoiding overfitting in the course of fine-tuning pretrained language models.  Instead of estimating a static sub-network, this paper explores gradient-related information to figure out important parameters through feeding the current min-batch data several times. The experimental results though not be improved a lot but shows the superiority of the proposed method than previous works.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Equation 2 seems not correct in terms of j"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The experiments in the paper is sound. All reviewers agree the results support the claim about generalization in the paper.

However, there are a few concerns. Reviewer uYzG raises a valid concern that the improvement is not as significant. Reviewer TJ6o and NTTU raises concern on the applicability and computational cost of the approach. Overall, the authors should revise the paper to discuss whether the marginal improvement is worth the extra effort.

The writing needs some more work. There are several bogus claims and statements.

For example, as pointed out by Reviewer NTTU, in line 187,

> ... CHILD-TUNING_D only optimizes an unchanged sub-net during fine-tuning and ignores the update of other parameters, which may degrade the model’s performance on downstream tasks.

The sentence is too defensive that it's simple vacuous, but at the same time, the statement is likely blaming the wrong thing.

Another example is the problem stated in the Introduction:

> (1) hysteresis in sub-net updating: the sub-net preference is estimated with the model parameters in previous iterations and may be incompatible with the current update step; and (2) insufficient utility of training data: FI estimation requires cumulative gradients through multiple mini-batches, so these methods cannot fit in situations with data scarcity.

The description is imprecise, and the paper does not validate these problems before going about solving them. Again, we do not know if the paper is blaming the right things.

Overall, the paper has merit, but I would highly recommend the authors review and improve the wording in their revision.
Decision: Accept-Findings