Fine-Tuning Pre-Trained Language Models Effectively by Optimizing Subnetworks Adaptively
Authors: Zhang Haojie
Ge Li
Jia Li
Zhongjin Zhang
YUQI ZHU
Zhi Jin
Venue: NeurIPS 2022 Accept (2022)

Abstract: Large-scale pre-trained language models have achieved impressive results on a wide range of downstream tasks recently. However, fine-tuning an extremely large-scale pre-trained language model on limited target datasets is often plagued by overfitting and representation degradation. In this paper, we propose a Dynamic Parameter Selection (DPS) algorithm for the large-scale pre-trained models during fine-tuning, which adaptively selects a more promising subnetwork to perform staging updates based on gradients of back-propagation. 
Experiments on the GLUE benchmark show that DPS outperforms previous fine-tuning methods in terms of overall performance and stability, and consistently achieves better results with variable pre-trained language models. In addition, DPS brings a large magnitude of improvement in out-of-domain transferring experiments and low-resource scenarios, which shows that it can maintain stable general contextual features and reduce the representation collapse. We release our code at \url{https://github.com/ZhangHaojie077/DPS}.

Fine-Tuning Pre-Trained Language Models Effectively by Optimizing Subnetworks Adaptively
Haojie Zhang', Ge Jia; Li', Zhongjin Zhang', Yuqi Zhu', Zhi Jin'
Keyy Laboratory of High Confidence Software Technologies (Peking University), Ministry of Education; Institute of Software, EECS, Peking University, Beijing, China changhaojie@st pku. edu. en, lige@pku. edu. lijiaßstu. .ku. edu. en, zjz1230stu. pku. edu.cn, en, zhuyuq1970gmail com, zhi edu. cn
Abstract
Large-scale pre-trained language models have achieved impressive results on a wide range of downstream tasks recently. However, fine-tuning an extremely large-scale pre-trained language model on limited target datasets is often plagued by overfitting and representation degradation. In this paper, we propose Dy- namic Parameter Selection (DPS) algorithm for the large-scale pre-trained models during fine-tuning, which adaptively selects a more promising subnetwork to per- form staging updates based on gradients of back-propagann pooepprrpaaation. Experiments on the GLUE benchmark show that DPS outperforms previous fine-tuning meth- ods in terms of overall performance and stability, and consistently achieves bet- ter results with variable pre-trained language models. addition, DPS brings large magnitude of improvement in out-of-domain transferring experiments and low-resource scenarios, which shows that can maintain stable general con- textual features and reduce the representation collapse. We release our code at ttttps://iit gighhub.... com http:/hnngaaooo077//PPPP2
Introduction
The paradigm of pre-training on large-scale unlabeled corpus and fine-tuning on task-specific datasets has led a remarkable breakthrough in the field of Natural Language Processing (NLP) [Devlin et al. 2018 Raffel et al.] [019 Clark et al. 2020 He et al. [02T]. While the great success has been achieved on various downstream tasks, previous research has pointed out that such approaches are still facing two limitations: 1 Fine-tuning instability. With the same
different restarts will lead to huge differences in the performance pre-trained language models, especially on small datasets (Phang al., 2018 Dodge al., 2020 Lee et al., 2020 Zhang et al. 2021 Mosbachi (ct al. 2021]. Poor generalizability. Limited by the size of the downstream datasets, aggressive fine- tuning can lead to catastrophic forgetting and overfitting, which causes the model to perform poorly when transferring to -ut-oi-domain data or other related tasks [Mahabadi et al. 2021 Aghajanyan al. 2021 Xu et al. 2021]. Therefore, it remains challenging topic adapt large-scale pre- trained models to various scenarios while maintaining their stability and generalizability to the fullest.
Optimizing subnetwork of pre-trained models has been proved to be effective approach to improve stability or mitigate overfitting, such as Mixout [Lee et al., 2020) and CHILD-TUNINGG [Xu et al. 202T). Mixout utilizes pre-trained weights to randomly replace some parameters with probability of p and updates only the subnetwork composed of the parameters without replacement. CHILD-TUNING, computes the importance of parameters based all training samples' gradients before training and chooses an unchanged subnetwork composed important parameters for updating.
"Corresponding author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Despite their great performance, we argue that (i) choosing subnetworks randomly (Mixout) ignores immootanne the importance of parameters: (ii) choosing an unchanged subnetwork and ignoring the update of some may affect the model's perception of downstream tasks. Moreover, accumulating all training samples' gradients calculated by proppaaaaattoon outside training would induce heavy additional computational overhead.
parameters
To overcome the above two restrictions of existing subnetwork optimizations, we propose Dynamic Parameter Selection (DPS) algorithm to fine-tune pre-trained models adaptively. Overall, DPS consists of cyclic two-stage updating strategy: Stage accumulates in-batch gradients of updated parameters during fine-tuning. Stage If utilizes accumulated values of Stage to derive a stage-specift subnetwork composed of important parameters to update while keeping non-subnetworr constant. Note that both two stages consist of several iterations and the whole fine-tuning process will repeat several two-stage updating processes until the model convergence. Compared to previous subnetwork optimizations, we summarize the advantages of DPS as follows: (i) DPS can adaptively choose out more promising subnetworks based on the changing parameter importance while paying attention to more parameters possibilities, which enables better optimization of the model. (ii) DPS can effectively combine two processes: fine-tuning and deriving subnetworks, which greatly reduces the additional heavy computational costs introduced by separating above two processes like CHILD-TUNING,
In details, we propose two variants, DPS Dense and DPS Mix, to optimize the fine-tuning process. DPS Dense aims at optimizing the full network update process. In Stage I, DPS Dense updates the whole network while accumulating gradients' values. In Stage II, DPS Dense updates the stage- specific subnetwork composed of important parameters measured by ranking the accumulated values of Stage In contrast, DPS Mix is designed to optimize the stochastic parameter update process. In Stage I, DPS Mix randomly replaces some outgoing parameters of the neural network with pre-trained parameters and accumulates in-batch gradients of the remaining updated parameters. In Stage II, selection strategy, which utilizes updated gradients' average
DPS Mix applies frequency-penalize value and an exponential penalty factor negatively correlated with update frequency to choose out the stage-related important parameters for updating. It adaptively chooses a more promising subnetwork in stages while avoiding deviations from the pre-trained weights due to aggressive optimization.
We re-evaluate several state-of-the-ee fine-tuning regularization approaches [Lee et al. 2020 Agha
janyan et al. 2021 Zhang et al.) 2021 Xu et al.] 2021 Wu et al., 2021] on GLUE Benchmark
and find that DPS outperforms prior approaches by 0.44 ~ 1.33 points gains and 0.29 ~ 0.80 std (standard deviation) drops. Moreover, DPS brings an improvement in out-of-domain transferring seerriments/low-rsoouue
scenarios with up to 1.86/3.27 average score, indicating that DPS can obtain excellent generalization ability. In extensive experiments, we further explore DPS in the following four settings: (i) subnetwork size; (ii) various pre-trained models; (iii) sufficient number of training examples; (iv) time usage, Based on the above extensive experiments, we observe that (i) DPS can better capture task-relevant parameters across a variety of subnetwork sizes than Mixout and
(ii) DPS can consistently bring a stable improvement in various are-trained models; (iii) our proposed DPS is helpful model performance in fine-tuning with sufficient number of training examples; (iv) DPS is a more time-efficient approach than several fine-tuning methods.
Approach
2.1
Background and Related Work
We first start to introduce the paradigm of fine-tuning the subnetwork by giving formulations of forward and back propagation during standard fine-tuning, Mixout, and CHILD-TUNING,, We will describe all formulations for Stochastic Gradient Descent (SGD) iterations. We assume W((t) = (W/(1) ... w(!)), ) where wr(*) represents parameters of the i-th neuron at the t-th iteration. Standard fine-tuning formulates model parameters as follows:
((++1) w OC (W(1)
@W(+))
where L represents in-batch loss; 7 represents learning rate. Differ from fine-tuning whole network, Mixout first derives mask matrix (4) (M .... ! where M. is same-sized w:!" and

---

' are mutually independent Bernoulli(1 p) for all and I. Then Mixout replaces parameters corresponding to zero values in M with pre-trained parameters and only update the subnetwork composed of remaining parameters. Its formulation as follows:
Oc ((MOWW + +(I -I- M w pw()) (1 (1-p)-) p)
W(++) W(o 1)
OW())
where / is full-one matrix and is same-sized ( ( represents the parameter matrix of the pre-trained model; the larger p is, the smaller the subnetwork.
Besides selecting a subnetwork randomly, CHILD-TUNING has explored fine-tuning subnetwork composed of measured important parameters. Considering Fisher Information [Fisher 1922] serves as good way of measuring the amount of information that random variable carries about parameter of the distribution [Tu et 2016) and been widely used in modern machine learning and deep learning [Achilee et al. 2019 Kirkpatrick et al. 2017 Amari 1996. Pascanu and Bengio, 2013 Singh
and Alistarh [020] Crowley et al. 2018) Martens 2014 Theis et al. 2018].
utilizes Fisher Information to compute the importance of parameters based on all training samples
before training, then derives an unchanged subnetwork composed of important parameters to update. Its formulation is as follows:
0L (W(11)
OC (W(O))
MCT, E( (W > sort
Σ(
(3)
OW(()
OC (W(+)
w(++1) w(()
McTD
(4)
(W
where sorl(A), represents the value of percentile in matrix A after sorting in ascending order; MCTD is an unchanged mask matrix, which determines the subnetwork used for updates throughout the fine-tuning process.
Since their introduction that aggressive fine-tuning will lead to instability and overfitting, especially on small datasets (Dveli et al. 2018 Phang et al. (018)), various fine-tuning methods have been proposed solve the above problems, except for subnetwork optimizations. Some of them utilize output
representations or inject noise into input to regularize models [Mahabadi et al. 2021 Aghajanyan al. 2021 Wu et al.] [021], while others adjust the weights of top layers of Transformer or
BERTAdam optimizer to make the pre- are-trained models more task-relevant. (Zhang et al. 2021
Mosbach et 2021
Algorithm 1 Training Algorithm for DPS
Require: ms: total training steps; update ratio; update steps; p: the proportion of parameters not updated the overall parameters in Stage II; current cycle times; n: total cycle times ur, [ms/2us
1: Initialize timestep <-
for = 0
3: k-th cycle
4: while < (k + 1) 2us do
5: // Stage 1
if [t/us] %2 then
Accumulate in-batch gradients of updated parameters during fine-tuning.
//Stage II if [t/us]] 2 = and (t - 1) us 0 then
9:
10:
Derive a Stage-speciffe mask matrix by selecting the top (1 pp parameters by a specified selection strategy.
11:
Clear the accumulated values of Stage Update selected subnetwork by stage-specific mask matrix.
12:
13:
else
14:
Update the selected subnetwork by stage-specific mask matrix.
15:
1 1-6 +-++1
16:
end while
end for
3

---

2.2
Overview of DPS
DPS consists of cyclic two-stage updating strategy: Stage I accumulates in-batch gradients of updated parameters during fine-tuning; Stage II utilizes accumulated values of Stage to derive stage-specific subnetwork composed of important parameters to update while keeping non-subnetwork constant. We provide a pseudo-code of DPS in Algorithm I We argue that fine-tuning dynamic promising subnetwork while paying attention to more parameters possibilities can align all parameters more relevant for a specific downstream task, which in turn better unlocks the potential of the model. To select promising subnetwork adaptively, we propose two enhanced fine-tuning strategies: DPS Dense (in Sec. 2.3) and DPS Mix (in Sec. (.4), which respectively optimize the fine-tuning processes of full network update and stochastic network update.
2.3 DPS Dense
We firstly explore the DPS in full network update called DPS Dense. Before fine-tuning, DPS Dense initializes a Gradient Accumulation Matrix (GAM), which is the same-sized as W In Stage I, DPS Dense fine-tunes the whole network as Equation [ while accumulating the gradients' square of all parameters at every iteration with the help of GAM; In Stage II, DPS Dense first ranks GAM and derives stage-specific mask matrix M filtered by selecting top (1 p) highest Fisher Information of GAM, then utilizes MDPS to update the stage-specific subnetwork composed of important parameters. we denote the formulation of Stage II as follows:
MDPS GAM > sort(GAM)pp
(5)
aL (W(!))
W(++1)= ((t) 17 OW MDPS-
(6)
The specific update process can be seen in Figure
step 21r 2kt"ss1,.... step 2k'uus+i
Stage
loi
oi
sis
sia
Di
81
0i
DPS Dense
step (2ke1)uss11 step 12k+1/us
Stage
Mors
il
DP8 Mix
step 2k'ust1,....
step 2k'us
Stage
GAM l
GAM
+
db
+
8ia
PAM 0
1
FAM
1+
14
Figure 1: In this figure, show the k-th two-stage updating process of DPS Dense and DPS Mix. Pi represents the the gradient value of parameter at time step 2k * us + (i <= us); the parameters on the red edges are same-valued as corresponding parameters of W ((22+t)us++1) the parameters on the orange edges are same-valued as corresponding parameters of (())) Both red and orange edges will not be updated during back propagation; GAM, F FAM and Mprs will be re-initialized the beginning of every cycle.
2.4 DPS Mix
Considering stochastic parameter update is common to deploy in fine-tuning pre-trained models, we also propose an alternative to optimize this fine-tuning process called DPS Mix. Before fine-tuning, DPS Mix needs to initialize another Frequency Accumulation Matrix (F AM) besides GAM, which

---

is used to store the frequency of every parameter being updated in Stage I. In Stage I, DPS Mix derives mask matrix ( like Mixout, and then fine-tunes the subnetwork determined by M while replacing non-subnetwork as pre-trained weights as Equation 2 In Stage II, DPS Mix derives
MDPs according to a trequency-penalizee selection strategy as follows:
GAM PAM FAM
sort
FAM p
GAM if us < 50
MDrs
GAM FAMM
C
> sort
GAM FAM C
FAM
else
As Equatior illlsstrates, DPS Mix follows the ratio of GAM and FAM as the basis for deriving the Mprs. During the random selection of Stage I, considering that each parameter may be updated at an inconsistent frequency and parameters with more frequent updates may have smaller potential, DPS Mix also multiplies exponential penalty factor negatively correlated with update frequency in addition to the ranking selection, to give more chances to parameters with fewer updates. Since this penalty factor may introduce some fluctuations, we grid search the boundary and decide to exclude this factor when us is less than 50. After obtaining the MDPs, DPS Mix formulates model parameters follows:
w(++1) = w(0 ac ((MDPsW()) (1- W' - -wW) (1 (1) OW())
(8)
where W are same-valued [((2+++)w+++) Note that the subnetwork size of Stage I is the same-sized as Stage II's in DPS Mix. The specific update can be seen in Figure [ In terms of implementation details, apply DPS Dense and DPS Mix to specific layers.
3 Experiments
3.1 Datasets
GLUE benchmark. Following previous studies [ee et al. 2020 Dodge et al. 2020 Zhang et al.
202T]. we conduct series of extensive experiments on eight datasets from the GLUE benchmark [Wang et al. [019]. The datasets cover four tasks: natural language inference (RTE, QNLI, MNLI), paraphrase detection (MRPC, QQP), linguistic acceptability (CoLA),, and sentiment classification (SST-2). Appendix B provides dataset statistics and brief description of each dataset. Since the test set can only be submitted online for a very limited times, we follow several previous studies (Phang
et al.
2018
Lee et al.) 2020 Dodge et al., 2020 Aghajanyan et al.) 202 Zhang et al. 202] Xu
ct al.
202T
that fine-tune on the training sets and report the results on the development sets.
NLI Datasets. We evaluate and probe the generalization ability of DPS on several Natural Language Inference (NLI) tasks, including SNLI [Bowman al. (2015), MNLI (Williams et al. [2018). MNLI- M [Williams et al. 2018). RTE [Bentivogli et al. 2009], SICK (Marelli et al.) [2014] and SciTail Khot et al. 2018). All tasks are reported by Accuracy. Since the target datasets have different label spaces, during the evaluation, we map predictions to each target dataset's 's space (in Appendix E).
3.2 Experimental Setup
Woll We the pre-trained models and codes provided by HuggingFacd Wolf et al. 2020)] Appendix C provides specific hyper-paramen parameter details, and unless noted otherwise, we follow the default hyper-parameter setup of HuggingFace.
3.3
Comparing Prior Methods for Few-Sample BERT Fine-tuning
3.3.1
Baseline
Mixout [Lee et al. 2020 is a stochastic parameter update technique based on the Bernoulli dis-
tribution for preventing catastrophic forgetting during fine-tuning. It randomly replaces the model parameters with pre-trained parameters with probability of p. R3F [Aghajanyan et al. [021] is motivated by trust region theory and injects parametric noise to the embedding of the pre-trained

---

representations. R-Dropout [Wu et al. 2021 minimizes the bidirectional KL KL-divergence between the output distributions of two sub models sampled by Dropout. CHILD-TUNING, [Xu et al.) 2021] utilizes the downstream task data to detect the most task-related parameters as the child network and freezes the parameters in the non-child network to their pre-trained weights during fine-tuning. Re-init [Zhang et al.] 2021] re-initializes the pooler layers and the top L BERT Transformer layers using the original BERT initialization, N (0,0.023).
3.3.2 Results
In this section, we compare DPS with a variety of fine-tuning regularization approaches on BERTLARGE following [Lee et al. 2020 Zhang al. 2021 Xu et al. 2021]. The hyper-paramettr search spaces of different fine-tuning regularization methods are supplemented in Appendix D. Table summarizes the experimental results. Firstly, we can notice that Mixout and CHILD-TUNING, achieve better results than vanilla finettuning, and perform nearly the best baseline Re-init, which competitive approach to
proves that fine-tuning a subnetwork while utilizing pre- pre-trained weights oeeeeeruuppeeiee regularize the model. Secondly, our proposed straightforward DPS Dense outperforms all baselines on average, which demonstrates that fine-tuning subnetworks adaptively is more effective. Since DPS accumulates in-batch gradients to derive important sub-networks during fine-tuning, we also consider DPS to be a more time-efficient approach than CHILD-TUNING,, which utilizes Fisher Information to derive important sub-network based on all training samples outside fine-tuning. Thirdly, it is easy to find that DPS Mix achieves better results while further improving stability, especially on CoLA with great std drop. This reveals that DPS can be effectively compatible with various fine-tuning strategies, enabling better optimization of the model.
COLA
MRPC
RTE
STS-B
Avg.
Methods
mcan
std
mcan
std
mcan
std
mcan
std
mcan
std
vanilla fine-tuning
64.11
1.33
90.80
1.77
70.69
2.83
89.92
0.61
78.88
1.64
Mixout
64.42
1.51
91.31
1.08
72.05
1.67
90.39
0.57
79.54
1.21
R3F
64.62
1.38
91.63
0.93
70.75
1.76
89.92
0.61
79.23
1.17
R-Dropout
64.14
1.58
91.87
0.78
70.24
2.83
90.25
0.49
79.13
1.42
DHILD-TUNING
64.85
1.32
91.52
0.81
71.69
1.95
90.42
0.44
79.62
1.13
Re-init
64.24
2.03
91.61
0.80
72.44
1.74
90.71
0.14
79.77
1.18
DPS Dense
64.98
1.08
91.50
0.83
73.
1.97
90.51
0.55
80.03
1.11
DPS Mix
65.11
0.63
91.78
0.74
73.46
1.46
90.47
0.55
80.21
0.84
Table 1: We report the mean and standard deviation results of 10 random seeds. Avg. represents the average score of the four tasks, the best results are bolded. The higher the mean value, the better the effect; the smaller the standard deviation, the more stable the performance. Since R3F is not applicable to the regression task (STS-B), we use the results of vanilla fine-tuning as the results of R3F STS-B.
3.4 Results on Out-of-Domain Generalization
Previous research has found that models trained annotated datasets create superficial shortcut features and do not generalize well to out-of-domain datasets [Belinkov et al. 2019). Since DPS possesses strong and stable performance, features and achieve better generalization in transferring to out-of-domain datasets or other related tasks. In detail, to evaluate out-of-domain generalization, we fine-tune BERT, LARGE medium- sized 8K subsampled MNIJ and SNLII datasets respectively, and directly evaluate the fine-tuned models on several NLI datasets, including MNLI, MNLI-M, SICK, SciTail, SNLI, and RTE. The experimental results are shown in Table 2 On the models trained on MNLI, DPS Dense/DPS mix outperforms vanilla fine-tuning by 1.16 average score, and consistently outperforms CHILD- TUNINGD by 0.32/1. 1.05 average score. On the models trained on SNLI, although CHILD-TUNING, only brings a very slight improvement over vanilla fine-tume mun-tnnnng. DPS Dense/DPS mix consistently delivers large improvement of 1.23/0.63 average score. All the above statistics can fully verify that DPS can better explore deeper semanties and reduce superficial biases unique to source task than vanilla fine-tuning and CHILD-TUNING,n We also conclude that making the model closer to real distribution for the source task may facilitate generalization ability better than freezing some parameters pre- pre-trained weights like CHILD-TUNING,,
also expect DPS can help to discover deeper semantic

---

Datasets
MNLI
SNLI
vanilla
CTp
DPS Dense
DPS Mix
vanilla
CTp
DPS Dense
DPS
MNLI
77.51
78,01
77.59
77.75
66.43
67.26
67.32
66.81
MNLI-M
78.75
79.13
79.03
79.16
68.07
68.75
69.08
68.47
SICK
51.91
55.69
55.76
58.18
53.67
53,46
56,23
55.49
SciTail
79.70
79.86
79.35
80.36
78.22
77.17
78.42
77.80
SNLI
72.80
73.36
74.28
74.72
84.87
84.41
84.74
84.83
RTE
71.44
70.87
72.80
73.16
67.35
67.47
70.28
69.07
Avg.
72.01
72.82
73.14
73.87
69.78
69.79
71.01
70.41
Table All models are trained on MNLI or SNLI and evaluated on the out-of-domain datasets. We report the average results of three random seeds. C represents
3.5 Results on Low-Resource Scenarios
To further explore the generalization ability of DPS extremely small datasets, we downsample eight datasets in GLUE to 0.5K and IK, and fine-tune them all on BERT LARGE- As illustrated Table 3 DPS outperforms vanilla fine-tuning by large margin almost all tasks in variable low-resource scenarios, which demonstrates that DPS, as a novel fine-tuning technique, can effectively mitigate the risk of overfitting, exploit model potential, and remove irrelevant features.
0.5K
1K DPS Dense
vanilla
DPS Dense
DPS Mix
vanilla
DPS Mix
Datasets
mean
std
mean
std
mcan
std
mcan
std
mean
std
mean
std
CoLA
26.01
13.2
39.42
11.8
40.58
10.6
47.97
5.62
52.89
2.50
53.34
2.35
MRPC
82.96
0.84
83.29
0.78
83.33
0.56
85.34
1.30
85.90
0.86
85.40
1.20
RTE
58.30
4.90
58.34
2.90
59.13
3.49
62.60
3.46
64.44
1.76
64.77
1.66
STS-B
81.77
2.69
83.38
1.55
83.60
1.18
85.86
1.34
87.15
1.77
87.02
1.04
SST-2
86.51
6.48
89,38
0.52
89.07
0.38
90,28
0.55
90.92
0.64
90.84
0.58
ONLI
76.82
2.09
77.09
1.75
76.73
1.92
81.61
1.32
82.58
1.19
82.66
1.26
QQP
71.68
1.90
74.43
1.35
74.20
1.42
76.96
1.50
77.81
0.57
77.71
0.31
MNLI
42.81
4.49
46.61
2.70
46.28
2.64
54.93
5.94
58.33
3.56
58.53
3.27
AVG
65.85
4.57
68.99
2.92
69.12
2.78
73.19
2.62
75.00
1.61
75.04
1.45
Table 3:
Comparison between DPS and vanilla fine-tuning with varying low-resource scenarios (K), We report the results of 10 random seeds and the best result of each task is bolded.
Analysis and Study
4.1
Effect of Suunnewoork Size
[Lee et al. 2020 have explored the apper-parameter p and verified that Mixout shows great perfor- mance at many different probabilities p (Note that the larger p. the smaller the subnetwork). We also wonder what will happen when DPS probabilities vary, we plot the validation performance distribution of CoLA and RTE of 10 random seeds when fine-tuning BERTHARGR LARGE across different Firstly, it is easy find the overall performance across all probabilities, which demonstrates fine-tuning subnetworks of the pre-trained model adaptively can better optimize the model than fine-tuning an unchanged subnetwork or fine-tuning random subnetworks. Secondly, compared vanilla fine-tuning, there is visible performance drop in CHILD-TUNING, and Mixout with some probabilities. However, DPS Dense/DPS Mix consistently outperforms vanilla fine-tuning across all probabilities. These fully indicate that to capture important parameters in some sizes of the subnetwork. resulting in failed runs (we refer to the average scores lower than vanilla as failed runs). However, Our proposed DPS can better capture more task-relevant parameters, which in turn results in consistent and stable improvement in all sizes of the subnetwork. To further justify the effectiveness of DPS from another perspective, we conduct case study in Appendix A.
probabilities p 10.1, 0.2, 0.3, 0.4, 0.5) in Figure 2 of DPS Dense/DPS Mix better than CHILD- SUI.D--TNINGG/MMEN
and Mixout may fail
7

---

67
74
Val Val
ACC
72
63
62
mis
Dunce
0.2
0.3
0.4
0.1
0.3
0.6
0.5
Prohubility
Probability
(a) CoLA
(b) RTE
Figure 2: Validation performance distribution of 10 random seeds when fine-tuning BERT LARGE with vamilla fine-tuning, Mixout, CHILD-TUNING, and DPS. Solid circles represent mean results, hollow circles represent outliers.
4.2
Results on Different Pre-Trainee Language Models
CoLA
MRPC
RTE
STS-B
Avg.
Models
Methods
mean
std
mean
mean
mean
mcan
std
vanilla
64.11
1.33
90.80
1.77
70.69
2.83
89,92
0.61
78,88
1.64
BERT
DPS Dense
64.98
1.08
91.50
0.83
73.14
1.97
90.51
0.55
80.03
1.11
DPS Mix
65.11
0.63
91.78
0.74
73.46
1.46
90.47
0.55
80.21
0.84
vanilla
66.82
1.16
92.96
0.41
85.61
1.02
92.31
0.12
84,43
0.68
RoBERTa
DPS Dense
68.05
1.25
93.01
0.33
86.02
2.26
92.62
0.16
84,93
1.00
DPS Mix
68.81
1.33
93.27
0.27
85.74
1.16
92.57
0.17
85.10
0.73
vanilla
61.34
1.83
92.58
0.43
82.84
1.22
91.89
0.25
82.16
0.93
BART
DPS Dense
64.56
1.92
92.54
0.52
85.28
0.87
92.38
0.22
83.69
0.88
DPS Mix
64.37
1.21
92.88
0.39
85.31
0.69
92.44
0.20
83.75
0.61
vanilla
65.83
14.5
93.00
0.42
89.25
0.88
90.36
6.56
84.66
5.59
ELECTRA
DPS Dense
69.25
2.98
93,95
89.45
0.83
92.69
86.34
1.12
DPS Mix
70.41
2.01
93.58
0.33
90.03
0.76
92.65
0.28
86.67
0.84
vanilla
65.51
1.54
92.60
0.69
85.71
1.26
91.60
0.44
83.86
0.98
DeBERTa
DPS Dense
67.38
0.93
93.07
0.22
86.86
0.91
92.25
0.40
84.89
0.62
DPS Mix
67.07
0.84
92.85
0.34
86.91
0.77
92.21
0.37
84.76
0.58
Table We compare DPS and vanilla fine-tuning on five widely used PLMs and report the mean and standard deviation results of 10 random seeds. Avg. represents the average score of the four tasks, the best results are bolded.
In this subsection, we conduct experiments on the four tasks in GLUE on five different Pre-trained Language Models (PLMs): BERT, LARGE (Devlin al.) 2018 RoBERTa LAGE [Liu al. 2019).
BART LARGE Lewis et al. 2019). ELECTRALARGE [Clrk et al. 2020 and DeBERTaLARCE He al. 2021]. The experimental results are shown Table In addition the most classical model
BERT. DPS consistently achieves better results on both RoBERTa and ELECTRA trained on larger
pre-trained corpus and better pre-training tasks. DPS also improves large magnitude on models with various structures: DeBERTa (relative position encoding model) and BART (encoder-decoder model).
These fully demonstrate that DPS can bring a stable improvement regardless of model structure and quality.

---

4.3 DPS with Sufficient Number of Training Examples
In addition to few-sample datasets, we believe it is also worth exploring whether DPS can deliver a boost with larger training sets. Therefore, we fine-tune BERTLARGE on two datasets GLUE with training samples over 50K, including SST-2 and QNLI Since it has been stable fine-tune LARGE with a sufficient number of training examples Devlin et al. [2018), Phang et al. [0188] we focus the expressiveness of the above two tasks. As the experimental results in the Table noted, when the training samples are large enough, DPS not only doesn't damage performance but also brings certain improvements compared other subnetwork optimizations. This indicates that our proposed dynamic parameter selection algorithm suitable for a wider range of data scenarios, except for few-sample scenarios, and thus is superior than previous subnetwork optimizations.
Methods
SST-2
QNLI
vanilla fine-tuning
93.23 (93.69)
92.31 (92.54)
Mixout
93.45 (93,80)
92.21 (92.45)
CHILD-TUNING
89.28 (94.04)
92.36 (92.58)
DPS Dense
93.77 (94.49)
92.61 (92.94)
DPS Mix
93.71 (94.16)
92.45 (92.72)
Table 5:
Comparison between several fine-tuning methods and DPS with a sufficient number of training examples datasets. We report mean (max) development scores of 10 random seeds, and the best results are bolded.
4.4 Time Usage
Methods
Vanilla
Mixout
R3F
R-Dropout CHILD-TUNING,, DPS Dense
DPS Mix
Time Usage
x1.00
xL.18
xL.64
x1.64
x3.13
xl.12
xl.30
Table 6: Time usage for DPS and multiple fine-tuning regularization methods.
In this subsection, we investigate the computational efficiency of DPS. Specifically, we count the time usage of DPS in fine-tuning the BERT LARGE on RTE and compare time usage with various fine-tuning regularization methods, following [Lee al. 2020]. We can notice from Table o that although DPS introduces slight additional computational costs than vanilla fine-tuning, DPS is about 25% much faster than R3F and R-Dropout, and about 240% much faster than CHILD-TUNING.. We also discuss memory consumption for above methods in Appendix F.
Conclusion
In this work, we propose a family of new fine-tuning techniques to fine-tune large-scale PLMs adaptively: DPS Dense and DPS Mix, which respectively optimize full network update and stochastic parameter update processes. Our methods outperform various fine-tuning approaches in terms of stability and overall performance while saving computational costs a lot, and consistently improve the generalization ability of fine-tuned models by a large margin. Extensive results and analysis show that DPS can enhance largely on five different pre-trained language models meanwhile possessing extremely stable performance across wider range of subnetwork sizes. We believe that training neural networks adaptively may have broader application scenarios. We leave this area of exploration for future work.
6 Acknowledgements
We thank all reviewers for their constructive comments. This research is supported by the National Key R&D Program under Grant No. No.20222D0110300 the National Natural Science Foundation of China under Grant No. 62072007, 62192733, 61832009, 62192731, 62192730.

---

References
Alessandro Achille, Giovanni Paolini, and Stefano Soatto. Where is the information in a deep neural network? arXiv preprint arXiv: 12213, 2019.
Armen Aghajanyan, Akshat Shrivastava, Anchit Gupta, Naman Goyal, Luke Zettlemoyer, and Sonal Gupta. Better fine-tuning by reducing representational collapse. In ICLR, 2021.
Shun-ichi Amari. Neural learning in structured parameter spaces-natural riemannian gradient. In NeurlPS, 1996.
Yonatan Belinkov, Adam Poliak, Stuart M Shieber, Benjamin Van Durme, and Alexander M Rush. On adversarial removal of hypothesis-only bias in natural language inference. In StarSem, 2019.
Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. The fifth pascal recognizing textual entailment challenge. In TAC, 2009.
Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. A large annotated corpus for learning natural language inference. EMNLP, 2015.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et Language models are few-shot learners. In NeurlPS, 2020.
Daniel Cer, Mona Diab, Eneko Agirr, Inigo Lopez-Gazpis and Lucia Specia. Semeval-2017 task 1: Semantic textual similarity-ullilinnaal and cross-lingual focused evaluation. arXiv preprint arXiv: 1700v:770.000000 00055, 2017.
Kevin Clark, Thang Luong, Quoc Le, and Christopher D Manning. Electra: Pre-training text encoders discriminators rather than generators. In ICLR, 2020.
Elliot J Crowley, Jack Turner, Amos Storkey, and Michael O' O'Boyle. Pruning neural networks: time to nip it in the bud? 2018.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL, 2018.
Ming-Wei
Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, and Noah Smith. Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping. arXiv preprint arXiv:2002.06300, 2020.
Bill Dolan and Chris Brockett. Automatically constructing corpus of sentential paraphrases. In
IWP, 2005.
Ronald Fisher. On the mathematical foundations of theoretical statistics. Philosophical transactions of the Royal Society of London. Series A, containing papers of mathematical or physical character, 2226594-604;:30993688 1922.
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Decoding-enhanced bert with disentangled attention. In ICLR, 2021.
Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classification. In ACL, 2018.
Tushar Khot, Ashish Sabharwal, and Peter Clark. Seitail: A textual entailment dataset from science question answering. In AAAI, 2018.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka catastrophic forgetting in neural networks. In PNAS, 2017.
et al. Overcoming
Cheolhyoung Lee, Kyunghyun Cho, and Wanmo Kang. Mixout: Effective regularization to finetune large-scale pretrained language models. In ICLR, 2020.
10

---

Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising pruuence-o--equeene pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv: 1910. 13461, 2019.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 11692, 2019.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization (2017). arXiv preprint arXiv:1711.05101, 2019.
Rabeeh Karimi Mahabadi, Yonatan Belinkov, and James Henderson. Variational information bottle- neck for effective low-resource fine-tuning. In ICLR, 2021.
Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zamparelli. A sick cure for the evaluation of compositional distributional semantic models. In LREC, 2014.
James Martens. New insights and perspectives on the natural gradient method. arXiv preprint arXiv:1412.1199 2014.
Marius Mosbach, Maksym Andriushchenko, and Dietrich Klakow. On the stability of fine-tuning bert: Misconceptions, explanations, and strong baselines. In ICLR, 2021.
Razvan Pascanu and Yoshua Bengio. Revisiting natural gradient for deep networks. arXiv preprint arXiv: 1301.3330 2013.
Jason Phang, Thibault Févry, and Samuel R Bowman. Sentence encoders on stilts: Supplementary training on intermediate labeled-data tasks. arXiv preprint arXiv: 1.01088, 2018. 8.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv: 10683, 2019.
Pranav Rajpurkar. Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. EMNLP, 2016.
Nikhil Dandekar Shankar Iyer and Kornel Csernai. First quora dataset release: Question pairs. https: /innuu... com/ y2y8u5ed 2017.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogmition. arXiv preprint arXiv:1409.1553 2014.
Sidak Pal Singh and Dan Alistarh. Woodfisher: Efficient second-order approximation for neural network compression. In NewrIPS, 2020.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In EMNLP, 2013.
Lucas Theis, Iryna Korshunova, Alykhan Tejani, and Ferene Huszár. Faster gaze prediction with dense networks and fisher pruning. arXiv preprint arXiv:1801.0577 2018.
Ming Tu, Visar Berisha, Martin Woolf, Jae-sun Seo, and Yu Cao. Ranking the parameters of deep neural networks using the fisher information. In ICASSP, 2016.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurlPS, 2017.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: multi-task benchmark and analysis platform for natural language understanding. In ICLR, 2019.
Alex Warstadt, Amanpreet Singh, and Samuel Bowman. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625-641, 2019.
11

---

Adina Williams, Nikita Nangia, and Samuel R Bowman, A broad-coverage challenge corpus for sentence understanding through inference. In NAACL, 2018.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi,
Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Transformers: State-of-the-ar natural language processing. In EMNLP, 2020.
Lijun Wu, Juntao Li, Yue Wang, Qi Meng, Tao Qin, Wei Chen, Min Zhang, Tie-Yan Liu, et al. R-drop: regularized dropout for neural networks. In NeurIPS, 2021.
Runxin Xu, Fuli Luo, Zhiyuan Zhang. Chuanqi Tan, Baobao Chang, Songfang Huang, and Fei Huang. Raise a child in large language model: Towards effective and generalizable fine-tuning, In EMNLP, 2021.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoe Le. XInet: Generalized autoregressive pretraining for language understanding. In NeurIPS, 2019.
Tianyi Zhang, Felix Wu, Arzoo Katiyar, Kilian Q Weinberger, and Yoav Artzi. Revisiting faw-semple bert fine-tuning. In ICLR, 2021.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? Our limitations lie in inducing additional
training cost and memory overhead. We have discuss this Sec. (.444..am and Appendix
(c) Did you discuss any potential negative societal impacts of your work? [No] We think our methods will not lead any negative societal impact.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes] We will upload the code in the supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] We provide the details in Sec. 3.4 Sec. 3.5| Appendix C and
Appendix D)
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] We report the running times below experimental tables in main text.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] Please refer to Appendix C.
4. If you are using existing assets (c.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] We provide the citations for each data and the url for open-souce code.
(b) Did you mention the license of the assets? [No] We use the publicly popular datasets for each experiments.
(c) Did you include any new assets either in the supplemental material or as a URL? [No] We do not provide new datasets.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A] We use public datasets.
12

---

(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A] We use public datasets.
5. If you used crowdsoureing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "q2S06-V0F5", "reviewer": "Reviewer_SWtE", "summary": "The motivation behind this paper leaves from current limitations of fine-tuning, namely fine-tuning instability and poor generalizability (particularly in OOD settings). To address these limitations, the authors propose a Dynamic Parameter Selection (DPS) algorithm, which fine-tunes pre-trained networks adaptively in two stages: 1) Stage I accumulates gradients of updated parameters during fine-tuning, and 2) Stage II derives a subnetwork composed of the important parameters to update based on the accumulated values in Stage I. The authors propose two versions of their algorithm, DPS Dense (which optimizes the entire network update process) and DPS Mix (which optimizes the stochastic parameter update process). \n\nThe authors conduct experiments in OOD setting and low-resource scenarios on GLUE and NLI datasets, showing their proposed method outperforms prior approaches. On the GLUE benchmark, DPS gains range between 0.44-1.33 points and 0.29-0.8- std. Moreover, the authors conduct ablation experiments where they analyze performance of DPS when varying the subnetwork size, using various pre-trained models, sufficient number of training examples and time duration. The authors conclude that their proposed DPS method leads to stable improvement in pre-trained model performance with sufficient number of training examples, is time-efficient and captures task-relevant parameters across a variety of subnetwork sizes.\n", "questions": "Line 175: “DPS Dense outperforms all baselines” - from the results in Table 1, DPS Dense does not seem to outperform all baselines, particularly on MRPC and STS-B datasets. While on average it does, I would recommend clarifying this aspect in the presentation of the results. \n\nLines 177-178: “DPS saves 177 additional computational overhead imposed by computing Fisher Information based on all training 178 samples”\n\nRegarding the results in Table 2 (Out-of-Domain Generalization), how do you explain that DPS Dense is performing better than DPS mix on the MNLI dataset, while results on the SNLI dataset seem to show the opposite trend that DPS Dense outperforms DPS Mix? What about the standard deviation?\n\nTable 4: Have any of the pre-trained models been trained on the datasets you are evaluating their performance on?\n\nSection 4.3: “two datasets in GLUE with training samples > 50K” - which is the motivation behind selecting datasets with > 50K in particular? The paper does not provide an explanation for your particular choice of > 50K.\n", "limitations": "The work seems largely incremental in nature, and the methods proposed by the authors are largely inspired by previous work in the literature. The authors nevertheless convincingly show the good performance of their method.\n\nThe paper lacks a comprehensive discussion of the limitations of the proposed methods, the computational overhead introduced. While Section 4.4 talks about the slight computational costs compared to vanilla fine-tuning, this is only in time usage and not in computational complexity.\n\nLine 80: typo - “Differ from fine-tuning whole network” (--> different)\n\n\n\nThe paper addresses a relevant problem on how to more efficiently fine-tune pretrained models, particularly in data-limited regime and out of distribution settings. The method proposed by the authors is inspired by previous work, nevertheless empirical results demonstrate it performs well. The paper would benefit from a more clear discussion of the limitations of the proposed approach, particularly in terms of the computational complexity and the need to first do fine-tuning before applying the proposed approached (DPS Dense), as well as the additional data structures needed to store the frequency of each parameter being update (DPS Mix). \n", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n- The intuition behind the idea that fine-tuning a subnetwork instead of the entire network can yield more relevant parameters for a specific downstream task is valuable and does make a lot of sense\n\n\n- The authors use an extensive set of baselines from the literature to compare their proposed method with, including Mixout (Lee et al, 2020), R3F (Aghanjanyan et al, 2021), R-Dropout (Wu et al, 2021), Child-tuning (Xu et al, 2021), and Re-init (Zhang et al, 2021)\n\n\n- The authors convincingly demonstrate that their method performs well in Out-of-distribution and low-resource scenarios, and the ablation experiments show that the proposed method works when combined with different pre-trained models\n\nWeaknesses: \n- DPS dense needs a first pass in Stage I that finetunes the whole network for gradient accumulation. While I understand the need for this pass, it seems wasteful, why would someone use DPS on top of fine-tuning the network? Unfortunately the authors do not discuss this limitation of their proposed method in the paper.\n\n\n- While DPS Mix is proposed as an alternative to optimize the fine-tuning process, it requires a Frequency Accumulation Matrix (FAM) to store the frequency of every parameter being updated, in addition to the Gradient Accumulation Matrix (GAM) required by DPS Dense. The authors do not discuss the computational bottleneck and complexity of these design choices. Moreover, Stage I of DPS Mix is identical to previously proposed methods in the literature (Mixout).\n\n\n- The authors are mainly comparing the obtained results with ChildTuning, without motivating why they focus particularly on this baseline method.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "OS_I_UlP90j", "reviewer": "Reviewer_sChX", "summary": "This paper proposed a new large-scale language model fine-tuning algorithm called Dynamic Parameter Selection (DPS). This algorithm essentially uses a two stage cyclic updating strategy, where in Stage-1 it accumulates in-batch gradients of updated parameters during fine-tuning, and in Stage-2, it derives a subnetwork based on the Stage-1 accumulated gradients and their relative importance. The paper has proposed two variants of the algorithm called DPS Dense and DPS Mix, and show that it consistently performs better than previous approaches on sub-network finetuning strategies. Also, the experimental results on the out-of-domain show the usefulness of DPS. \n", "questions": "1) In Table-1, what is the reason to choose only four tasks from GLUE and why only these tasks? \n2) Suggestion: Maybe it is better to merge Section-5 with Section-2. \n3) In all the ablations, comparison is based on vanilla fine-tuning, can you also add any sub-network optimization model as well? \n", "limitations": "N/A", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n1) The paper is well written and easy to follow. \n2) The proposed DPS model is interesting and novel with better results than other algorithms related to sub-network optimization.\n3) The results on out of domain generalization are very exciting. However, it is not clear why this approach leads to better generalization. \n4)  The ablations are very thorough and useful such as experiments on low resource scenario, effect of subnetwork size, different pre-trained language models, and DPS with sufficient number of training examples. \n\nWeaknesses: \n1) It would have been more interesting if the paper could show any theoretical guarantees or understanding of why this algorithm is superior over other sub-network optimizations. \n2) It would be interesting to check whether the observations hold for more than the current 4 tasks reported in Table-1. Is there any reason for not showing on a complete GLUE dataset?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "DK_5UdKnTVG", "reviewer": "Reviewer_BfEh", "summary": "Large pre-trained LMs form the baselines of many impressive results, but fine-tuning can, among other challenges, lead to overfitting and degradation of internal representations. This paper introduces Dynamic Parameter Selection, which selects subnetworks to perform staging updates based on gradients. The GLUE benchmark is used for evaluation, and results are positive. ", "questions": "- why are so many (8) references needed at L17 to support the idea that pre-trained LMs are useful in NLP?\n- while the limitations on pre-trained LMs in Sec 1 around overfitting and catastrophic forgetting are well known, how are the specific arguments (esp L34) related to those limitations?\n- much of the information stored in neural networks concerns the relative differences between weights in the network — does altering only a subpart of a network affect not only its internal structure but its relationship with other information elsewhere in the network?\n- As this approach is fairly generic, would it be straightforward to add vision datasets to those used in Sec 3.1?", "limitations": "Sec 4.4 is said to discuss limitations, though this is done only superficially, essentially only serving to further hype DPS.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "_Strengths_\n\n- the two-stage process of accumulating in-batch gradients during fine-tuning, and subsequently using those gradients to derive stage-specific subnetworks to update is conceptually simple, and therefore clear.\n- GLUE and the NLI datasets are appropriate, generic datasets. \n- The models are very competitive, though in many cases the best means are within the indicated std (Table 1) therefore some statistical significance testing (correcting for multiple comparisons) is advised, as little can be conclusively concluded otherwise. The multitude of experimental scenarios (e.g., OoD generalization, low-resource scenarios) is a positive.\n\n\n_Weaknesses_\n\n- some minor spelling and grammatical mistakes (e.g., ‘has been prove’, L28; ‘to randomly replaces’, L30; ‘same sized’, L85; CHID-TUNING, L87, L92; ‘will not be update’, fig 1 caption) should be addressed. Along these lines, Figures 1 and 2 need to have their text made larger, for readability. References should be checked for proper formatting\n- the treatment of related work (Sec 5) is quite superficial — more synthesis of the various approaches should be attempted.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "6Y5Edwvn7xM", "reviewer": "Reviewer_kBeJ", "summary": "This paper proposes a Dynamic Parameter Selection (DPS) algorithm to address the overfitting and catastrophic forgetting issues in fine-tuning pre-trained language models. \nThe proposed DPS consists of a cyclic two-stage updating strategy:\n1) accumulates the in-batch gradients of updated params during fine-tuning.\n2) using the accumulated values in stage 1 to derive a subnetwork and keep non-subnetwork params constant.\n\nThe experiments are mainly carried out on the classification tasks of GLUE. \nThe proposed DPS is tested under various settings, and the results show that DPS shows consistent improvements over various baseline systems, which demonstrates the effectiveness of DPS.", "questions": "### Major comments\n1. The experiments are only carried out on classification tasks. Did you apply your method to other tasks, such as machine reading comprehension (MRC), or named entity recognition (NER)?\n2. In Section 4.4, the authors have demonstrated that the proposed DPS is more efficient than other methods in time usage. How about memory consumption?\n3. line 187-188: Did you arbitrarily choose the size (8K) of subsampled MNLI/SNLI? \n4. It would be nice if the authors could explicitly discuss and compare two variants of DPS along with the experimental results. \n5. Section 3.5: If I understand correctly, all experiments are carried out on the same subsets (either 0.5k or 1k) of the tasks in GLUE. It would be nice if the authors could release what the exact subsets are sampled to enable the researchers to compare the results in the future.\n\n\n### Minor comments\n1. line 63: I know what is \"std,\" but please use its full name when it first appears in a formal paper.\n2. line 226: DEBERTA -> DeBERTa\n3. Figure 1: it seems that \"us\" is a single variable. Thus, \"2kus+i\" seems to be quite confusing (might be interpreted as 2*k*u*s). Please try to make the naming of the variable more clear.", "limitations": "The authors claim in the checklist that they have discussed their limitation in Section 4.4, noting the additional training cost of their method. However, the Section 4.4 seems not to be discussing their limitation but advantages over other methods. The authors are encouraged to discuss some \"true\" limitations of their method. For example, does your model consume more memory than other methods? Please try to take a balanced view in interpreting the limitation of your method.\n", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n1. The proposed method addresses an important topic of the overfitting and catastrophic forgetting issues in fine-tuning pre-trained language models.\n2. The formulation of the proposed DPS is technically sound and reasonable. \n3. The experiments are carried out in various aspects to fully demonstrate the effectiveness of the proposed model. \n\n\n### Weaknesses\n1. The experiments are only carried out on classification tasks. Using one more type of NLU task would have further strengthened the results.\n2. The discussion of the limitation is not informative.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept