Better Fine-Tuning by Reducing Representational Collapse
Authors: Armen Aghajanyan
Akshat Shrivastava
Anchit Gupta
Naman Goyal
Luke Zettlemoyer
Sonal Gupta
Venue: ICLR 2021 Poster (2021)
Keywords: finetuning
nlp
representational learning
glue

Abstract: Although widely adopted, existing approaches for fine-tuning pre-trained language models have been shown to be unstable across hyper-parameter settings, motivating recent work on trust region methods. In this paper, we present a simplified and efficient method rooted in trust region theory that replaces previously used adversarial objectives with parametric noise (sampling from either a normal or uniform distribution), thereby discouraging representation change during fine-tuning when possible without hurting performance. We also introduce a new analysis to motivate the use of trust region methods more generally, by studying representational collapse; the degradation of generalizable representations from pre-trained models as they are fine-tuned for a specific end task. Extensive experiments show that our fine-tuning method matches or exceeds the performance of previous trust region methods on a range of understanding and generation tasks (including DailyMail/CNN, Gigaword, Reddit TIFU, and the GLUE benchmark), while also being much faster. We also show that it is less prone to representation collapse; the pre-trained models maintain more generalizable representations every time they are fine-tuned.

Published as a conference paper at ICLR 2021
BETTER FINE-TUNING BY REDUCING REPRESENTA- TIONAL COLLAPSE
Armen Aghajanyan, Akshat Shrivastava, Anchit Gupta & Naman Goyal Facebook
{armenag, akshats, anchit, naman atb. com
Luke Zettlemoyer & Sonal Gupta Facebook {1sz, sonalqupta)eib.co com
ABSTRACT
Although widely adopted, existing approaches for fine-tuning pre- rre-traind lan- guage models have been shown be unstable across hyper-parameter settings. motivating recent work on trust region methods. This paper presents a simpli- fied and efficient method rooted in trust region theory that replaces previously used adversarial objectives with parametric noise (sampling from either nor- mal or uniform distribution), thereby discouraging representation change during fine-tuning when possible without hurting performance. We also introduce analysis to motivate the use trust region methods more generally, by studying representational collapse; the degradation of generalizable representations from pre-trained models as they are fine-tuned for a specific end task. Extensive exper- iments show that our fine-tuning method matches exceeds the performance of previous trust region methods on a range of understanding and generation tasks (including DailyMail/CNN, Gigaword, Reddit TIFU, and the GLUE benchmark). while also being much faster. We also show that it is less prone to representa- tion collapse; the pre-trained models maintain more generalizable representations every time they are fine-tuned.
INTRODUCTION
Pre-trained language models (Radford al. 2019 Devlin et al. 2018 Liu et al. 2019 Lewis al.
2019 2020) have been shown to capture wide array of semantic, syntactic, and world knowledge CCaak et al. 2019), and provide the defacto initialization for modeling most existing NLP tasks.
However, fine-tuning them for each task is a highly unstable process, with many hyperparameter settings producing failed fine-tuning runs, unstable results (considerable variation between random
seeds). over-fitting, and other unwanted consequences (Zhang et al. 22020] Dodge et al. 2020
Recently, trust region or adversarial based approaches, including SMART (Jiang et al. 2019) and FreeLB (Zhu et al., 2019), have been shown to increase the stability and accuracy of fine-tuning by adding additional constraints limiting how much the fine-tuning changes the initial parameters. However, these methods are significantly more computationally and memory intensive than the more commonly adopted simple-gradiens gradient-basee based approaches.
This paper presents lightweight fine-tuning strategy that matches or improves performance relative SMART and FreeLB while needing just fraction of the computational and memory overhead and no additional backward passes. Our approach is motivated by trust region theory while also reducing to simply regularizing the model relative to parametric noise applied to the original pre-trained representations. We show uniformly better performance, setting new state of the art for RoBERTa fine-tuning on GLUE and reaching state of the art on XNLI using no novel pre-training approaches (Liu et al. 2019; Wang al. 2018 Conneau et al. 2018). Furthermore, the low overhead of our family tine-tuning methods allows our method to be applied to generation tasks where we consistently outperform standard fine-tuning, setting state of the art summarization tasks.

---

Published as a conference paper at ICLR 2021
We also introduce new analysis to motivate the use of trust-region-stylle methods gener- ally, by defining a new notion representational collapse and introducing a new methodology for measuring it during fine-tuning. Representational collapse is the degradation of generalizable representations of pre-trained models during the fine-tuning stage. We empirically show that standard fine-tuning degrades generalizable representations through series of probing experiments on GLUE tasks. Furthermore, we attribute this phenomenon to using standard gradient descent algo- rithms for the fine-tuning stage. We also find that (I) recently proposed fine-tuning methods rooted in trust region, i.e., SMART, can alleviate representation collapse, and (2) our methods alleviate representational collapse to an even greater degree, manifesting in better performance across almost all datasets and models.
Our contributions in this paper are the following.
We propose a novel approach to fine-tuning rooted in trust-region theory, which we show directly alleviates representational collapse at a fraction of the cost of other recently pro- posed fine-tuning methods.
Through extensive experimentation, we show that our method outperforms standard fine- tuning methodology following recently proposed best practices from Zhang et al. (2020). We improve various SOTA models from sentence prediction to summarization, from mono- lingual to cross-lingual.
We further define and explore the phenomena of representational collapse in fine-tuning and directly correlate it with generalization tasks of interest.
LEARNING ROBUST REPRESENTATIONS THROUGH REGULARIZED FINE-TUNING
We are interested in deriving methods for fine-tuning representations that provide guarantees on the movement of representations, in the sense that they do not forget the original pre-trained represen- tations when they are fine-tuned for new tasks (see Section 4 for more details). We introduce a new fine-tuning method rooted in approximation trust region, which provides guarantees for stochastic gradient descent algorithms by bounding some divergence between model at update and t+1 (Pascanu Bengio 2013 Schulman al. 2015b Jiang et al. 2019).
Let f RRux + RP be a function which returns some pre-trained representation parameterized by 0f from m tokens embedded into fixed vector of size n. Let the learned classification head g Iº + Ra be function which takes an input from f and outputs valid probability distribu- tion parameterized by 0g in q dimensions and let X be our dataset. the case of generation, we can assume the classification head is simply an identity function or softmax depending on the loss function. Let L(O) denote loss function given by 0 = (0,
We are interested in minimizing L with respect to such that each update step is constrained by movement in the representational density space p(f). More formally given an arbitrary e
arg min L(O + A0)
26
(1)
s.t.
=c
This constrained optimization problem is equivalent to doing natural gradient descent directly over the representations (Pascanu Bengio 2013). Unfortunately, we do have direct access to the density of representations; therefore, it is not trivial to directly bound this quantity. Instead, we propose to natural gradient over g f with an additional constraint that g is most 1-L-isshhizz (which naturally constrains change of representations, see A. in the Appendix). Traditional computation of natural gradient is computationally prohibitive due to the need for inverting the Hessian. An alternative formulation of natural gradient be stated through mirror descent, using Bregmann divergences (Raskutti & Mukherjee 2015 Jiang et al., 2019).
This method primarily serves as robust regularizer by preventing large updates in the model's probability space. This family of methods is classically known as trust-region methods (Pascanu & Bengio 2013. SShulman al. 2015a).
2

---

Published as a conference paper at ICLR 2021
(f. = [()) + XE
sup KLs(g f(z) ll f(x^))
(2)
/ - =|sc
However, the supremum is computationally intractable. An approximation is possible by doing gradient ascent steps, similar to finding adversarial examples. This was first proposed by SMART with symmetrical KLs(X,Y) KL(X||Y) + KL(Y] term (Jiang et al. 2019).
We propose an even simpler approximation which does not require extra backward computations and better than SMART. We altogether remove the adversarial nature
empirically works well from SMART and instead optimize for a smoothness parameterized by KLs. Furthermore, we optionally also add a constraint on the smoothness of g by making at most I-Lipschitz, the intuition being if we can bound the volume of change in we can more effectively bound f.
= L(0)+ - +A [KLslg- f(x) g f(x + +))]
R3F Method
(3)
s.l. ~~ ~~,N00, of or u(-0,oo)
(4)
s.t. Lipig} <
Optional R4F Method
(5)
where KLg is the symmetric KL divergence and z a sample from a parametric distribution. In work we test against two distributions, normal and uniform centered around O. We denote this as the Robust Representations through Regularized Finetuning (R3F) method.
Additionally propose extension to R3F (R4F; Robust Representations through Regularized and Reparameterized Finetuning, which reparameterizes to be at most I-Lipschitz via Spectral Normalization (Miyato et al., 2018). By constraining g to be most 1-L-iscchtzz directly bound the change representation (Appendix Section [.I].. Specifically we scale all the weight matrices of g by the inverse of their largest singular values WeN W/o(W). Given that spectral radius ((ss)) = 1 we can bound Lip(g) < 1. In the case of generation, g does not have any weights therefore we can only apply the R3F method.
more
2.1 RELATIONSHIP TO SMART AND FREELB
Our method is most closely related to the SMART algorithm. which utilizes an auxiliary smoothness inducing regularization term, which directly optimizes the Bregmann divergence mentioned above in Equation 2 Jiang et al. 2019).
SMART solves the supremum by using an adversarial methodology to ascent to the largest KL divergence with an e-ball. We instead propose to remove the ascent step completely, optionally fixing the smoothness of the clas- sification head g. This completely removes SMART's adversarial nature and is more akin to optimizing the smoothness of g directly. Another recently proposed adversarial method for fine-tuning, FreeL.B optimizes di- rect ad adversarial loss LFreeLB (0) = sup A0]5 L( + Ad) through iterative gradient ascent steps. This is sim- ilar to SMART in the sense that both are adversarial and require gradient ascent steps. Unfortunately, the need for extra
FP
BP
xFP
FreeLB SMART
1 1 S
1+
+
3S
1
1
+ 3S
R3F/R4F
2
1
Standard 1
Table 1: Computational cost of recently proposed fine-tuning algorithms. We show Forward Passes (FP), Backward Passes (BP) as well computation cost as factor of forward passes (xFP). is the number of gradient ascent steps, with a minimum 2 1
passes can be prohibitively ex- pensive when fine-tuning large pre-trained models (Zhu) ctt al. (019)).
Our method is significantly more computationally effi-
cient than adversarial based fine-tuning methods, as seen in Table I We show that this efficiency match or exceed FreeLB and SMART on large number of tasks.
does not hurt performance; In addition, the relatively low costs our methods allow us to improve over fine-tur tuning on an array of generation tasks.
3

---

Published as a conference paper at ICLR 2021
3 EXPERIMENTS
We will first measure performance by fine-tuning range of tasks and languages. The next sections report why methods rooted in trust region, including ours, outperform standard fine-tuning. We aimed for fair comparisons throughout all of our experiments by using fixed budget hyper- parameters searches across all methods. Furthermore, for computationally tractable tasks, we report median/max numbers well show distributions across large number of runs.
3.1 SENTENCE PREDICTION
GLUB
We will first test R3F and R4F on sentence classification tasks from the GLUE benchmark (Wang)
ct al. 2018).
We select the same subset of GLUE tasks that have been reported by prior work this
space
Jiang al. 2019): MNLI (Williams et al. 2018), QQP (Iyer et al., 2017). RTE (Bentivogli 2009
ct al.
QNLI (Rajpurkar et 2016), MRPC (Dolan & Brockett 2005
CoLA (Warstadt
al.
2018),
SST-2 (Socher et al. 2013)
Consistent with prior work @Jiag et al. 2019 Zhu et al. 2019), we focus on improving the perfor-
mance of RoBERTa-Large based models in the single-task setting (Liu et al. 2019). We report the performance of all models on the GLUE development set.
We fine-tune each of the GLUE tasks with four methods: Standard (STD), the traditional fine- tuning scheme done by RoBERTa (Liu al.) 2019); Standard++ (STD++), a variant of stan- dard fine-tuning that incorporates recently pro- posed best practices for fine-tuning, specifically longer fine-tuning and using bias correction in Adam (Zhang et al. 2020); and our proposed methods R3F and R4F. We compare against the numbers reported by SMART, Freel.,, and RoBERTa on the validation set. For each method, applied hyper-parameter search with equivalent fixed budgets per method. Fine-tuning each task has task-specific hyper- parameters described in the Appendix (Sec- tion A.2 After finding the best hyperparam- eters, we replicated experiments with optimal parameters across ten different random seeds. Our numbers reported are the maximum of 10 seeds to be comparable with other benchmarks in Table
SST-2 Walltime Analysis
9000
Finetuning Method
8000
Standard+*
Seconds
SMART
7000
R3F
R3F ~ N
6000
R4F
R4F
5000
Method
Figure I: Empirical evidence towards the compu- tational benefits of method present train- ing wall time analysis on the SST-2 dataset. Each method includes violin plot for 10 random runs. We define wall-time as the training time in sec- onds best checkpoint.
In addition to showing the best performance, also show the distribution of various meth- ods across ten seeds to demonstrate the stability properties of individual methods in Figure
R3F and R4F unanimously improve over Standard and Standard++ fine-tuning. Furthermore, our methods match or exceed adversarial methods such as SMART/FreeLB a fraction the computa- tional cost when comparing median runs. We show computational cost in Figure I for single task, but the relative behavior wall times is consistent across all other GLUE tasks. We note that we could not find a discernable difference in the experimental setting, which would make the selection between R3F vs. R4F trivial.
We do not test against STS-B because it is regression task where our KI. divergence is not defined Cer 2017
4

---

Published as a conference paper at ICLR 2021
MRPC
SST-2
0.9756
0.92
0.9150
0.9725
Finetuning Method
0.9100
0.88
0.9675
0.9075
0.9050
-U
0.9050
0.84
RAP -V
R4F +
0.9000
0.52
0.9575
Method
Method
Figure 2: We show the results of our method against Standard++ fine-tuning and SMART across tasks. Across 10 random seeds both max and median of our runs were higher using our method than both SMART and Standard++.
MNLI Acc-m//m
QQP Acc/F1
RTE
QNLI
MRPC
CoLA
SST-2 Acc
MNLI Acc-m/mm
QQP
RTE
QNLI Acc
MRPC
CoLA
SST-2
Acc
Ace
Acc
Mee
Acc/F1
Acc
Acc
Moc
Acc
STD
90.2/-
92.2/-
86.6
94.7
89.1
68.0
96.4
90.2/-
91.9/-
86.6
92.1
84.4
66.2
96.4
STD++
91.01-
92.2/-
87.4
94.8
91.1
69.4
96.9
90.8/
92.1/-
87.4
92.5
89.1
68.4
96.9
FreeLB
SMART
91.1/1.33
92.4/89.8
92.0
95.6
89.2
70.6
96.9
90.85/91.10
91.7/88.2
89.5
94.8
83,9
69.4
96.6
R3F
91.1/9..3
92.4/89.9
88.5
95.3
91.6
71.2
97.0
91.10/11.0
92.1/88.4
88.4
95.1
91.2
70.6
96.2
R4F
90.1/90.8
92.5/89.99 9 88.8
95.1
90.9
70.6
97.1
90.0/90.6
91.8/88.2
88.3
94.8
90.1
70.1
96.8
Table 2: We present our best results on the GLUE development set for various fine-tuning methods applied the RoBERTa Large model. On the left side table, we present our best numbers and numbers published in other papers. On the right side, we present median numbers from 10 runs for the mentioned methods.
XNLI
We hypothesize that staying closer to the original representations especially crucial for cross- lingual tasks, especially in the zero-shot fashion where drifting away from pre-trained representa- tions for single language might manifest in loss of cross-lingual capabilities. In particular, we take look at the popular XNLI benchmark, containing 15 languages (Conneau et al. 2018). We compare our method against the standard trained XLM-R model in the zero-shot setting (Conneau
al. 2019)
Model
fr
el
th
zh
hi
Avg
XLM-R Base
85.8
79.7
80.7
78.7
77.5
79.6
78.1
74.2
73.8
76.5
74.6
76.7
72.4
66.5
68.3
76.2
XLM-R Large
83.9
82.9
84.0
81.2
79.6
76.9
73.9
73.8
+ R3F
89.4
84.2
85.1
83.7
83.6
84.6
82.3
80.7
80.6
79.4
80.1
77.3
72.6
74.2
81.2
+ R4F
89.6
84.7
85.2
84.2
83.6
84.6
82.5
80.3
80.5
80.9
79.2
80.6
78.2
72.7
73.9
81.4
InfoXLM
89.7
84.5
85.5
84.1
83.4
84.2
81.3
80.9
80.4
80.8
78.9
80.9
77.9
74.8
73.7
81.4
Table To remain consistent with prior experiments, we report an average runs of zero-shots results on the XNLI test set for our method applied to XLM-R Large. Various versions of our method win over the majority of languages. The bottom row shows the current SOTA on XNLI, which requires the pre-training of novel model.
We present our result in Table 3 R3F and R4F dominate standard pre-training on 14 out of the 15 the best known XLM-R XNLI results reaching
languages in the XNLI task. R4F improves SOTA with an average language score 81.4 across five runs. The current state the art, INFO- XLM required novel pre-training method to reach the same numbers (Chi al. 2020
5

---

Published as a conference paper at ICLR 2021
CNN/DailyMail
Gigaword
Reddit TIFU (Long)
Random Transformer
38.27/15.03/35.44 48
35.70/16.75/3..83
15.89/1 04.89/1.94/1..2 94/11.22
BART
44.16/21.28/40.999
39.29/20.09/35.63
24.19/8. 12/21.31
PEGASUS
44.17/21.47/41.11
26.63/9.01/21.06
ERNIE-GEN
44.02/21.17/41.26
39.25/ 20.25/36.53
ProphetNet (Old SOTA)
44.20/21.1744..33
39,51/20.42/36.66
BART+R3F (New SOTA)
44.38/21.53/1..17
40.45/20.69/36.56
30.31/10.98/24.44
Table 4: Our results various summarization data-sets. We report Rouge-1, Rouge-2 and Rouge-L. per element in table. Following PEGASUS, we bold the best number and numbers within 0.15 of the best.
3.2 SUMMARIZATION
While prior work in non-standard finetuning methods tends to focus sentence prediction and GLUE tasks (Jiang et al. (2019) Zhuu et al. 2019; 2019) /Zhang et al., 2020), we look to improve abstractive summarization, due its additional complexity and computational cost, specifically we look at three datasets: CNN/Dailymail (Hermann et al. 2015), Gigaword (Napoles et al. 2012) and Reddit TIFU
(Kim et al. 2018).
Like most other NLP tasks, summarization recently has been dominated by the fine-tuning of large pre-trained models. For example, PEGASUS explicitly defines pre-training objective to facilitate the learning of representations tailored to summarization tasks manifesting in ofte-of-the-ee art per- formance on various summarization benchmarks (Zhang et al. 2019). ProphetNet (Yan et al. 2020) improved over these numbers by introducing their own novel self-supervised task as did ERNIE-
GEN (iiao et al. 2020).
Independent of the pre-training task, standard fine-tuning on downstream tasks follows a simple formula of using label smoothing loss while directly fine-tuning the whole model without adding any parameters. We propose the addition of the R3F term directly to the label smoothing loss. We note that R4F cannot be applied directly to generation tasks due to its reparameterization nature.
We present our results in Table 4 Our method (R3F) outperforms standard fine-tuning across the board for three tasks across all of the ROUGE metric variants. Notably, we improve Gigaword and Reddit TIFU ROUGE-1 scores by point and four points, respectively.
REPRESENTATIONAL COLLAPSE
Catastrophic forgetting, proposed initially as catastrophic interference, is phenomenon that curs during sequential training where new updates interfere catastrophically with previous updates manifesting in forgetting of particular examples for fixed task (McCloskey & Cohen 1989). Catas- trophic forgetting has been historically associated with continuous learning, and recent work (Mos-] bach al., 2020) showed that catastrophic forgetting concerning the original MLM objective is detrimental for end task training. Instead, the issue lies in optimization. Inspired by this work, we explore the related problem of representational collapse, the degradation of generalizable repre- sentations of pre-trained models during the fine-tuning stage. This definition is independent of specific fine-tuning task but is rather over the internal representations generalizability over large union of tasks. Another view of this phenomenon is that fine-tuning collapses the wide range of information available the representations into a smaller set needed only for the immediate task and particular training set.
Measuring such degradations is non-trivial. Simple metrics such as the distance between pre-trained representations and fine-tuned representations are not sufficient (e.g., adding constant to the pre- trained representations will not change representation power, but will change distances). One ap- proach would be estimate mutual information of representations across tasks before and after fine- tuning, but the estimation of mutual information is notoriously hard, especially in sigh-dimessions (Tschannen al. 2019). We instead propose series of probing experiments meant provide us

---

Published as a conference paper at ICLR 2021
0.660
0,580
0.760
0.650
0.570
Accuracy
Accuracy
Accuracy
0,755
0.752
0.540
0.947
0.530
0.610
0,765
0.400
0.742
QQP
0.730
0.76.
0.620
0.725
0.720
0.600
Accuracy
Accuracy
Accuracy
0.715
0.752
0.580
0.710
0.560
0.705
0.540
0.700
0.745
0.685
0.742
0.690
Figure 3: Results from probing experiments comparing proposed algorithms R3F, R4F to standard fine-tuning. Variants of our method consistently outperform past work.
with empirical evidence of the existence of representation collapse the GLUE benchmark (Wang ct al. 2018).
4.1
PROBING EXPERIMENTS
PROBING GENERALIZATION FINE-TUNED REPRESENTATIONS
To measure the generalization properties of various fine-tuning methodologies, we follow probing task and then fine-
methodology by first freezing the representations from the model trained tuning a linear layer on top of the model for another task. Doing this form of probing can directly measure the quality of representations learned by various fine-tuning methods and how much they collapse when fine-tuned on a sequence of tasks.
In particular, fine-tune RoBERTa model SST-2 and train linear layer for six other GLUE tasks, respectively. Our results are shown in Figure Appendix AA2 presents the hyperparameters. Across all tasks, of the two variants of method performed best across various fine-tuning methods.
Conversely, standard fine-tuning produced rep- resentations that were worse than other fine- tuning methods across the board, hinting at the sub-optimality of standard fine-tuning. Further- more, R3F/R4F consistently outperforms the adversarial fine-tuning method SMART.
0.96
0.94
Accuracy
0.92
Finetuning Method
0.90
R4F
Standard+
0.88
PROBING REPRESENTATION DEGRADATION
0.86
To show the effect of representation collapse, we propose an experiment to measure how the fine-tuning process degrades representations by sequentially training on a series of GLUE tasks. We arbitrarily select 3 GLUE tasks (QNLI, QQP, and RTE) and a source task (SST-2). We begin by training model on source task and then train on QNLI, QQP, and RTE in a se- quential order using the best checkpoint from
0.84
SST-2 QNLI
QQP RTE
Figure 4: We show the results of the chained prob- ing experiments. We do not show the distribu- tional properties of the runs because there was minimal variance in the results.

---

Published as a conference paper at ICLR 2021
the prior iteration. At each point in the chain, probe the source task and measure performance. We compare standard SGD with the best trust-region fine-tuning approach (R4F). Our results are depicted in Figure 4
As we can see with the standard fine-tuning process, our model diverges from the source task re- sulting in lower performance probes; however, with method, the probes change much less with sequential probing resulting in better probing and end performance.
PROBING REPRESENTATION RETENTION
To further understand representational collapse's impact, we extend our probing experiments to train cyclic chain of tasks. We showed that traditional fine-tuning degrades representations during the fine-tuning process in our prior experiments, meaning standard fine-tuning learns poorer represen- tation compared to alternative fine-tuning methods. The dual to looking at degradation is to look at the retainment learned representations. To do this, we take a look at cyclic sequential probing. Sequential probing involves training model task A, probing on and probing task C, and so forth. We then create a cyclic chain
then training model fine-tuned
B
+
B C
Cycle 1
Cycle
from where we compare tasks via their probe performance at each cycle.
We expect probing performance to increase at every cycle; since every cycle, the task we are probing on will undergo full fine-tuning. What are interested in is the level of retention in representa- tions after the fine-tuning. Specifically, we hypothesize that our method, specifically R4F, will retain representations significantly better than the Standard++ fine-tuning method.
In our experiments we consider the following sequence of GLUE tasks: SST-2 - QNLI + QQP RTE. We defer hyperparameter values Appendix (Section A.2
Proting
Probing GMU
Proting DGP
Proting RTT
2.00
0.94
9.85
0.725
0.92
1.0
0.81
0.675
0.30
5.75
0.550
Fhrthning Herhed
R4r Sancarte
0.85
3.70
0.825
0.86
2.55
0,80
0.500
0.54
3,60
0.500
Odte
2
Gyde
Sydle
Cyds
Gyde
Gyde
Cyce3
Gyde
Cyce
Cycle
Figure 5: We present the results of cyclical sequential probing for 3 cycles.
Looking at Figure 5 we see that R4F retains the quality of representations significantly better than standard fine-tuning methods.
CONCLUSION
We propose family of new fine-tuning approaches for pre-trained representations based on trust- region theory: R3F and R4F. Our methods are more computationally efficient and outperform prior work in fine-tuning via adversarial learning (Jiang et al.) 2019; Zhu et al.) 2019). We show that this is due to a new phenomenon during fine-tuning: representational collapse, where representations learned during fine-tuning degrade, leading to worse generalization. Our analysis shows that stan- dard fine-tuning is sub-optimal when it comes to learning generalizable representations, and instead, our methods retain representation generalizability and improve end task performance.
With our method, we improve upon monolingual and multilingual sentence prediction tasks as well as generation tasks compared to standard and adversarial fine-tuning methods. Notably, we set state of the art Dai//NN,, Gigaword, Reddit TIFU, improve the best-known results fine- tuning RoBERTa on GLUE, and reach state of the art on zero-shot XNLI without the need for any new pre-training method.
8

---

Published as a conference paper at ICLR 2021
We note there many flavors of RXF that can occur with various noise distributions perturbation strategies. We believe a larger. more general framework exists which connects trust region methods and fine-tuning in general. We leave this area of exploration for future work.
REFERENCES
Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. The fifth pascal recognizing textual entailment challenge. In TAC, 2009.
Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio and Lucia Specia. Semeval-2017 task
I: Semantic textual momillarty-mulilingeen and cross-lingual focused evaluation. arXiv preprint arXiv: 170500005, 2017.
Zewen Chi, Li Dong. Furu Wei, Nan Yang, Saksham Singhal, Wenhui Wang, Xia Song, Xian-Ling Mao, Heyan Huang, and Ming Zhou. Infoxlm: An informattin--heoorticc framework for cross- lingual language model pre-training, 2020.
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D Manning. What does bert look 2019.
at? an analysis of bert's attention. arXiv preprint
Alexis Conneau, Guillaume Lample, Ruty Rinott, Adina Williams, Samuel R Bowman, Holger Schwenk, and Veselin Stoyanov. Xnli: Evaluating cross-lingual sentence representations. arXiv preprint 1rXiv:1809.008 05053, 2018.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Un- 2019.
supervised cross-lingual representation learning at scale. arXiv preprint
Jacob Devlin, Ming- Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 1 04805, 2018.
Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannanch Hajishirzi, and Noah Smith. Fine-tuning pretrained language models: Weight initializations. data orders, and early stopping. arXiv preprint arXiv:2002.06:3 0000000000060......0 2020.
William B Dolan and Chris Brockett Automatically constructing corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005), 2005.
Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. In Advances in neural information processing systems, pp. 1693-1701, 2015.
Shankar
Tyer,
Nikhil Dandekar, and
Kornel Csernai.
First
quora
dataset
release:
Question
pairs, 2017.
URL
attps://data.com quora. com/
First Quora
Dataset- Release- Question. Pairs
Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Tuo Zhao. Smart: Robust and efficient fine-tuning for pre-trained natural language models through principled regu- 2019.
larized optimization. arXiv preprint
Byeongchang Kim, Hyunwoo Kim, and Gunhee Kim. Abstractive summarization of reddit posts with multi-level memory networks. arXiv preprint arXiv:1811.00755 2018.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequencceto-ssequence pre- training for natural language generation, translation, and comprehension. arXiv preprint 1rXiv:1910.13461, 2019.
Mike Lewis, Marjan Ghazvininejad, Gargi Ghosh, Armen Aghajanyan, Sida Wang, and Luke Zettle- moyer. Pre-training via paraphrasing, 2020.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1907. 1692, 2019.

---

Published as a conference paper at ICLR 2021
Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pp. 109-165. Elsevier, 1989.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. arXiv preprini arXiv: 1rXxi:/802.05957 2018.
Marius Mosbach, Maksym Andriushchenko, and Dietrich Klakow. On the stability of fine-tuning bert: Misconceptions, explanations, and strong baselines, arXiv preprint arXiv:2006.04884, 2020.
Courtney Napoles, Matthew R Gormley. and Benjamin Van Durme. Annotated gigaword. In Proceedings of the Joint Workshop on Automatic Knowledge Base Construction and Web-scale Knowledge Extraction (AKBC-WEKEX), pp. 95-100, 2012.
Razvan Pascanu and Yoshua Bengio. Revisiting natural gradient for deep networks, arXiv preprint arXiv:1301.3884 2013.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. Language models are unsupervised multitask learners. OpenAl Blog, 1(8)99 2019.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv: arXiv:/606.00 06066662660000 2016.
Garvesh Raskutti and Sayan Mukherjee. The information geometry of mirror descent. IEEE Trans- actions on Information Theory, 615)11151-14477 2015.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference machine learning, pp. 1889-1897, 201 5a.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pp. 1889-1897, 2015b.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference empirical methods in natural language pro- cessing, pp. 163 1-34-164, 2013.
Michael Tschannen, Josip Djolonga, Paul K Rubenstein, Sylvain Gelly, and Mario Lucic. On mutual information maximization for representation learning. arXiv preprint /9Xiv: 13625, 2019.
Alex Wang. Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceed- ings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pp. 353-355, Brussels, Belgium, November 2018. Association for Computational Lin- guistics. doi: 10. 11118653/11118-54440 URL
W18- 5446
Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. Neural network acceptability judgments. arXiv preprint arXiv: 1eX.::1805./247. 2018
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sen- tence understanding through inference. In Proceedings of the 2018 Conference the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Pupers), pp. 11p.11111 1112-1122. Association for Computational Linguistics, 2018. URL
Dongling Xiao, Han Zhang, Yukun Li, Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang. Ernie-gen: An enhanced multi-flow pre- pre-training and fine-tuning framework for natural language generation. arXiv preprint arXiv:2001.11314 2020.
Yu Yan, Weizhen Qi. Yeyun Gong, Dayiheng Liu, Nan Duan, Jiusheng Chen, Ruofei Zhang, and Ming Zhou. Prophetnet: Predicting future n-gram for eequence-to-sqqeene pre-training. arXiv 2020.
preprint
10

---

Published as a conference paper at ICLR 2021
Jingqing Zhang, Yao Zhao, Mohammad Salch, and Peter J Liu. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization, arXiv preprint arXiv:1912.08777, 2019.
Tianyi Zhang, Felix Wu, Arzoo Katiyar, Kilian Q Weinberger, and Yoav Artzi. Revisiting few- sample bert fine-tuning, arXiv preprint arXiv:2006. 05987, 2020.
Chen Zhu, Yu Cheng, Zhe Gan, Siqi Sun, Tom Goldstein, and Jingjing Liu. Freelb: Enhanced adversarial training for natural language understanding. In International Conference on Learning Representations, 2019.
A APPENDIX
A.I
CONTROLLING CHIANGE OF REPRESENTATION VIA CHIANGE OF VARIABLE
Let us say we have random variables in some type of markovian chain z, 2; y f(;; g 0g)
The change of variable formulation for probability densities
p(f(x;0)) - = det dg(f(z; 0y))
(6)
df(x;0,)
Direct application of change of variable gives us
KL(p(S(x; ( ~01))) =
p(f(x00,)
-p(s(z;0p) log p(s(;; O, + 20,))
Σ Ep(o(f(r(())) det dg(s(x;()))
df(r;0,)
logplg(f(r; 0,)) + log det dg(f(x;0,))
(10)
df(x;0/)
- -logp(s(/(r; (0,) log det dg(f(x; AB
(11)
(z; Adj)
(12)
Let us make some more assumptions. Let gly) Wy where the spectral norm of W,p(WW
1.
We can then trivially bound det W < 1. Then we have
det dg(s(2; 04))
E((o(((=(0)))
[logp((((=;;01)) 1ogpl((((;
(13)
df(x; 0y)
-x(o(((x: det dg(s(2; 0f))
((x(((z;0,)))
log
(14)
df(z;0g)
p(g(s(x;0,))
Ep(o(f(=0))))) p(g(f(r; (0,)))
(15)
Kupploff(( (0))))
(16)
We also see that tightness is controlled by det WI, which is bounded by the singular value giving us intuition to the importance of using spectral normalization.
A.2 EXPERIMENT HYPER- PARAMETERS
For our GLUE related experiments, both full fine-tuning and probing, the following parameters used. For probing experiments, the difference is our RoBERTa encoder is frozen, and the encoder dropout is removed.
11

---

Published as a conference paper at ICLR 2021
Hyper Parameter
MNLI
QNLI
QQP
SST-2
RTE
MRPC
CoLA
Learning Rate
5c-6
5e-6
5e-6
5e-6
le-5
le-5
le-5
Max Updates
123873
33112
113272
20935
3120
2296
5336
Max Sentences
8
32
32
16
Table 5: Task specific hyper parameters for GLUE experiments
Hyper parameter
Value
Optimizer
Adam
Hyper parameter
Value
Adam-betas
(0.9, 0.98)
Adam-eps
le-6
[0.1, 0.5, 1.0, 5.0] [U. le 5
LR Scheduler
polynomial decay
Noise Types
Dropout
0.1
Weight Decay
0.01
Warmup Updates
0.06 max updates
Table 6: Hyper parameters for R3F and R4F experiments on GLUE
Hyper Parameter
CNN/Dailymail
Gigaword
Reddit TIFU
Max Tokens
1024
2048
2048
Total updates
80000
200000
200000
Warmup Updates
1000
5000
5000
Table 7: Task specific hyper parameters for Summarization experiments.
Hyper parameter
Value
Hyper parameter
Value
[0.001, 0.01, 0.1] [(
Optimizer
Adam
Noise Types
Adam-betas
(0.9, 0.98)
le - 5
Adam-eps
le-8
Dropout
0.1
LR Scheduler
polynomial decay
Weight Decay
0.01
Learning Rate
3e-05
Clip Norm
0.1
Table 8: Hyper parameters for R3F and R4F experiments on Summarization experiments.
Hyper parameter
Value
Hyper parameter
Value
Optimizer
Adam
[0.5, 1, 3, [U
Adam-betas
(0.9, 0.98)
Noise Types
Adam-eps
le-8
le-5 5
LR Scheduler
polynomial decay
Total Updates
450000
Learning Rate
3e-05
Max Positions
512
Dropout
0.1
Max Tokens 4400
Weight Decay
0.01
Max Sentences
8
Table 9: Hyper parameters for R3F and R4F experiments on XNLI.
12


Reviews:
[{"review_id": "w4Eapvo_FtX", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a method for fine-tuning to address the issue of representation collapse. The authors claim that their proposed approach is more robust to hyper-parameters of the fine-tuning process and more computationally efficient compared to its counter parts, e.g., SMART. The proposed method is called Robust Representations through Regularized Finetuning (and an extension called Robust Representations through Regularized and Reparameterized Finetuning)  the main idea, as I understand, is to minimize the amount of change in representations of the model at each training step during the fine-tuning.\n\nIn order to show that the representational collapse problem exist when using standard fine-tuning techniques, and that their method, indeed, resolves this problem, they design a series of probing experiments where the apply fine-tuning on a set of datasets/tasks in a sequential order using the best checkpoint from the prior iteration.  They measure the performance of the iteratively fine-tuned model on the source task at each step and show that with their method, the performances drops much less with sequential probing in contrast to the standard fine-tuning approach. Furthermore, they apply this iterative fine-tuning in cycles, revising the sequence of tasks in each cycle, and they find that in most cases, they get much bigger improvements in the performance of the model on the target tasks in each cycle (again compared to standard fine-tuning).\n\nI find the problem addressed in the paper is super important and proposed solution very intuitive. I think the paper can be written in a way more clear way with a bit bigger audience in mind. In addition, in order to better show the merits of the proposed approach, if applicable maybe it would be more fair if the approach is compared with more simpler solutions to pose constraints on the amount of change in the representations, e.g., to simply  mix the data from source and target or add a distance term to the fine-tuning loss, or existing approaches like elastic weight consolidation that are used in continual learning setups (where the constraint is on the parameters of the model rather than the representations).\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "RfTjRDXIx7w", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n#### Summary\n\n- This paper presents a simple but effective method rooted in trust region theory for fine-tuning pre-trained models without 'representational collapse'. Compared to previous methods (such as SMART by Jiang et al. (2019)), the newly proposed methods (R3F and R4F) are computationally simple while achieving more strong performance on several NLP tasks including GLUE, XNLI and summarization. The authors also introduce the concept of 'representational collapse', which means the degradation of generalizable representations of pre-trained models during the fine-tuning stage. Moreover, they empirically demonstrated that SMART and their proposed methods are effective in relieving representational collapse, compared to typical fine-tuning based on normal gradient descent (i.e., one without constraints).\n\n#### Pros (Reasons to Accept)\n\n- The paper is clearly written.\n- The introduction of simple but effective & efficient methods for fine-tuning pre-trained models.\n- Extensive experiments. It's good to see experiments on XNLI and summarization tasks in addition to one on GLUE.\n- Strong emprical results, achieving SOTA on several tasks.\n\n#### Cons (Reasons to Reject)\n\n- The proposed methods (R3F and R4F) are a simple and incremental revision of the exitsing method (SMART), and there are no fundamental grounds or intuitions from which the proposed formulation is derived and justified (except for its empirical superiority).\n- There is no details or explanations about how the proposed methods are directly related to trust region theory (and what exactly trust region theory is).\n\n#### Comments\n\n- I'm just wondering whether the proposed methods can also bring improvement to InfoXLM (in addition to XLM-R) in the XNLI experiment. If possible, showing this would make your claim much stronger.\n- Why should **fine-tuned** representations be also **generalizable** in cases where we only consider the end performance of a specific target task (and the proposed methods do not bring significant improvement on the target task performance)? I understand that the proposed methods are desirable in the case of XNLI where zero-shot cross-lingual transfer explicitly requires fine-tuned representations to be still general enough to be properly transferred to other languages. Is there any other plausible story where generalizability is very important when fine-tuning for **only** a designated task?\n- It would be much better if the definition of 'representational collapse' can be well-defined in a mathematical and measurable manner, instead of just relying on empirically showing its existence with probing (though it is also desirable).\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "DIKqZkFzVpe", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n\nThe paper proposes a method for finetuning pre-trained models that ensures the generalization ability of the representation is maintained. The key innovation is that the computationally expensive ascent step in the mirror descent method of SMART can be replaced by simply injecting noise. The results support the hypothesis that this works well for keeping the generalization-ability of the model. The authors also define the degradation of the generalizability of the representation during finetuning as “representational collapse”. \n\nStrengths\n- The proposed approach is based on the change of the model in the output space g.f which seems like a very sensible way to constrain the model. The proposed approach therefore shares the advantage that the “change” being minimised has some meaningful interpretation. This is in contrast to many continual learning approaches which operate purely in weight space. \n\n- Constraining the output function g to be 1-Lipschitz is also sensible and well explained in the paper as it ensures the Bregmann-divergance-based smoothness constraint applied on the output will also constrain the representation, f.\n\n- The experiments are quite strong. The method has been evaluated on a large range of NLP tasks using various transformers as the base model. All experiments include multiple runs and the average/median statistics have been reported.\n\n- The approach is much faster than the closest existing method, SMART and achieves comparable accuracy in most cases.\n\n- Overall the paper is very well written and easy to understand. The proposed novelty compared to the closest existing approach is clearly highlighted and validated by the experiments.\n\nConcerns\n- The generalization experiments in Figure 4 only compares the proposed method to standard fine-tuning with best practices (i.e. Standard++), why has a more sophisticated methods like SMART not been included in this figure? Also, the authors state that “R3F/R4F consistently outperforms the adversarial fine-tuning method SMART”, but from Figure 3 it seems that the converse is also true - in at least 2/6 of the tasks, SMART outperforms all the variants of the proposed method and is on par in two others.\n\n- There is quite a range of performance between the variants R3F and R4F, but there aren’t any guidelines or suggestions on why this is the case or which one should be used in a particular situation. \n\n- The results in Table 3 and Table 2 show fractional improvements over the existing methods, however not variance is reported for these numbers. Another issue is that Table 2 uses median whereas Table 3 uses average. Is there a reason for this discrepancy?\n\n- The need for a new term “representational collapse” is not really justified in the paper. Most authors just use the term generalization. What exactly is the difference between “representational collapse” and just saying the models lacks the ability to generalize?\n\n- Perhaps the most significant weakness of the paper is that the novelty seems a bit limited. The difference compared to SMART is not really justified in a theoretical or principled manner. For example, what are the implications for using noise samples in Eqn. 4? Is it simply a heuristic to encourage smoothness? It would be good if the authors could explain this in more detail in the paper. At the moment it just appears as if ad-hoc modifications have been made to the cost function.\n\nMinor comments\n- There are some very minor typos throughout the paper that can be fixed. Eg. “even great degree”\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "0mgWqbL9-OE", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a lightweight fine-tuning strategy motivated by the trust-region theory, which achieved SOTA results on GLUE and  XNLI using no novel pertaining approaches. The paper also introduces a new analysis by defining a notion of representational collapse and provides a new methodology for measuring it during fine-tuning, which is interesting.\n\nIn my opinion, the defining and analysis of representational collapse is the major contribution of this paper. This paper  is well-written and strongly motivated with solid experimental results. \n\nStrength: \n\n+ A novel view of representation learning, a.k.a. representational collapse with comprehensive evaluations and detailed analysis. It may motivate other works of robust representation learning, and it is well suited for ICLR.\n+ A novel fine-tuning method which does not require extra backward computations and empirically works as well as or better than SMART\n+ SOTA evaluation results on both NLU and NLG datasets\n\nWeakness:\n\n- The overall design of R4F is simple, as leveraging the Spectral Normalization to make the function 1-Lipschitz is not new,\n- Some notions and symbols are missing, such as x~ in equation 2. And the analysis of the relationship to SMART and FreeLB is also a bit vague. I recommend the authors to carefully revise this part.\n \nQuestions:\n\nwhat is the major difference between catastrophic forgetting and representational collapse?\n\nWha will happen if fine-tuned with different kinds of noise z? or with different sigma?", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a pair of related regularization-oriented techniques for fine-tuning pretrained transformer models for NLP tasks, and shows that both are more efficient and more effective than prior work in thorough experiments on a wide range of tasks. The techniques are motivated by the idea of 'representational collapse', which is defined as drops in the ability of a linear model trained on an input representation to solve tasks _other than_ the one being trained on.

Pros:
- The new method is demonstrated to be broadly efficient and effective on a wide range of tasks.

Cons:
- It's not clear why 'representational collapse' warrants a new term, or whether it's desirable in general.
- The motivations for some of the precise technical decisions behind the new methods are unclear.
Decision: Accept (Poster)