Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary

The paper proposes a method for finetuning pre-trained models that ensures the generalization ability of the representation is maintained. The key innovation is that the computationally expensive ascent step in the mirror descent method of SMART can be replaced by simply injecting noise. The results support the hypothesis that this works well for keeping the generalization-ability of the model. The authors also define the degradation of the generalizability of the representation during finetuning as “representational collapse”. 

Strengths
- The proposed approach is based on the change of the model in the output space g.f which seems like a very sensible way to constrain the model. The proposed approach therefore shares the advantage that the “change” being minimised has some meaningful interpretation. This is in contrast to many continual learning approaches which operate purely in weight space. 

- Constraining the output function g to be 1-Lipschitz is also sensible and well explained in the paper as it ensures the Bregmann-divergance-based smoothness constraint applied on the output will also constrain the representation, f.

- The experiments are quite strong. The method has been evaluated on a large range of NLP tasks using various transformers as the base model. All experiments include multiple runs and the average/median statistics have been reported.

- The approach is much faster than the closest existing method, SMART and achieves comparable accuracy in most cases.

- Overall the paper is very well written and easy to understand. The proposed novelty compared to the closest existing approach is clearly highlighted and validated by the experiments.

Concerns
- The generalization experiments in Figure 4 only compares the proposed method to standard fine-tuning with best practices (i.e. Standard++), why has a more sophisticated methods like SMART not been included in this figure? Also, the authors state that “R3F/R4F consistently outperforms the adversarial fine-tuning method SMART”, but from Figure 3 it seems that the converse is also true - in at least 2/6 of the tasks, SMART outperforms all the variants of the proposed method and is on par in two others.

- There is quite a range of performance between the variants R3F and R4F, but there aren’t any guidelines or suggestions on why this is the case or which one should be used in a particular situation. 

- The results in Table 3 and Table 2 show fractional improvements over the existing methods, however not variance is reported for these numbers. Another issue is that Table 2 uses median whereas Table 3 uses average. Is there a reason for this discrepancy?

- The need for a new term “representational collapse” is not really justified in the paper. Most authors just use the term generalization. What exactly is the difference between “representational collapse” and just saying the models lacks the ability to generalize?

- Perhaps the most significant weakness of the paper is that the novelty seems a bit limited. The difference compared to SMART is not really justified in a theoretical or principled manner. For example, what are the implications for using noise samples in Eqn. 4? Is it simply a heuristic to encourage smoothness? It would be good if the authors could explain this in more detail in the paper. At the moment it just appears as if ad-hoc modifications have been made to the cost function.

Minor comments
- There are some very minor typos throughout the paper that can be fixed. Eg. “even great degree”

review_title: A simple and effective method for preserving generalizability of a model when fine-tuning.

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper presents a lightweight fine-tuning strategy motivated by the trust-region theory, which achieved SOTA results on GLUE and  XNLI using no novel pertaining approaches. The paper also introduces a new analysis by defining a notion of representational collapse and provides a new methodology for measuring it during fine-tuning, which is interesting.

In my opinion, the defining and analysis of representational collapse is the major contribution of this paper. This paper  is well-written and strongly motivated with solid experimental results. 

Strength: 

+ A novel view of representation learning, a.k.a. representational collapse with comprehensive evaluations and detailed analysis. It may motivate other works of robust representation learning, and it is well suited for ICLR.
+ A novel fine-tuning method which does not require extra backward computations and empirically works as well as or better than SMART
+ SOTA evaluation results on both NLU and NLG datasets

Weakness:

- The overall design of R4F is simple, as leveraging the Spectral Normalization to make the function 1-Lipschitz is not new,
- Some notions and symbols are missing, such as x~ in equation 2. And the analysis of the relationship to SMART and FreeLB is also a bit vague. I recommend the authors to carefully revise this part.
 
Questions:

what is the major difference between catastrophic forgetting and representational collapse?

Wha will happen if fine-tuned with different kinds of noise z? or with different sigma?
review_title: A well-written paper with interesting ideas. 

final_score: rating: 7: Good paper, accept
review_content: This paper proposes a method for fine-tuning to address the issue of representation collapse. The authors claim that their proposed approach is more robust to hyper-parameters of the fine-tuning process and more computationally efficient compared to its counter parts, e.g., SMART. The proposed method is called Robust Representations through Regularized Finetuning (and an extension called Robust Representations through Regularized and Reparameterized Finetuning)  the main idea, as I understand, is to minimize the amount of change in representations of the model at each training step during the fine-tuning.

In order to show that the representational collapse problem exist when using standard fine-tuning techniques, and that their method, indeed, resolves this problem, they design a series of probing experiments where the apply fine-tuning on a set of datasets/tasks in a sequential order using the best checkpoint from the prior iteration.  They measure the performance of the iteratively fine-tuned model on the source task at each step and show that with their method, the performances drops much less with sequential probing in contrast to the standard fine-tuning approach. Furthermore, they apply this iterative fine-tuning in cycles, revising the sequence of tasks in each cycle, and they find that in most cases, they get much bigger improvements in the performance of the model on the target tasks in each cycle (again compared to standard fine-tuning).

I find the problem addressed in the paper is super important and proposed solution very intuitive. I think the paper can be written in a way more clear way with a bit bigger audience in mind. In addition, in order to better show the merits of the proposed approach, if applicable maybe it would be more fair if the approach is compared with more simpler solutions to pose constraints on the amount of change in the representations, e.g., to simply  mix the data from source and target or add a distance term to the fine-tuning loss, or existing approaches like elastic weight consolidation that are used in continual learning setups (where the constraint is on the parameters of the model rather than the representations).

review_title: Addressing an important problem but hard to read and understand

final_score: rating: 6: Marginally above acceptance threshold
review_content: 
#### Summary

- This paper presents a simple but effective method rooted in trust region theory for fine-tuning pre-trained models without 'representational collapse'. Compared to previous methods (such as SMART by Jiang et al. (2019)), the newly proposed methods (R3F and R4F) are computationally simple while achieving more strong performance on several NLP tasks including GLUE, XNLI and summarization. The authors also introduce the concept of 'representational collapse', which means the degradation of generalizable representations of pre-trained models during the fine-tuning stage. Moreover, they empirically demonstrated that SMART and their proposed methods are effective in relieving representational collapse, compared to typical fine-tuning based on normal gradient descent (i.e., one without constraints).

#### Pros (Reasons to Accept)

- The paper is clearly written.
- The introduction of simple but effective & efficient methods for fine-tuning pre-trained models.
- Extensive experiments. It's good to see experiments on XNLI and summarization tasks in addition to one on GLUE.
- Strong emprical results, achieving SOTA on several tasks.

#### Cons (Reasons to Reject)

- The proposed methods (R3F and R4F) are a simple and incremental revision of the exitsing method (SMART), and there are no fundamental grounds or intuitions from which the proposed formulation is derived and justified (except for its empirical superiority).
- There is no details or explanations about how the proposed methods are directly related to trust region theory (and what exactly trust region theory is).

#### Comments

- I'm just wondering whether the proposed methods can also bring improvement to InfoXLM (in addition to XLM-R) in the XNLI experiment. If possible, showing this would make your claim much stronger.
- Why should **fine-tuned** representations be also **generalizable** in cases where we only consider the end performance of a specific target task (and the proposed methods do not bring significant improvement on the target task performance)? I understand that the proposed methods are desirable in the case of XNLI where zero-shot cross-lingual transfer explicitly requires fine-tuned representations to be still general enough to be properly transferred to other languages. Is there any other plausible story where generalizability is very important when fine-tuning for **only** a designated task?
- It would be much better if the definition of 'representational collapse' can be well-defined in a mathematical and measurable manner, instead of just relying on empirically showing its existence with probing (though it is also desirable).


review_title: Review

Meta-review: This paper introduces a pair of related regularization-oriented techniques for fine-tuning pretrained transformer models for NLP tasks, and shows that both are more efficient and more effective than prior work in thorough experiments on a wide range of tasks. The techniques are motivated by the idea of 'representational collapse', which is defined as drops in the ability of a linear model trained on an input representation to solve tasks _other than_ the one being trained on.

Pros:
- The new method is demonstrated to be broadly efficient and effective on a wide range of tasks.

Cons:
- It's not clear why 'representational collapse' warrants a new term, or whether it's desirable in general.
- The motivations for some of the precise technical decisions behind the new methods are unclear.
Decision: Accept (Poster)