Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: The paper proposes a method to avoid overfitting while finetuning the large pretrained models for downstream tasks on small scale datasets. It has been shown that many SOTA models usually overfit w.r.t. spurious correlations in the data and as a result fail miserably when tested for generalization on the out of domain datasets. The proposed method tries to maximally filter out task-irrelevant information in the feature vectors by minimizing the mutual information between the original features and the bottleneck features while simultaneously optimizing for performance. Experiments on several datasets show improved performance on both in-domain and out-of-domain datasets.

Strong Points:
Simple to implement the method and strong empirical results and analysis. In Section 3, Table 2 clearly shows that the method provides significant improvements under the low-data regimes and the model also achieves significant improvements in most of the datasets when tested for out of domain generalization. Analysis in section 4 shows that the method is indeed able to avoid overfitting to spurious correlations. 

Weak Points:
Since the paper deals with low-resource scenarios, I would have really appreciated if the experiment section also included some experiments on multilingual datasets while focusing on low-resource languages.  The general usefulness of the proposed method might have been more apparent if the paper could also cover a few additional tasks beyond text classification (E.g. NER, Translation e.t.c.)

Questions:
In Section 3.3, the authors methon the following:
> Following prior work (Belinkov et al., 2019a;Mahabadi et al., 2020), we select hyper-parameters based on the development set of each target dataset and report the results on the test set.

Where is the hyperparameter selection required while evaluating the model on OOD data? I guess I am missing something here...
review_title: The paper proposes a method to avoid overfitting while finetuning the large pretrained models for downstream tasks on small scale datasets. 

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This work applies information bottleneck as a way to compress the pre-trained representation so that only meaningful features are employed for the target task. It is applied for the number of GLUE tasks especially focusing on low resource settings and show consistent gains over previously known strong baselines, e.g., Mixout and L2-of-difference. This work also demonstrates that the learned model has generalization capacity so that the tuned model works on out-of-domain data.

# Pros

* An elegant solution to the fine tuning settings especially for the low-resource settings.

* Experiments are performed extensively on various tasks and demonstrates its effectiveness in generalization for out-of-domain settings.

* Interesting analysis of the experimental results.

# Cons

* Basic idea is already demonstrated by Li and Eisner (2019), and I was not very surprised by this results.

# Details

It is a very sophisticated way of avoiding overfitting especially when the data size is limited, and it might have an impact of broader application when exploiting pre-trained models. Thus, I'd recommend acceptance for this submission.
review_title: Nice work on using information bottleneck for fine-tuning pre-trained models

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper studies fine-tuning BERT-like pretrained language models (PLMs) on low resource target tasks. The authors hypothesize that the general-purpose knowledge obtained by the PLMs from pre-training might be irrelevant and redundant for a given target task. When fine-tuned onto a low resource target task, overfitting is likely to happen. To this end, a fine-tuning framework based on variational information bottleneck (VIB) is proposed to address these challenges. Specifically, the sentence representation will be mapped to a latent Gaussian variable  which compresses information in the sentence and also suppress irrelevant and redundant features, and a reconstructed version of the representation is used for task prediction. Empirical evaluations on sever datasets demonstrates the effectiveness of the method over previous research.

The paper is presented well, and it's a good read. However, my major concern is on the novelty of the proposed method. As cited by the paper, VIB has been proposed and explored in various different settings, including supervised learning, semi-supervised learning, etc., and in a similar sense, variational encoder decoders have also been thoroughly explored. The proposed method is a direct application of VIB and/or variational encoder decoder. Apart from the competitive experimental results shown on the GLUE benchmark and a set of other tasks over standard baselines including Dropout, mixout and weight decay, I find it hard to justify the novelty of the proposed method. In other words, the VIB framework is general and if additional task/fine-tuning specific insights were identified and shown to necessary when applying to the low-resource fine-tuning, novelty is also justified. However, with the current set up of plainly applying VIB to fine-tune a PLM, I find novelty rather limited. 

A minor question: as hypothesized if the pretrained LM contains many general purpose features, thus those irrelevant and redundant features needs to be suppressed, would one imagine the framework to work even better with large pretrained model pretrained on a much larger corpus (like BERT-large compared to BERT-base)? The main results in the paper seem to suggest otherwise, i.e., with a larger model, VIBERT actually has much less room to improve. How would the VIB framework work with a different PLM, e.g., XLM-Roberta, XLNet or T5?
review_title: Well written paper with good results, but limited novelty

final_score: rating: 4: Ok but not good enough - rejection
review_content: **Short summary of the paper**:
The authors apply the Deep Variational Information Bottleneck (DVIB) to a NLP setting, using pretrained BERT
as a fixed part of the encoder and fine-tune subsequent MLP layers of the encoder as well as an MLP decoder.
The proposed architecture shows state-of-the-art results compared to other recent regularization methods, especially in low-resource and out-of-domain benchmarks.

**Contributions**:
- Proposal of the use of DVIB with large-scale pretrained models such as BERT in a NLP setting (low significance)
- Extensive experiments showing higher generalization and robustness to bias compared to other SOTA regularization methods in NLI benchmarks and low-resource transfer learning (medium significance)

**Pros**:
- The shown results show SOTA results in terms of generalization for a wide range of benchmarks with only marginal increase of model complexity (in terms of # of parameters & training-time).


**Cons**:
Limited novelty & incremental contribution:
- Although SOTA results are shown in very extensive experiments, the methodical contribution is rather marginal,
as it boils down to adding a pre-trained BERT to the encoder part of a DVIB.
- Aside from the pre-trained BERT part, no contributions or changes to a vanilla DVIB architecture were made.
- The novelty mainly stems from applying the DVIB to a new specific setting ("fine-tuning large-scale language models on low-resource scenarios").

**Style**:
Overall, the paper is well written and structured.

**Experiments**:
In principal, the experimental setup seems well reasoned, comprehensible & extensive.
However, I'm rather concerned about the general concept of "fine-tuning across random seeds".
In my opinion the random seed should not be a tunable hyperparameter.

**Minor Comments**:
I think for the effect of the Lagrange parameter on the losses (Figure 3), an IB curve plotting the two mutual information terms against each other for different betas would be more suitable. 

review_title: Recommendation to reject due to lack of novelty and exclusively empirical contributions.  

Meta-review: The paper shows the success of a relatively simple idea -- fine tune a pretrained BERT Model using Variational Information Bottleneck method of Alemi to improve transfer learning in low resource scenarios.

I agree with the reviewers that novelty is low -- one would like to use any applicable method for controlling overfitting when doing transfer learning, and of the suite of good candidates, VIB is an obvious one -- but at the same time, I'm moved by the results because of: the improvements and the success on a wide range of tasks and the surprising success of VIB over other alternatives like dropout etc, and hence I'm breaking the tie in the reviews by supporting acceptance.  Its a nice trick that the community could use, if the results of the paper are an indication of its potential.
Decision: Accept (Poster)