Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper studies the calibration problem of BERT-like models on classification tasks. They show that the problem is created mainly during fine-tuning - the pre-trained model itself is well calibrated on the MLM task it is trained on. This MLM knowledge and calibration is destroyed during fine-tuning (a process called catastrophic forgetting in the literature). 

To solve the calibration problem, the authors propose to continue fine-tune the model on the MLM task jointly with the actual classification 
task at hand. 

Experimentations on 3 datasets and both in-domain and out-of-domain settings show that the proposed method yield lower calibration error compared to other methods (that do not use post-training calibration). In terms of raw quality, the proposed model perform on par with the existing method (and frequently slightly better). 


strength_and_weaknesses: ## Strengths
- The task of calibrating the results of a large language model is important and necessary in some practical applications. 
- The observation that the problem of mis-calibration is created during fine-tuning is interesting and novel. 
- The experimentation was done on multiple datasets and in two different settings. The authors run fine-tuning 5 time and report average and standard deviation, showing a more complete picture compared to the results of a single run. 

## Weaknesses
- It is unclear why post-processing techniques, such as temperature scaling, are insufficient, that we need a new training paradigm for overcoming this post processing step, especially that temperature scaling is quite lightweight. This makes the overall contribution quite limited. 
- Using MLM loss in addition to the standard classification loss is known (for a long time) to produce better results. Most work run an additional pre-training step on the target dataset, e.g., https://arxiv.org/pdf/1905.05583.pdf, https://arxiv.org/pdf/2004.11493.pdf, but joint training is also a known technique (which performs slightly worst, so less cited. Still, it was already published on e.g., https://dl.acm.org/doi/pdf/10.1145/3437963.3441777). This makes the comparison to the baseline unfair, since additional (unsupervised) training data was used. 
- It is unclear what is the performance implications are on the training time. If the training scheme alternates between MLM and classification losses, this would result in 2x longer training time, which is a significant impact. This should be at least clarified in the paper. 
- It is unclear how Section 5.1 is related to the rest of the paper (except for naming the algorithm), given that after all complex formulas the paper uses the standard MLM loss, and then move to knowledge distillation loss. Clarifying the connection to the rest of the paper (or dropping this part) would make it easier to follow the paper's contribution. 
- Table 1, showing that after fine-tune the model is unable to do the original task, is not by itself surprising or interesting. The question is how this is impacting calibration on the classification task, which is not directly related to the quality of the model on the MLM task. 
clarity,_quality,_novelty_and_reproducibility: ### Quality
- The experiments are clear and well done. 

### Clarity 
- The paper itself has one (seemly important) section whose connection to the rest of the paper is unclear. 

### Originality
- The idea of training classification together with MLM is not novel. 
- Studying the impact of this on the calibration error is novel, but the scope/impact of this is quite narrow. 

summary_of_the_review: - The paper is well executed, but the task of reducing calibration error without post processing is quite narrow and has low impact. The approach of jointly training classification and MLM loss is not novel, but the impact on reducing calibration error is somewhat novel.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper studies the ECE of language models trained using different recipes, including the pretrained model, finetuned model and parameter-efficient finetuned models. They find that the finetuned model has larger ECE and tend to be over confident when making predictions. The paper proposes a baseline method of mixing the finetuning objective and the pretraining objective during finetuning to address this and this method works to lower ECE. 
strength_and_weaknesses: Strength: the paper studies ECE in language models which is somewhat of an overlooked area.

Weaknesses: 
1, The paper did not provide sufficient motivation for why the broader community should care about a metric like ECE. They have a few references and some vague description of "disastrous consequences in safety critical applications". But they did not concretely describe such applications and such consequences or demonstrate that this is the most important issue of language models when applied in such applications. Thus this reviewer is not convinced that they are solving a problem of high enough priority.

2, The paper did not go deep into the reason for why finetuning causes over-confidence in language models. They observed this phenomenon and proposed the most basic solution, which is mixing finetuning with pretraining. The result is also non-surprising. When the finetuning is mixed with the pretraining, then the resulting model behaves a bit more similar to the pretrained model. One might imagine that the overconfidence and the regression of the finetuned model as demonstrated in Table 1 is a result of the lack of diversity in the finetuning dataset. It would be interesting to establish the connection between the diversity of the training dataset and the ECE/overconfidence issue empirically, which would be a meaningful contribution to the community in understanding the response of deep networks to different data distributions.  

3, The discussion about parameter efficient finetunings are interesting. However if one wants to make these model still do MLM, one can just remove the trained small parameter components and keep the pretrained model. The motivation for testing a parameter efficient model that is finetuned on specific downstream task on MLM is not very clear.
clarity,_quality,_novelty_and_reproducibility: 1, The paper in general is clearly written. 
2, The originality is somewhat lacking - the results are not surprising and aligns with general expectations. The paper succeeded in putting them in a quantitative light. 
summary_of_the_review: The paper has made some good observations on how LMs training differently would have different ECE and overconfidence behavior. They proporsed a baseline mixture of pretrain and finetune to reduce ECE for finetuning, which was successful. It was not clear why this is an important discovery or innovation that significantly benefits the ICLR community.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work observes that while a pretrained language model is well-calibrated on the pretraining task as measured by ECE, it becomes poorly calibrated on downstream tasks after finetuning. It is further observed that efficient finetuning methods, such as Adapter, LoRA, and Prefix Tuning, result in better calibrated models than full finetuning. This leads the authors to conclude that the source of the miscalibration is the distortion of features learned during pretraining.

The authors then proposed to include the pretraining objective during finetuning to preserve pretraining features. This improves model calibration as measured by ECE while largely retaining performance on SNLI/MNLI, QQP/TwitterPPDB, and SWAG/HellaSWAG.
strength_and_weaknesses: Model calibration is often overlooked in the context of large language models. This paper makes the intriguing observation that efficient finetuning methods yield better calibrated models than full finetuning, likely due to the former’s ability to preserve pretraining features.

This work leverages this observation to further improve calibration by retaining the pretraining objective during finetuning. However, the denoising variational auto-encoding interpretation seems unnecessary. The referenced paper, Im et al., 2017, is motivated by the ability to obtain a more expressive variational distribution by marginalizing over input corruptions (e.g., to obtain a mixture of Gaussians). This marginalization makes the usual ELBO intractable, which motivated their new DVAE training criterion.

This work, however, uses a deterministic mapping from x to z, i.e., the variational distribution is the Dirac Delta distribution. By forgoing the stochasticity in the latent space, this work neither shares the core motivation of nor benefits from the DVAE paper. It is perhaps okay to mention Denoising Autoencoder in this context, but it basically describes the MLM objective which is not a contribution of this paper.

The method proposed in this work is better understood by inspecting Eq. 6 and can be summarized as “performing MLM on the finetuning data in addition to the finetuning objective (plus an L2 regularization in the latent space).” It is also unclear what the role of the L2 regularization is since it is present in neither the usual pretraining nor the usual finetuning objective. Indeed, as shown in Table 8, successful values of beta are many orders of magnitude smaller than those of alpha.

I would encourage the authors to reframe their method without the DVAE framework and draw from the established literature on catastrophic forgetting. Moreover, given the baseline success of efficient finetuning methods, it seems worthwhile to investigate how they can be made better calibrated for scenarios where calibration is much more important than a slight performance drop.

clarity,_quality,_novelty_and_reproducibility: As mentioned above, the presentation of the key technical contribution could be vastly simplified by forgoing the DVAE formalism. Otherwise, the paper is easy to understand and generally well-written. The novelty is somewhat limited given that the core idea is simply including the MLM objective during finetuning. This can be strengthened by considering other ways to preserve pretraining features, such as “weight decay” back to the pretrained weights and prior work on combatting catastrophic forgetting. If including the MLM objective is indeed to best way to preserve pretraining features for better calibration, the conclusion of this paper would be much informative.
summary_of_the_review: Great motivation and interesting observations. However, the presentation can be simplified by forgoing the DVAE formalism, which adds little to this work. This paper can be further strengthened by considering other baselines that preserve pretraining features.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper addresses a problem where fine-tuned pre-trained language models are not well calibrated. They observe that the problem lies in the fine-tuning phase and suggest to rectify this by modifying the fine-tuning process in a way that forces it to preserve (in some way) the pre-trained weights. The paper does not suggest a new method for fine tuning, rather explores several known methods in terms of their performance and calibration, both in distribution and out of distribution.

In its original version, the paper suggested a solution involving Denoising Variational Auto-Encoders. The reviews mentioned this to be an unnecessary addition that only confuses the reader. As a result, the authors rewrote the paper (and changed its title) making it clear and easy to read, as well as an honest presentation of the real contribution: Not a new method but a study of existing methods. 

In the discussion phase I found that the scores for the paper remain only mildly enthusiastic due to two reasons: (1) the impact of the paper may be considered limited as it doesn't provide a new method, but a study of existing methods in a new lens (of calibration), (2) the changes are substantial, and some of the reviewers did not have the time to properly review the revised version.

As for (1), given the overall support from all 3 reviews as well as my opinion, I believe that studying the issue of calibration is well motivated and a thorough study should be of interest to the ICLR community. For issue (2), I went over the revised version and find it easy to read, and agree with the reviewers that the main issues of clarity w.r.t other methods and motivation are solved.




justification_for_why_not_higher_score: 1) The impact is limited.
2) The revised version is quite different from the original lowering the reviewers confidence
justification_for_why_not_lower_score: The paper provides a thorough analysis to an interesting problem, its easy to read and would be interesting to the ML community, as well as motivate follow up works providing better solutions to the highlighted problem
Decision: Accept: poster