Preserving Pre-trained Features Helps Calibrate Fine-tuned Language Models
Authors: Guande He
Jianfei Chen
Jun Zhu
Venue: ICLR 2023 poster (2023)

Abstract: Large pre-trained language models (PLMs) have demonstrated strong performance on natural language understanding (NLU) tasks through fine-tuning. However, fine-tuned models still suffer from overconfident predictions, especially in out-of-domain settings. In this paper, we tackle the problem of calibrating fine-tuned language models. We demonstrate that the PLMs are well-calibrated on the masked language modeling task with robust predictive confidence under domain shift, yet the fine-tuned models fail to retain such property due to catastrophic forgetting, which impacts the calibration on the downstream classification task. In light of these observations, we evaluate the calibration of several methods that preserve pre-trained features and show that preserving pre-trained features can improve the calibration of fine-tuned language models. Among these methods, our proposed method that encourages the fine-tuned model to learn generative representations with auxiliary language modeling objective achieves competitive accuracy and the lowest expected calibration error compared to several strong baselines under both in-domain and out-of-domain settings on three downstream NLU tasks.

Published as a conference paper at ICLR 2023
PRESERVING PRE-TRAINED FEATURES HELPS CALIBRATE FINE-TUNED LANGUAGE MODELS
Guande Hel, Jianfei Chen'r, Jun Zhu
Dept. of Comp. Sci. & Tech., Institute for AI, Tsinghua-Bosch Joint Center for MI., BNRist Center, State Key Lab for Intell. Tech. & Sys., Tsinghua University, Beijing, China Pazhou Lab, Guangzhou, 510330, China heguande1 Sottteeee@@ueeeeee look. com,
(Jianfeic, dcszjj@ttsinghua. edu.cn
ABSTRACT
Large pre-trained language models (PLMs) have demonstrated strong perfor- mance natural language understanding (NLU) tasks through fine-tuning, How- ever, fine-tuned models still suffer from overconfident predictions, especially in out-of-domain settings. In this paper, tackle the problem of calibrating fine- tuned language models. We demonstrate that the PLMs are well-calibrated on the masked language modeling task with robust predictive confidence under domain shift, yet the fine-tuned models fail to retain such property due to catastrophic forgetting, which impacts the calibration on the downstream classification task. In light of these observations, we evaluate the calibration of several methods that preserve pre-trained features and show that preserving pre-trained features can improve the calibration of fine-tuned language models. Among these methods, our proposed method that encourages the fine-tuned model to learn generative representations with auxiliary language modeling objective achieves competitive accuracy and the lowest expected calibration error compared to several strong baselines under both in-domain and out-of-domain settings three downstream NLU tasks.
INTRODUCTION
Fine-tuning pre-trained language models (PLMs) is dominating paradigm for natural language derstanding (NLU) with state-of-the-art results for a variety of NLU tasks (Peters et al. [008 Devlin al. 2019 Liu et al./ (2019) He al., (021a). The powerful fine-tuned language models have been experimented with for decision-making in real-world applications such as the healthcare domain (He al. 2020) and safety-critical domain (Sandagiri et al., 2020), where the classification networks need to be highly accurate and provide calibrated confidence for their predictions to improve the safety and trustiness the models (Guo et al. 2017). For example, suppose a medical language inference LM that predicts the disease given the description of symptoms is well-calibrated, i.e., the model's posterior probabilities (or confidence) align well with the true correctness likelihood. In that case, the wrong predictions can be easier to detect and correct by human doctors by given low predictive confidence. However, as with other modern neural networks, the fine-tuned LMs shown to suffer from overconfidence (Desai & Durrett 2020 Jiang al. 2021). which creates obstacles and concerns for their deployment in real-world applications.
Uncertainty estimation of fine-tuned models is challenging due to the small amount of available data for fine-tuning, especially under out-of-domain settings (Desai & Durrett 2020; Guo et al.) 2021)
While prior works illustrate that simple calibration techniques such temperature scaling (Guo
ct al. 2017) and label smoothing (Szegedy et al.) (0166) are not sufficient to calibrate the fine-tuned LMs under both in-domain (ID) and out-of-domain (OD) settings (Desai Durrett 2020 Park & Caragea 2022), several approaches with strong regularization have been developed to calibrate the
fine-tuned model on NLU tasks, including knowledge distillation from deep ensembles (Guo al.
2021), stochastic network architectures (Fan et al.) 2020) Zhang et al., (222), and Mixup (Park) & Caragea 2022). However, these existing works mostly utilize general calibration methods for
"Corresponding authors.

---

Published as a conference paper at ICLR 2023
supervised learning, while specific properties of the pre-training & fine-tuning paradigm still largely neglected.
In this work, we tackle the calibration of fine-tuned models from the perspective of better leveraging the powerful PLMs. Through a carefully designed empirical study on both pre-trained and fine-tuned models, we first observe that PLMs themselves actually well-calibrated on the masked language modeling (MLM) task and robust to higher levels of perturbation to the inputs, which suggests the PLMs can model the predictive uncertainty well across different domains. However, the pre-trained features are only used as initialization and are distorted by the fully discriminative fine-tuning. The phenomenon is known as catastrophic forgetting (McCloskey & Cohen 1989. Kirkpatrick et al. 2017 Howard & Ruder 2018). We show that such forgetting can make the fine-tuned language models fail to hold proper predictive confidence toward the OD and outlier samples, which leads to miscalibration on the downstream tasks. Based on the observations, we hypothesize that preserving the pre-trained features helps calibrate the fine-tuned LMs.
To validate our hypothesis, we first evaluate the calibration of some previous methods that can pre- serve pre-trained features, including (1) Parameter-efficient tuning (Houlsby et al. 2019 Hu et al.)
2021 Li & Liang [20TT) (2) Pre-trained weight decay, (3) Mixout (Lee et al. 20020). Although these methods were originally designed to improve the performance beyond uncertainty estimation, our experiment demonstrates that these methods outperform vanilla fine-tuning in terms of calibration, especially under out-of-domain settings. Based on our observation that the PLMs are well-calibrated on the MLM task yet the fine-tuned LMs that forget the pre-trained features struggle with overconfi- dence under domain shift, we propose simple baseline that utilizes the MLM objective to maintain the consistency between the pre-trained and fine-tuned models. The proposed method achieves the lowest expected calibration error and competitive accuracy compared to existing calibration methods in both ID and OD settings on three NLU tasks, including natural language inference, paraphrase de- tection, and commonsense reasoning, showing that preserving the pre-trained features is an effective approach for improving the calibration of fine-tuned L.Ms.
PRELIMINARIES
2.1
MASKED LANGUAGE MODELS
Masked language models generally consist of a transformer-base text encoder fo parameterized by and linear language modeling head Je parameterized by 6. the pre-training phase, the model handles the masked language modeling task (Devlin et al., 2019). Assume we have unsupervised sequence inputs x sampled from large-scale corpora pr. (z). A subset of æ is first masked by a corruption function distribution. Denote the indices of the masked tokens M, the set of masked tokens as TM, and the observed unmasked input as x M. The model is trained to recover the masked tokens FM- In particular, the masked language model first uses the text encoder get hidden representation of the input, denoted as f (a)))) Then the language modeling head go with softmax function is applied to fe (a) to obtain a conditional categorical distribution Pmim (Ir (xxx)) over the vocabulary V for each masked position i M. The masked language modeling objective is:
Cmlm = -E (xe)
Σ VEM
log Pmmm (we Pr Φ, 0)]
In the fine-tuning phase, assume we have labeled data in the form of (æ, y) sampled from data dis- tribution Pa, where corresponds to the text input, and y corresponds to the label. For classification tasks, a task-specific head h. that parameterized by y is applied on the hidden representation of input to obtain the logit for each class. The predictive posterior distribution ((y]z) is given by the logits after softmax operation. In standard fine-tuning, the pre-trained encoder fo and the task- ssecifp head hep are jointly optimized using the cross-entropy loss:
Less =
(ps())) [log <0v/2;; 0, sp]
(2)
which is also known as full fine-tuning (Full-FT) (Peters et al. 2018 Devlin al.
2019
2

---

Published as a conference paper at ICLR 2023
2.2 CONFIDENCE CALIBRATION
The framework of confidence calibration under the supervised classification setting can be expressed as a joint distribution P(g. P) over the label prediction á E | and the corresponding confidence I E [0, A perfectly calibrated model holds P vlp p)
(Guo et al. 2017). 7). One way to evaluate calibration through finite samples expected calibration error, i.e., ECE (Naeini al., 2015). To compute ECE, the model's predictive confidences are first grouped into M equal-sized bins. Denote B, as the indices of samples whose confidences are in the interval (M m) Sup- pose we have N samples, the ECE is calculated by the weighted average of the difference between confidence and accuracy in each bin:
acc
Σ
(9: ye), conf((mm) 1mm]
Σ p.
[Bm]
ie ie
(3)
A
Bm
ECE
Σ
acc Bm) conf) Bm)
N
In this work, we set M = -00 following Desai & Durrett (2020).
A CLOSER LOOK TO THE PRE-TRAINED AND FINE-TUNED LANGUAGE MODELS IN CALIBRATION
In this section, we explore the connection between the pre-trained and fine-tuned language mod- els in terms of calibration by examining: (1) The calibration of the pre-trained language models themselves. (2) How fine-tuning affects calibration the downstream classification tasks.
3.1 WHAT WE FORGET AFTER FINE-TUNING THE PRE-TRAINED LANGUAGE MODELS
Pre-trained language models have demonstrated their ability to capture informative linguistic fea- tures from large corpora (Tenney al.) 2019 Jawahar al. 2019 Ethayarajh 2019). Intuitively, the pre-trained features learned on diverse corpora should be capable of performing uncertainty estimation well. In this subsection, we validate that the pre-trained language models are indeed well-calibrated on the MLL task, which suggests that the predominant full fine-tuning method that forgets such pre-trained features is suboptimal.
Table 1: The Expected Calibration Error (ECE) of the pre-trained RoBERTaBASE on the MLM task.
Penve 15, E0Ex3
0.8
Powce BCb-3.15
Accuracy
Auaee bobe421
2.6
Dataset
0.15
0.3
0.5
WikiText-I00 103
3.89
3.73p.12
4.30g
0.0
SNLI
2.99).2
3. 3.65p.1s
5.80
0.0
0.2
94
0.6
0.6
3.0
MNLI
2.68
3. 3.61g.10
5.65g
Confidence
4.236.13
4.74p.06
6.19
TwitterPPDB
7.52
8 8.15..07
10.12a.1
Figure 1: Reliability diagram of the pre-trained
SWAG
5.6To.08
5.860.02
6.88%.05
ROBERTaBASE on the MLM task on WikiText- 103.
HellaSWAG
3.188.10
3.440.0s
4.820.02
Setup: We evaluate the calibration of the pre-trained ROBERTABASE (Lu al., 2019) through the MLM task. We use the test split of the WikiText- 103 (Merity et al. 2016) one of the corpora in the pre-training phase, and six downstream datasets from different domains (see $5.1 and A. for details) sequence inputs for masked language modeling. The inputs masked and corrupted with three levels 15%, 30%, and 50% mask probability with the same masking approach (i.c., the 80-10-10 strategy) in the pre-training phase (Devlin al. 2019).
Results and Analysis: Table and Figure show the ECE and the reliability diagram (DeGroot & Fienberg, 1983 Niculescu-Mine & Caruana 2005) of the pre- treitrained RoBERT the MLM task. The results suggest that the PLM is relatively well cell-calibrated across different domains, where the model needs to recover the corrupted position with the options of [V]. Moreover, the mask
3

---

Published as a conference paper at ICLR 2023
probability grows higher than in the pre-training phase, the ECE of the PLM increases only by relatively small amount, which indicates that the PLM's predictive confidence Pmim ( on the MLM task is robust to higher corruption levels. Figure 2 demonstrates that although the hidden representations of 50% masked inputs (visualized by t-SNE (Van der Maaten & Hinton 2008)) have shifted significantly from the original input, the PLM still make calibrated predictions to the original inputs. Intuitively, the calibrated confidence on the MLM task suggests that the pre-trained features of PLMs are good at modeling the samples under large domain shifts, which may benefit the calibration of the downstream classification task under OD settings. However, this property is less likely to be retained by the fine-tuned LM due to catastrophic forgetting caused by full fine-tuning with the discriminative objective (Howard & Ruder 2018).
100
Avg. Conf.
Out Woleo.
0001405)
7.5
5.0
Tos
100
200
500
1k
2k
5k
10k
30k
Training Steps
Figure 2: t-SNE visualization for hidden repre- sentations of the sampled inputs from different domains given by the pre-trained (PT) and fine- tuned (FT) language models.
Figure 3: Average predictive confidence (up) and ECE (down) for the validation split of QQP (ID), TwitterPPDB (OD), and WikiText- 103 (outlier) dataset in different training steps.
3.2 How FINE-TUNING AFFECTS CALIBRATION ON THE DOWNSTREAM TASKS
Although previous works have shown that fine-tuned language models can outperform non-pre- trained models in terms of calibration (Desai & Durrett 2020), the fine-tuned LMs' calibration performance is still far from satisfactory, especially under the OD settings (Desai & Durrett [2020] Guo et al.] 202T). To study why fine-tuned LMs are miscalibrated under the OD settings, we conduct case study on the LM fine-tuned on the QQP dataset, which is a typical failure case that the fine- tuned model exhibits dramatic disparity ECE on ID and OD settings, as shown in Figure 3
Setup: We fine-tune the pre-trained RoBERTa BASE on the QQP training set following the default configuration of the Huggingface Transformers library (Wolf et al. 2020). Compared to the pre- trained model, we visualize the hidden representations of the same inputs with $3. given by the fine-tuned model. Besides, we evaluate the average confidence the fine-tuned model's ''s prediction on several datasets, including the in-domain QQP validation set, the oft-of-domain TwitterPPDB validation set, and the outlier WikiText-103 validation set that does not hold any particular attributes of the downstream classification task.
As shown in Figure compared to the pre-trained models, fine-tuning changes the hidden representation of the LM in two ways: (1) For the inputs within the same domain, fine-tuning enlarges the difference of the corresponding hidden representations, which aligns with the quantitative results of the previous work (Zhou & Srikumar 2022). (2) For the inputs across different domains, fine-tuning makes the hidden representations from different domains much harder distinguish by projecting them simpler data manifold, which causes the fine-tuned model fails to give proper predictive confidence for OD and outlier samples.
Results and Analysis:
As shown Figure g] the average predictive confidence of the ID, OD, and outlier validation sets increases the training step increases. The gap between the average predictive confidence and the correctness likelihood (i.c., classification accuracy) in the OD setting relatively larger than in the ID setting, which results in a larger OD ECE. More crucially, the average confidence of the outlier samples is higher during the whole training process than both ID and OD settings and increases to nearly 100% after three training epochs, which not be fixed by simple techniques such tem- perature scaling and early stopping. Ideally, the model should be uncertain about the outlier samples significantly deviating from the training samples. However, the fine-tuned LM exhibits overcon-
4

---

Published as a conference paper at ICLR 2023
fidence toward the OD and outlier samples, which implies that strong regularization methods needed to improve the confidence modeling for OD and outlier samples. Based on the observations that the pre-trained features of the PLMs model the predictive uncertainty well across differ- ent domains and are distorted by the fine-tuned LMs in $3.1] we hypothesize that preserving the pre-trained features of PLMs helps the fine-tuned LMs better model the predictive confidence and improve calibration on downstream classification tasks.
4 METHODS
To validate the hypothesis that preserving the pre-trained features helps the calibration of fine-tuned LMs, examine existing methods that preserve the pre-trained features in different ways. Although these methods are not originally designed for enhancing uncertainty estimation, such achieving better trade-off between tunable parameters and model's performance for parameter- efficient tuning, or improving classification accuracy and stability for pre-trained weight decay and Mixout, we anticipate that these methods may improve calibration by mitigating catastrophic for- getting and evaluate their effectiveness in calibration in $
4.1 PARAMATER-EFFCIIS TUNING
Parameter-effichh tuning is series of fine-tuning methods which keep the pre-trained parame- ters of the text encoder @ frozen and update only a small number of extra parameters da and the task-specific head hep while preserving competitive performance with Full-FT. Since the pre-trained knowledge is encoded to the model's parameters and there are only small amount of extra parame- ters. parameter-efficien tuning methods can preserve more pre-trained features compared to Full-FT. In this work, we choose three mainstream parameter efficient tuning methods: (1) Adapter (Houlsby) al., 2 which adds a weight bottleneck module after the output of each sub-layer in the transformer block; (2) LoRA (Hu et al., (2021), which updates the attention weight matrix using low-rank reparameterization (3) Prefix Tuning (Li & Liang 202T), which prepends tunable prefix vectors to keys and values of the multi-head attention layers.
4.2 REGULARIZATION WITH PRE-TRAINED WEIGHT
Introducing regularization terms using the pre-trained weight can also better leverage the pre-trained features during fine-tuning. In this work, we adopt two common regularization techniques:
Pre-trained Weight Decay: Traditional weight decay methods add a regularization term | that penalizes large weights to improve generalization (Krogh & Hertz 1991). where À is a regular- ization coefficient. As an alternative, performing weight decay towards the pre-trained weight we by adding 2 w w (2 to the task loss function is shown by previous works as an effective way to mitigate catastrophic forgetting caused by fine-tuning (Wiese et al. 2017) and can improve the performance of the downstream task (Chen et al. 2020).
Mixout: explicitly prevent the deviation from the pre-trained weight wp. Lee et al. (020)) pro- pose Mixout that stochastically replaces the model parameters with their pre-trained counterparts with probability p at each training iteration, which has been shown to improve the stability of fine- tuning and enhance the classification accuracy of the fine-tuned LMs downstream task.
4.3 JOINT LEARNING WITH MLM OBJECTIVE
Besides fixing or constraining the model parameters to the pre-trained counterpart, we can enforce the consistency between the pre-trained and fine-tuned LMs by utilizing the MLM objective. Previ- ous works have demonstrated that performing the MLM task before or during the fine-tuning process can yield better performance on the downstream tasks (Sun et al. 2019 Wiedemann al.) 2020 Ma jet al. 2021). In this work, to better preserve the pre-trained features, we jointly optimize the MLM objective in the fine-tuning phase: Lyoit Qmim Luim + Loss where Ommm is the scaling factor of the MLM loss. In addition to introducing the MLM objective, we propose three simple techniques to further strengthen the connection between the pre-trained model and fine-tuned model:
5

---

Published as a conference paper at ICLR 2023
Utilizing corpus of the pre-training phase: The dataset for the MLM task is not required to be labeled as for the downstream task, so we can choose any text dataset. In this work, we present two criteria for selecting Pu (x) of the MLM task. The first one uses the dataset of downstream tasks, referred to as JL-D. The second one introduces the corpus of the pre-training phase, referred to as JL.P. We suppose that using the corpus in the pre-training phase is helpful in preserving the pre- trained features, while increasing data diversity could enhance the model's uncertainty estimation under OD settings.
Distillation from the pre-trained model: Knowledge distillation (Hinton et al. 2015) has been shown be an effective technique to reap the benefits of powerful teacher model. In ad- dition to preserving the accuracy, Guo et al. (2021) illustrate that the calibration performance of the teacher model can be distilled into the student model and propose to distill from an ensemble of fine-tuned LMs for better calibration. Inspired by this, we perform knowledge distillation from the pre-trained language model when performing the MLM task. Specifi- cally, instead of calculating the MLM loss using the original text hard labels, use the KL-divergence DKL (Piim (I: aar Φu 0g) PP [ (2 Mi Φ, 0)) between the predictive distribution Pmlm (2ia 2: do; 0o) of the pre-trained language model and our model.
Regularization on the contextualized representation: Zhou & Srikumar 2022) validate that fine- tuning enlarges the distance feature space between samples from different classes. This behavior may increase the deviation of the fine-tuned model from the pre-trained model. To address this problem, we introduce a heuristic regularization by adding the L norm each training example's contextualized representation fa(z) (z) with regularization coefficient 312.
Putting Loint with the knowledge distillation and regularization term together, we get our final joint learning objective:
Li =0 (m)
Σ iEM
DKL (Pmlm (Ig MM Φo. 80) Pmlm (2y (i o,
(4)
+ p [log ((yla; d, v)] + Br. [x)())
Following previous works Desai & Durrett 2020 Park & Caragea 2022), we also apply label
smoothing (Szegedy et al., 2016) on the classification task, which can mitigate overconfident pre- dictions by distributing a fraction of probability mass of the ground-truth label equally to other non-ground-trum classes.
EXPERIMENTS
5.1 GENERAL SETUP
Datasets: We conduct experiments on three natural language understanding tasks: natural language inference (NLI), paraphrase detection (PD), and commonsense reasoning (CR). Each task consists of a pair of in-domain (ID) and out-of-domain (OD) datasets. Specifically, SNLI (Bowman et al., 2015) and MNLI (Williams al.) 2018) are ID and OD datasets for NLI: QQP (Shankar et al. 2017 and TwitterPPDB (Lan et al. (017)) ) are ID and OD datasets for PD; SWAG (Zellers et al., 2018 and HellaSWAG (Zellers al. 2019) are ID and OD datasets for CR. We use the same train/ Saiiiaalliaaii/ttee split for those six datasets published by Desai & Durrett (2020). For each task, we fine-tune the model using the ID training set and evaluate the model's performance with both ID and OD test sets. We use WikiText- 103 (Merity et al. 2016) the corpus of the pre-training phase for P.. The detailed statistics for each dataset can be found in Appendix A.
Setup: We follow the general training configuration provided by the Huggingface Transformers li- brary (Wolf et al. 2020). For parameter-efficient tuning methods, we use the default hyperparameter configuration provided by OpenDelta (Ding al. a grid search for learning rates on different tasks. For pre-trained weight decay (PWD), we follow the implementation of the RecAdam (Chen et al. 2020 which integrates the quadratic penalty be- tween the model parameters and the pre-trained parameters into the Adam optimizer (Kingma & Ba 2015). We tune the regularization strength \pwD for each task. For Mixout, tune the mixout
2022) library for all three methods and conduct

---

Published as a conference paper at ICLR 2023
probability Pmixout- For joint learning methods, conduct hyperparameter search for the mask probability Pmask, the scaling factor amim of the MLM loss, and the regularization coefficient 3n,- the contextualized representation. We also tune the hyperparameter db for label smoothing (ISS). We search all the hyperparameters on the validation set of each task independently. Completed setup details for each method each task can be found in Appendix A.2 All experiments are run for 3 training epochs and are deployed on a single NVIDIA A40 48G GPU within 3 hours to fine-tune a single model.
Bernoulli corruption distribution We
In this section, we use pre-trained RoBERTaRASI (Liu al. 2019) for all experiments. For each NLU task, we report accuracy and expected calibration error (ECE) on both ID and OD test sets. We evaluate "out-of-the-boe box" calibration of our method, which does not apply any post-hoc calibration methods such as temperature scaling (Guo al. 2017).
Evaluation:
Table 2: Out-of-the-box calibration results of different fine-tuning methods on in-domain (SNLI, QQP, SWAG) and out-of-domain (MNLI, TwitterPPDB, HellaSWAG) datasets. We report the av- eraged accuracy and ECE across five random fine-tuning runs. We also report the corresponding standard deviation in subscripts.
In- no-Domain
Out-of-Domain
RoBERTa-base
Acc
ECE
Ácc
ECE
Task: SNLUMNNI
2020
Baseline (Our rUn)
91.890.18
2.210.12
79.870.32
4.06g.
BABN Zhang et al. 2021
91-70
2.62
79.86
2.67
Mixup Park Caragen,
91.240.3
1.28p
78.86p.5
1.37
Adapter (Houlsby
2019
91.300,
1.320.20
79.170.0
1.980 .77
LoRA ct 2002
89.860 $9.869.23
1.28p.12
77.52p.25
1.70p.33
Tuning
Liany
89.08g.19
1.68p.as
76.09p 28
1.53g 39
Pre-trained Weight Decay
91 01.099.16
1.71p.15
78.35p. 30
0.87g.32
Mixout (Lee et 2020
90. 90.70o.07
1.70p.07
78.82p
1.140.30
DL. (wlo KD)
91.950.12
0.670.05
79.850 29
1.260.50
J1.PP (w/ KD)
91.74g.15
1.09p.15
79.36p.22 27
1.200.43
J1.P (w/ KD)+
91.780.09
1.48p.17
80.00p.15
1.90g.07
Task: QOP/TwiterPPDR
Baseline (Desai Dumret 2020
91.11
2.33
86.72
Baseline (Our
91.26g.13
2.44p.
86.15p.35
10.099.49
BABN Zhang et 2021
91.72
1.74
87.31
9.42
Mixup & Caragea
89.75g.6
2.18p.7
87.631.0
3.961.6
Adapter (Huusby
2019
90.18g.09
1.37p.
85.63p.
9.84g. .47
LoRA et ct alll 20221
88. 88.540.10
1.05p.09
85.51p.44
8.550.51
Prefix Tuning [Li Liang 2021
87 67..77.
1.46p.16
85. 85.26p.25
8, 8.02u.21
Pre-truined Weight Decay
$9.950.09
1.25p.09
85.28p.29
8.010.35
Mixout (Lee et 2020
89.770.00
1.17p.09
83.691.72
5.751.24
JL. (wla KD)
90. 42g.08
0.57p.12
86.67p.22
6.49g.44
(w/ KD)
90.50g.09
0.91o.50
86.50p.9 90
1.270.39
JL-P (w/ KD)
90.42o.
0.770.28
86.38p.93
2.211.06
Task: SWAG/HellaWWAG
Baseline (Deaii & Durrell 2020
82.45
1.76
41.68
11.93
Baseline
83.980,27
1.38p.25
42.99p.97
8,800.c
BABN ZZhang et al. 2021
83.
1.32
43.11
9.72
Mixup Park & Caragea, 2022
82. .2.
1.120.4
41.37
1.8Gg.9
Adapter (Houlsby
82.37o.
19255.19
43.911.03
11.130.70
LoRA (t al. [2021
80.0
5.120.15
43.17p.42
4.38n.90
Prefix luning (Li Liang 2021
80. 66.,66, 20
4.20p.56
42.86p.48 63
5.761.09
Pre-truned Weight Decay
81. .2
1.29p.15
42.731.34
8.210.67
Mixout (Lee et al. 2020
83. 170.12
0.78n.09
45 45.150.74
7.52n.6
JL-D (wl KD)
84.499.24
0.910.25
13.75p.92
7,641.
(w/ KD)
83.5
1.94p.12
44.78p.22
5.23n.43
JL. (w/ KD) LS
83.610.27
1.200.29
44.08p.61
1.620. 29
5.2
MAIN RESULTS
We summarize our results in Table 2 Our baselines include full fine-tuning and two advanced meth-
ods based on the pre-trained language models: Bayesian Attention Belief Networks (BABN) (Zhang et al. 2021 I and Mixup (Park & Caragea 2022). We copied the results of the same setting from the
For the MLM objective in joint learning method, we only perform the mask operation instead of the 80- 10-10 strategy as in the pre-training phase.
7

---

Published as a conference paper at ICLR 2023
original paper. We also report our full fine-tuning runs following the default behavior of the training scripts provided by the Huggingface Transformers library. We present our experimental results for pre-trained weight decay, Mixout, JL-D (w/o KD) and JL-P (w/ KD)
As shown in Table 2 full fine-tuned models generally have lower in-domain ECE compared to out- of-domain, which illustrates that the fine-tuned language models tend be overconfident under OD settings. BABN exhibits better generalization performance than deterministic methods, but it has a limited effect the OD calibration. Mixup significantly lowers the ECE under both ID and OD settings while preserving comparable accuracy to vanilla fine-tuning. Besides, all of the methods that preserve the pre-trained features in different ways are generally more calibrated than full fine- tuning, especially under the OD settings, which matches our expectations that pre-trained features are helpful to better model the predictive confidence for OD samples. In addition to achieving competitive performance, parameter-efficies tuning methods have advantages in calibration over full fine-tume tuning. Fine-tuning with regularization to pre-trained models in the parameter space also significantly improves calibration. However, Table and Table 5 that this requires maintaining a relatively high constraint strength to the pre-trained weights throughout the fine-tuning process, which leads to large loss of raw quality in some cases, e.g., Mixout QQP/TwitenPPDBB.
Notably, the proposed joint learning (JL) methods outperform previous calibration methods in ECE across three tasks simultancously under both ID and OD settings, which suggests that it may be more effective to encourage fine-tuned models to be consistent with pre-trained models in the function space. In addition being well-calibrated, JL models achieve the best accuracy in both ID) and OD settings on NLI and CR tasks and maintain accuracy within < 1% drop compared to vanilla fine- tuning on the PD task. We also highlight that the JL models have relatively low standard deviations for both accuracy and ECE compared to other methods.
5.3 PRESERVING PRE-TRAINED FEATURES HELPS CALIBRATE FINE-TUNED LMS
TwitterP008 Histegram
Lebs MoolE
New sn otter
Confidence
[LO 1o24e7eer020
3 10000
WikTestt 103 Confidence Histogram
Le 3333
Figure 4: t-SNE visualization for hidden repre- sentations of the sampled inputs from different domains given by different models fine-tuned on QQP.
Confidence histogram for OD sam- ples (up) and outlier samples (down) of differ- ent models fine-tuned on QQP.
Figure 5:
As shown in Figure 4 compared full fine-tuning, the hidden representations for OD samples of the methods described in $4rr Jare more consistent with the PLMs, showing that they are able to preserve the pre-trained features and can mitigate catastrophic forgetting. The representations for outlier samples are more distinguishable from the OD samples, as the pre-trained model does. Figure 5 illustrates that preserving the pre-trained features helps calibrate the fine-tuned models by mitigating the overconfident tendency to the OD and outlier samples discussed in $3.2) Specifically, parameter- efficient tuning, pre-trained weight decay, and JL-D slightly alleviate the overconfidence toward the OD and outlier samples, while JL-P and Mixout significantly improve the fine-tuned models' ability to model the predictive confidence for OD and outlier samples. Among these methods, JL-P with knowledge distillation is shown be the most effective regularization that achieve low ECE and competitive raw quality at the same time. Nevertheless, it requires access to the corpus of the pre-training phase, which may not be available in some cases.
There is clearly more room for further improve these methods. Mixout exhibits a promising ability model the confidence of OD and outlier samples properly and improve the OD generalization
We present the results of JL.D without knowledge distillation and IL- with knowledge distillation in Table 2 We show the full results for JL in A.3.T
8

---

Published as a conference paper at ICLR 2023
(e.g., the result on HellaSWAG) by taking advantage of the pre-trained models. However, Mixout fails to balance the preservation of pre- trained features and the learning of downstream tasks in some cases, which leads to low accuracy and high ECE. Figure demonstrates that the outlier samples can be easily distinguished from the hidden representations of the JL.-D models, but they do not provide reasonable predictive confidence JI..P, shown in Figure We leave these questions for future work.
6 RELATED WORK
Uncertainty estimation of PLMs. Previous works have demonstrated that PLMs can be benefi- cial for improving the robustness and calibration on downstream tasks compared to non-pre-trained models (Hendrycks et al. 2020 Desai & Durrett 2020). However, PLMs can still fail to model their predictive uncertainty on downstream tasks. For example, Desai & Durrett (020): Kong et al.
;2020) Guo et al. (202T) have shown that fine-tuned masked language models (e.g.. BERT (Devlin
et al. 2019), RoBERTa (Liu et al. 2019)) are overconfident on text classification tasks, while Jiang ct al. 2021 has shown that powerful generative PLMS (e.g., T5 (Raffel et al. 2020). BART (Lewis)
et al. 2020). and GPT-2 (Radford et al., (219h) are poorly calibrated on QA tasks. In this work, we study a specific failure case of fine-tuned LM in which the models are overconfident of the OD and outlier samples due to catastrophic forgetting.
Calibrating fine-tuned LMs. Several approaches have been developed to calibrate the fine-tuned LMs on NLU tasks. For instance, DDssa & Durrett (2020) demonstrate that temperature scaling and label smoothing can improve the calibration of the models in ID and OD settings, respectively. He al. (20215) introduce new discriminative objective under the noise contrastive estimation (NCE) framework to jointly train an energy-based model defined on the classifier, which leads to better ID calibration. Fan et al. (2020): Zhang et al. (202I) model the attention weights as random variables and design a series of methods to optimize the stochastic attention layer with variational inference, which yields better performance in accuracy and calibration compared to vanilla deterministic atten- tion layers. KKong et al.] (2020): Park & Caragea (2022) adopt Mixup (Zhang et al. 2018) to calibrate fine-tuned language models and exhibit effectiveness in calibration under both ID and OD settings. We tackle the problem of calibrating the fine-tuned LMs from new perspective by focusing specifi- cally on the pre-training & fine-tuning paradigm and validate that preserving the pre-trained features is an effective way improve the fine-tuned LMs' calibration.
Benefit from mitigating catastrophic forgetting. Previous works have shown that mitigating catas- trophic forgetting of PLMs can be helpful for various aspects of downstream tasks. For example, Chen et al. 22000); Lee et al. (2200) show that constraining the models' parameters closer to the pre-trained ones can improve the training stability and performance of fine-tuned LMs down- stream tasks. Xie et al. (2021) validate that standard fine-tuning can destroy the output structure of pre-trained generative denoiser such BART and show that preserving pre-trained features via lightweight fine-tuning can improve out-of-distribution generalization on downstream generation tasks. Dong et al. (202T) show that the pre- pre-trained features of PLMs are beneficial for robust ob- jective model and improve the adversarial robustness of fine-tuned language models by maximizing the mutual information between the hidden representation of the pre-trained and fine-tuned models during the whole fine-tuning process. Our work specifically focus on uncertainty estimation of the fine-tuned LMs and makes a complementary contribution that the calibration of fine-tuned LMs can be improved by mitigating catastrophic forgetting.
7 CONCLUSIONS
In this work, we show that PLMs that pre-trained with large corpora are inherently well-calibrated on the MLM task while the fine-tuned LMs suffer from overconfidence due to catastrophic forgetting. Our experimental results validate that preserving the pre-trained features can better calibrate the fine- tuned LMs. We hope work can draw more attention to the deeper exploitation of the pre- trained features learned by PLMS and contribute to building safe and reliable NLP systems for rorl-world applications.

---

Published as a conference paper at ICLR 2023
REPRODUCIBILITY STATEMENT
We have provided detailed setup for experiments in $3.1
332
$5.1
A.1
and A.2 we submit
our codes as the supplementary material. The information we provided is sufficeent to reproduce our results.
ACKNOWLEDGMENTS
Thanks Peng Cui, Zhijie Deng, Wenbo Hu, Weize Chen, Xu Han for valuable discussions. This work was supported by the National Key Research and Development Program of China (220AAA0106302)); NSF of China Projects (Nos. 62061 I 136001, 61620106010, 62076145, U19B2034, U1811461, U19A2081, 6197222); grant from Tsinghua Institute for Guo Qiang; the High Performance Computing Center, Tsinghua University. J.Z. was also supported by the XPlorer Prize.
REFERENCES
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. A large anno- tated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, 2015.
Sanyuan Chen, Yutai Hou, Yiming Cui, Wanxiang Che, Ting Liu, and Xiangzhan Yu. Recall and learn: Fine-tuning deep pretrained language models with less forgetting. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020.
Morris H DeGroot and Stephen E Fienberg. The comparison and evaluation of forecasters. Journal of the Royal Statistical Society: Series D (The Statistician), 1983.
Shrey Desai and Greg Durrett. Calibration of pre-trained transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), 2019.
Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, Jing Yi, Weilin Zhao, Xiaozhi Wang. Zhiyuan Liu, Hai- Tao Zheng, Jianfei Chen, Yang Liu, Jie Tang, Juanzi Li, and Maosong Sun. Delta tuning: A comprehensive study of parameter efficient methods for pre-trained language models, 2022.
Xinshuai Dong, Anh Tuan Luu, Min Lin, Shuicheng Yan, and Hanwang Zhang. How should pre- trained language models be fine-tuned towards adversarial robustness? Advances Neural formation Processing Systems, 34:4356-4369, 2021.
Kawin Ethayarajh. How contextual are contextualized word representations? Comparing the ge- ometry of BERT, ELMo, and GPT-2 embeddings. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 2019.
Xinjie Fan, Shujian Zhang, Bo Chen, and Mingyuan Zhou. Bayesian attention modules. Advances in Neural Information Processing Systems, 2020.
Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoee.. Mask-predict: Parallel decoding of conditional masked language models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 2019.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In Proceedings of the 34th International Conference Machine Learning, 2017.
10

---

Published as a conference paper at ICLR 2023
Han Guo, Ramakanth Pasunuru, and Mohit Bansal. An overview of uncertainty calibration for text classification and the role of distillation. In Proceedings of the 6th Workshop on Representation Learning for NLP (RepL4NLP-2022) 2021.
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Decoding-enhancee bert with disentangled attention. In International Conference on Learning Representations, 202 la
Tianxing He, Bryan McCann, Caiming Xiong, and Ehsan Hosseini-Asl. Joint energy-based model training for better calibrated natural language understanding models. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, 2021b.
Yun He, Ziwei Zhu, Yin Zhang, Qin Chen, and James Caverlee. Infusing Disease Knowledge into BERT for Health Question Answering, Medical Inference and Disease Name Recognition. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020.
Dan Hendrycks, Xiaoyuan Liu, Eric Wallace, Adam Dziedzic, Rishabh Krishnan, and Dawn Song. Pretrained transformers improve out-of-distribution robustness. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network, 2015.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, An- drea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for NLP. In Proceedings of the 36th International Conference Machine Learning, 2019.
Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text elassification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), 2018
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021.
Ganesh Jawahar, Benoît Sagot, and Djamé Seddah. What does BERT learn about the structure of language? Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019.
Zhengbao Jiang, Jun Araki, Haibo Ding, and Graham Neubig. How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answering. Transactions of the Association for Computational Linguistics, 2021.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd Interna- lional Conference on Learning Representations, 2015.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Baabska-Barwinska Demis Hass- abis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 11((13)33521-355 2017.
Lingkai Kong, Haoming Jiang, Yuchen Zhuang, Jie Lyu, Tuo Zhao, and Chao Zhang. Calibrated language model fine-tuning for in- and out- -ii---iiittiittii data. In Proceedings of the 2020 Con- ference on Empirical Methods in Natural Language Processing (EMNLP), 2020.
Anders Krogh and John Hertz. simple weight decay can improve generalization. Advances in Neural Information Processing Systems, 1991.
Wuwei Lan, Siyu Qiu, Hua He, and Wei Xu. A continuously growing dataset of sentential para- phrases. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, 2017.
Cheolhyoung Lee, Kyunghyun Cho, and Wanmo Kang, Mixout: Effective regularization to finetune large-scale pretrained language models. In International Conference on Learning Representa- fions, 2020.
11

---

Published as a conference paper at ICLR 2023
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising gruuence-t-sequenee pre- training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020.
Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario Saško, Gunjan Chhablani, Bhavitvya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh, Can- wen Xu, Nicolas Patry, Angelina McMillan-Major, Philipp Schmid, Sylvain Gugger, Clément Delangue, Théo Matussière, Lysandre Debut, Stas Bekman, Pierric Cistac, Thibault Gochringer, Victor Mustar, François Lagunas, Alexander Rush, and Thomas Wolf. Datasets: A community library for natural language processing. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, 2021.
Xiang Lisa Li and Perey Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), 2021.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach, 2019.
Iyya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2019.
Xinyu Ma, Jiafeng Guo, Ruqing Zhang, Yixing Fan, Xiang Ji, and Xueqi Cheng. Prop: Pre-training with representative words prediction for ad-hoc retrieval. In Proceedings of the 14th ACM Inter- national Conference on Web Search and Data Mining, 2021.
Michael McCloskey and Neal J. Cohen, Catastrophic interference in connectionist networks: The sequential learning problem. volume 24 of Psychology Learning and Motivation, pp. 109-165. Academic Press, 1989.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016.
Mahdi Pakdaman Naeini, Gregory F. Cooper, and Milos Hauskrecht. Obtaining well calibrated probabilities using bayesian binning. In Proceedings of the Twenty-Ninth AAAI Conference Artificial Intelligence, 2015.
Alexandru Miculescu-Mizil and Rich Caruana. Predicting good probabilities with supervised learn- ing. In Proceedings of the 22nd international conference on Machine learning, 2005.
Seo Yeon Park and Cornelia Caragea. On the calibration of pre-trained language models using mixup guided by area under the margin and saliency. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2022.
Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lec, and Luke Zettlemoyer. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long Papers), 2018.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67, 2020.
S.P.C.W Sandagiri, B.T.G.S Kumara, and Banujan Kuhaneswaran. Deep neural bewwork-based ap- proach to identify the crime related twitter posts. In 2020 International Conference on Decision Aid Sciences and Application (DASA), pp. 1000-1004, 2020.
12

---

Published as a conference paper at ICLR 2023
Iyer Shankar, Dandekar Nikhil, and Csernai Kornél. Quora question pairs. 2017.
Chi Sun, Xipeng Qiu, Yige Xu, and Xuanjing Huang. How to fine-tune bert for text classification?, 2019.
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Sam Bowman, Dipanjan Das, and Ellie Pavlick. What you learn from context? probing for sentence structure in contextualized word representations. In International Conference on Learning Representations, 2019.
Lifu Tu, Garima Lalwani, Spandana Gella, and He He. An Empirical Study on Robustness to Spurious Correlations using Pre-trained Language Models. Transactions of the Association for Computational Linguistics, 2020.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 2008
Gregor Wiedemann, Seid Muhie Yimam, and Chris Biemann. UHH-LT at SemEval task 12: Fine-tuning of pre-trained transformer networks for offensive language detection. I Proceedings of the Fourteenth Workshop on Semantic Evaluation, 2020.
Georg Wiese, Dirk Weissenborn, and Mariana Neves. Neural domain adaptation for biomedical question answering. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), 2017.
Adina Williams, Nikita Nangia, and Samuel Bowman. A cover-cov challenge corpus for sen- tence understanding through inference. In Proceedings of the 2018 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long Papers), 2018.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-ue natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, 2020.
Sang Michael Xie, Tengyu Ma, and Percy Liang. Composed fine-tuning: Freezing pre-trained denoising autoencoders for improved generalization. In Proceedings of the 38th International Conference on Machine Learning, 2021.
Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. SWAG: A large-scale adversarial dataset for grounded commonsense inference. In Proceedings of the 2018 Conference on Empir- ical Methods in Natural Language Processing, 2018.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can a ma- chine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019.
Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empiri- cal risk minimization. In International Conference on Learning Representations, 2018.
Shujian Zhang, Xinjie Fan, Bo Chen, and Mingyuan Zhou. Bayesian attention belief networks. In ICML, pp. 12413-12426, 2021.
Yichu Zhou and Vivek Srikumar. A closer look at how fine-tuning changes BERT. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2022.
13

---

Published as a conference paper at ICLR 2023
Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In The IEEE International Conference on Computer Vision (ICCV), December 2015.
14

---

Published as a conference paper at ICLR 2023
A APPENDIX
A.1
DATASET DETAILS
The general information of in-domain and out-of-domain datasets of the three NLU tasks are shown below:
Natural Language Inference: Stanford Natural Language Inference (SNLI) (Bowman et al, 2013) requires the model to learn the textual entailment by predicting the relationship between given premise and hypothesis among entailment, contradiction, or neutral. Multi-Gema Natural Language Inference (MNL.) (Williams et al. 2018) shares the same task form SNLI, but the samples from more diverse domains than SNLI
Paraphrase Detection; Quora Question Pairs (QQP) (Shankar et al. 2017 contains question pairs from Quora. The model needs to discriminate whether the given pairs are semantically equivalent. TwitterPPDB (Lan al. 2017) the out-of-domain dataset that collects sentence pairs shared with the same URLS.
Commonsense Reasoning: Situations With Adversarial Generations (SWAG) (Zellers al. 2018) is a common sense reasoning task that requires the model to choose the most plausible continuation of sentence given four possible candidates. HellaSWAG (Zellers al. 2019) is designed more challenging commonsense reasoning task for the pre- pre-trained language models built with Adversarial Filtering.
The statistic of the datasets for both MLM and NLU tasks are shown in Table 3 For the datasets of NLU tasks (SNLI/MMLL QQP/TWittePPPDBB SWAG/HellaSWAC) we use the published ver- sion by Desai & Durrett (2020). For WikiText-1033. 103, we use the version provided by Huggingface Datasets (Lhoest et al. 2021) library. Note that the training splits of the OD datasets (MNLI, Twit- terPPDB, HellaSWAG) are not used.
Table 3: The size of the training, validation, test splits and number of labels for all datasets.
Dataset
#Train
#Validation
#Test
#Labels
SNLI Bowman al. 2015
549,368
4,922
4,923
MNLI Williams al. 2018
392,702
4,908
4.907
QQP
Shankar et al. 2017
363,871
20,216
20,217
2
TwitterPPDB (Lan et al. 2017)
46,667
5,060
5,060
2
SWAG (Zellers et al. 2018)
73,547
10,004
10,004
4
HellaSWAG (Zellers et al.
2019)
39,905
5,021
5,021
4
WikiText-103 (Merity et al.
2016
1,801,350
3,760
4,358
A.2
SETUPS
A.2.1
SETUP FOR FULL FINE-TUNING AND CHAMETER-EEFICIREE TUNING
We conduct the experiments with the Huggingface Transformers library WWoff al. 2020). For parameter-efficient tuning methods, we use the implementations of OpenDelta (Ding et al. 2022) library for the three parameter-efficien tuning methods (Adapter, LoRA, Prefix Tuning). We use the same default hyperparameters provided by the OpenDelta library for each method across all three tasks
All fine-tuning methods are trained with the Adam W optimizer (Loshchilov & Hutter 2019). For full fine-tuning, we use learning rate of le-5 across all tasks. For parameter-effiiint tuning meth- ods, we search the learning rate among (1e-5, 2e-5, 5e-5, le-4, 2e-4, 5e-4). We use the one that has the best ID accuracy on the validation set, which is the widespread application scenario for the parameter-efficien tuning methods. Table [ shows the learning rate used to fine-tune all the methods. For other training hyperparameters, we follow the default setup of the Huggingface Trans-
Thhe default hyperparameters can he found in the definition of each class on the document: https:// opende 1 aa. readthedocs. wt/////aatee/////daaaaaaaaaaaaa
html
15

---

Published as a conference paper at ICLR 2023
formers library In specific, set batch size of 32, a maximum sequence length of 256, and weight decay of 0.1. We use a linear decay learning rate scheduler without warmup and do not apply gradient clipping. Note that the reported baselines for full fine-tuning and Mixup (Desai & Durrett 2020 Park & Caragea 2022) in Table do not use learning rate scheduler and use gradient clip of 1.0 compared to runs. There also some minor differences, such as the padding strategy for input texts between our fine-tuning setup and theirs.
Learning rate of fine-tuning methods on each task. The models are fine-tuned on the training split of SNLI for natural language inference (NLI), QQP for paraphrase detection (PD), SWAG for commonsense reasoning (CR).
Table 4:
Task
Full-FT
Adapter
LoRA
Prefix Tuning
NLI
le-5
2e-4
2e-4
le-4
PD
le-5
2e-4
2e-4
le-4
CR
le-5
le4
le-4
2e-4
A.2.2 SETUP FOR THE JOINT LEARNING METHOD
We conduct hyperparameter search with the ID/OD validation set for the models that joint learning with the MLM objective (JL). Specifically, set learning rate of le-5 and batch size of 32 across all three tasks as Desai & Durrett (2020) does, except for fine-tuning JL-P with label smoothing on SWAG, where
adopt the same setup described in A.2. except for fine-tuning JL-P on QQP, where we do not use a learning rate scheduler. For the hyperparameter of the JL models, we search the scaling factor of the MLM loss Oimlm E {0.11, 0.3, 0.5, 1, 2, 4, 5]. the coefficient of the regularization term on contextualized representation 3r.= {le-9, 1e-8, 1e-7, 1e-6, le-5, lc-4). the masking probability for a sentence Pmask E (0.05, 0.15, 0.3, 0.4, 0.5.0.6,,, 0.6}. and the hyperparameter of label smoothing T E {0.01, 0.03, 0.05) for each task. We also search the maximum sequence length of the MLM task for JL-P and the batch size of MIL tasks for both JL-D and JL-P. In detail, we use a batch size the MLM task 32 on NLI and PD tasks and a batch size for the MLM task of 8 on the CR task. The maximum sequence lengths of the MLM task for JL-P are set to 32/32/64 for NLL, PD, and CR tasks, respectively. Training a single model for epochs can be done in two hours with a single NVIDIA A40 48G GPU. We present the detailed hyperparameter setup of each method on each task in Table
larger learning rate of 5e-5. For other training parameters,
Table 5: Hyperparameters of models for each NLU task.
Omim
Pransk
81,2
Task: SNLI/MNLI
JL-D
0.3
0.4
le-5
JL-P
0.3
0.4
le-5
LS
0.5
0.6
le-8
0.03
Task: QOP/TwitterPPDB
JL-D
0.15
le-5
4
0.15
1e-7
0.15
1e-9
0.01
Task: SWAG/HellaSWAG
JL-D
1
0.3
le-9
JL-P
3
0.3
le-9
JL-P
3
0.05
1e-4
0.05
A.2.3 SETUP FOR REGULARIZATION WITH PRE-TRAINED WEIGHT
Pre-trained Weight Decay: We tune the regularization strength APwD {0.11 1, 10, 50, 1003 and use XPwD 10/20/1 1 for SNLI, QQP, and SWAG, respectively.
'The NIJ and PD tasks correspond to the text classification setting, while the CR task corresponds to the multiple choice criterion of huggingface transformers library.
16

---

Published as a conference paper at ICLR 2023
Mixout:
We tune the mixout probability Pmixout E (..1,0.3, 0.5, 0.7, 0.9) and use Pmixout 0.9 for all three NLU task.
For other training parameters, the same default setup described in A.2.2
A.2.4 COMPARISON OF TRAINING TIME
We compare the training time of different methods using single NVIDIA A40 48G GPU. We use the full fine-tuning as a baseline (1x). The time cost of 3 training epochs using full fine-tuning is 1/0.8/0.3 GPU hours for SNLI/QOP/SWAG. We present the time cost of different methods in Table 6
Table 6: Comparison of training time of different methods.
Task
Full-FT
Parameter- Parameter-Efficient Tuning
PWD
Mixout
JL.
NLI & PD
Ix
0.6x~~1x
1x
2.4x
2x
CR
1x
0.75x
1x
1.5x
1.5x
17

---

Published as a conference paper at ICLR 2023
A.3
ADDITIONAL EXPERIMENTAL RESULTS
A.3.1
ABLATION STUDY FOR THE JOINT LEARNING METHOD
85
2000
Arcuracy
75
1100
50-
1000
Gran
2500
2000
Accuracy
75
1100
500
Brim
Figure 6: Accuracy, ECE and 1.2 distance to the pre-trained hidden representations ([))(()))
fon (a= of JL-D (up) and JL.P (down) over different scaling factors O'uim of the MLM Loss in ID and OD settings.
Effeet of the MLM objective: As shown in Figure 6 compared to vanilla fine-tuning (amil
0), introducing the MLM objective lowers the ECE effectively in both ID and OD settings with relatively small effect on accuracy. As the magmitude of the MLM loss increases, the features of the fine-tuned models are closer to those of the pre-trained models, reflected in both the geometry of feature space (shown Figure 4 and euclidean distance between the representations of pre- trained and fine-tuned models (shown in Figure(6). and the ECE of the fine-tuned models decreases. However, when the weight of the MLM loss becomes too large compared to the classification loss, the performance of the NLU task will be apparently damaged. For the overconfidence toward the OD samples and outlier samples illustrated in $3.2 as shown in Figure)) 7 can be mitigated by using relatively small magnitude of Omimm while increasing it can have
significant effect.
Effect of introducing corpus of pre-training phase: From TableDand Figure 6 we observe that performing the MI.M task with the corpus of the pre-training phase has lower OD ECE all three tasks. This confirms our belief that using the corpus of the pre-training phase can preserve more helpful features from the PLMs. We also notice that sometimes JL-P degrades the ID calibration (e.g., calibration on SWAG in Table however, applying label smoothing on downstream tasks can relieve this negative effect. An interesting observation is that although the hidden representations given by both JL-D and JL-P can distinguish outlier samples easily, the JL-D holds higher confidence toward outlier samples than JL-P. As shown in Figure 7 compared to JL-P, the JL-D are more confident in samples from the BookCorpus (Zhu et al. (015)) dataset that is not seen by both JI.-P and JL.-D in the fine-tuning phase for fair comparison, which suggests utilizing the pre-training phase's 's corpora that more diverse than the training datasets are helpful for the fine-tuned LMs to model the confidence of the outlier samples better.
Effect of knowledge distillation from the pre-trained model: edge distillation has limited effect on JL-D but enhances the accuracy and ECE of JL-P in most cases. We hypothesize that performing the MLM task with the corpus that is distinct from the downstream datasets can hurt relatively more performance on the downstream tasks, and distilling from the pre- trained model's predictive distribution might be smoother and more effective regularization.
Table 7 shows that applying knowl-
18

---

Published as a conference paper at ICLR 2023
Effect of regularization the contextualized representation: As shown in Figure 8 the intro- duced heuristic regularization term could improve the ECE of JL-D models under both ID and OD settings by smoothing the models' predictive confidence. The effect of this regularization term is similar to applying temperature scaling under the ID validation set, where large magnitudes can lead overly conservative prediction confidence. We also find that a proper choice of β3a marginally benefit the accuracy under ID and OD settings. However, the effect of this term on Full-FT models limited.
1.0
B-D--wittrrWC
1.0
JL- Nel-123
10
Ju- BoO-BrokCCrpus
0.4
2-2
0.2
0.2
0.2
205
0.6
0.7
0.8
0.9
1.0
0.5
0.5
0.7
0.6
0.9
1.0
0.
0.6
0.7
0.5
10
Contidence
Gani denic
1.0
1.0
JL-+WWi: 1oot-103
1.
1-M-bokkCorpuu
20.
0.6
0.7
0.9
0.9
1.0
0.5
0.6
0.7
0.6
0.9
1.0
0.5
0.6
0.7
0.5
Gart denie
Figure 7: Confidence histogram for OD (TwitterPPDBB samples and two sets of outlier (WikiText- 103, BookCorpus) samples of JL-D and JL-P.
TuitterPDD
PUILFT.GOP
Full TtwtFFE
1.0
1.0
- Ro le:6
0.9
-4-
Ro le:5
Ro le-4
0.8
--. Zors Emoo
0.8
0.8
0.8
0.7
0.7
0.7
0.7
0.4
Confidence
Confidence
Confidence
Confidence
Figure Reliability diagram of JL-D and Full-FT models with different By² values.
Table 7: The results with/without knowledge distillation for JL models.
In- In-Domain
RoBERTa base
Ace
ECE
Acc
ECE
Task: SNLI/MLI
I[.D (w/o KD)
91.956.12
0.67
79.856.29
1.26p.50
I[.DD (w/ KD)
91.64p.19
0.75e.21
79.99o.30
1.31
JL-P (w/o KD)
91.72..14
1.12o.17
29.45n.25
1.190.17
JL-P (w/ KD)
91.74p.15
1.090.15
79.36n
1.20o.43
Task: OOP/TwitterPPDS
JL-D (w/o KD)
90.42o.a
0.57o.12
86.67o
6.490.44
JL-D (w/ KD)
90.
60.86p.04
0.50p.0s
$6.26n.35
6.06p.52
JL-P (w/o KD)
19.755.19
0.78o.47
86.77o.61
2.861.01
JL-P (w/ KD)
90.50b.a9
0.91
86.50o.90
1.270.a9
Task: SWAG/HellaSAA
IL-D (w/o KD)
84.49b.24 29
0.91
43.75o.92
7.640.43
JL- (w/ KD)
84.03o.14
0.58p.0s
44.680.49
10.080.a6
_L-P (w/o KD)
82.39u.a3
2.85g.11
42.680.4
7.64p.a7
JL-P (w/ KD)
83.51u.a5
1.940.12
44.78u.22
5.230.43
19

---

Published as a conference paper at ICLR 2023
A.3.2 SAMPLING FROM JOINT LEARNING MODELS
Since the model encouraged to learn both discriminative and generative representations, can perform conditional generation with the MLM head and classifier ltp. In this work, we choose Mask-Predict (Ghazvininejad al. 2019) as our basic sampling algorithm, which can incorporate the rejection sampling framework to generate samples from the conditional distribution p(æly" given the desired label y" We show the detailed sampling process Alg
Algorithm 1: Mask Predict with Rejection Sampling
Input: Number of iterations T, Text Encoder fo. MLM head yo. Classifier hi, Desired label 8" Number of iiitial masked tokens N.
Initialize: a²( [MASK] N
3 for t < to
do
// Mask-Predict iteration // Determine number of tokens to mask at iteration t // Rejection sampling loop
n < S(N,
repeat
(())
// Select prediction with highest probability for every [MASK] token. M, arg min: xi (z)))))))), n)
+ go (o(fo(z(()))
// Mask tokens with the lowest probability scores until u(0, 1), (oftmax(h,((())))111
(1) + [MASK]
(f+1) (())
10
11
end
12 Output: (TT) p(zly")
Table[]shows the samples generated by JL-D models with the sampling algorithm described above. The JL-D models are able generate text with the given labels, which be seen diagnostic for the model. For example, the models prefer to generate contradictory hypotheses by changing the objective's entity or adjectives with high confidence and tend to copy the prefix to generate positive textual entailment, which may expose some spurious correlations that the models rely on when performing the NLU tasks (Tu al. 2020).
20

---

Published as a conference paper at ICLR 2023
Table 8: Samples generated by JL-D models with SNLI and QQP test set using the Mask-Predict algorithm with rejection sampling. For SNLI, the model generates hypotheses given class labels fnone (-), entailment, contradiction, natural] and premise prefixes. For QQP, the model generates sentences given class labels (none (-), non-paraphrase (non-para), paraphrase (para)) and question prefixes. We mark the generated samples whose assigned label is consistent with the given label using and mark the failure cases where the assigned label is not consistent with the given label using []. We also report the corresponding confidence of the class of the generated text after the label.
Text Prefix:
A mountain biker rides up a hill on a red bicycle.
[-]
A mountain biker rides bike on hill.
Entailment, 99% %
A mountain biker rides bike up hill.
Contradiction, 99%
A mountain biker rides downhill on a blue bicycle.
Natural, 65%
A mountain biker is trying to climb a hill.
Text Prefix:
A man plays the french horn his pianist plays the supporting melody on stage.
[-]
A man is playing a french horn for concert.
Entailment, 97%]
A man is playing french horn on a stage.
Contradiction, 99%
A man is playing o flute for crowd.
[Nauural, 91%
A man is playing song concert stage.
Text Prefix:
Two young
in unusual clothing jumping in gym.
[-]
Two men are playing baskethall.
[Entailment, 96%]
Two are jumping arourd
Contradiction, 90%
Two are jumping outside.
Natural, 44%
Two are playing basketball.
Text Prefix:
blue gray race driving on dirt track.
[-]
is driving dirt track.
Entailment, 98%
race is driving dirt track.
Contradiction, 99%
race is parked dirt track
Natural, 66
race is racing on dirt track.
Text Prefix:
buy poses karute form and uniform.
[-]
boy is practicing karate.
Entailment, 75%
boy is practicing karate.
Contradiction, 2%
boy is a costune.
Natural, 25%
boy is practicing karute.
Text Prefix:
Four females wearing helments are riding on an ATV.
[-]
Four females are wearing helments on an ATV.
Entailment, 98%
Four females are wearing helments on an ATV.
Contradiction, 99%
Four females are riding on a horse the park.
Natural,, 99%
Four females are riding a ATV in the desert.
Text Prefix:
How does cloud computing work!
[-]
How does cloud computing in India work?
Noo--arapphasee, 95%
How does Google think cloud computing work?
[ [Paraphrase, 74%
How does I understond cloud computing work?
Text Prefix:
Do you think time travel is possible?
[-]
Do you think space and time travel is possible?
Nonn-arrpphraee, 73%
Do you think gravity can make time travel possible?
possibbl?
PParaphrase, 98%
Do you think it possible to time travel?
Text Prefix:
What good data analysis book?
[-]
What good free data analysis book?
Non-Paraphrase,, 78%
What good book for analysis books?
Paraphrase, 67%
What best free data analysis book?
Text Prefix:
Feeling bored. What do?
[-]
What get bored?
Non-Paraphrase, 97%
What should do to myself?
Paraphrase, 28% ]
What shoudd start to do?
21


Reviews:
[{"review_id": "Y8wg5_TJRRm", "reviewer": "Reviewer_Kwo1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies the calibration problem of BERT-like models on classification tasks. They show that the problem is created mainly during fine-tuning - the pre-trained model itself is well calibrated on the MLM task it is trained on. This MLM knowledge and calibration is destroyed during fine-tuning (a process called catastrophic forgetting in the literature). \n\nTo solve the calibration problem, the authors propose to continue fine-tune the model on the MLM task jointly with the actual classification \ntask at hand. \n\nExperimentations on 3 datasets and both in-domain and out-of-domain settings show that the proposed method yield lower calibration error compared to other methods (that do not use post-training calibration). In terms of raw quality, the proposed model perform on par with the existing method (and frequently slightly better). \n\n", "summary_of_the_review": "- The paper is well executed, but the task of reducing calibration error without post processing is quite narrow and has low impact. The approach of jointly training classification and MLM loss is not novel, but the impact on reducing calibration error is somewhat novel. ", "strength_and_weaknesses": "## Strengths\n- The task of calibrating the results of a large language model is important and necessary in some practical applications. \n- The observation that the problem of mis-calibration is created during fine-tuning is interesting and novel. \n- The experimentation was done on multiple datasets and in two different settings. The authors run fine-tuning 5 time and report average and standard deviation, showing a more complete picture compared to the results of a single run. \n\n## Weaknesses\n- It is unclear why post-processing techniques, such as temperature scaling, are insufficient, that we need a new training paradigm for overcoming this post processing step, especially that temperature scaling is quite lightweight. This makes the overall contribution quite limited. \n- Using MLM loss in addition to the standard classification loss is known (for a long time) to produce better results. Most work run an additional pre-training step on the target dataset, e.g., https://arxiv.org/pdf/1905.05583.pdf, https://arxiv.org/pdf/2004.11493.pdf, but joint training is also a known technique (which performs slightly worst, so less cited. Still, it was already published on e.g., https://dl.acm.org/doi/pdf/10.1145/3437963.3441777). This makes the comparison to the baseline unfair, since additional (unsupervised) training data was used. \n- It is unclear what is the performance implications are on the training time. If the training scheme alternates between MLM and classification losses, this would result in 2x longer training time, which is a significant impact. This should be at least clarified in the paper. \n- It is unclear how Section 5.1 is related to the rest of the paper (except for naming the algorithm), given that after all complex formulas the paper uses the standard MLM loss, and then move to knowledge distillation loss. Clarifying the connection to the rest of the paper (or dropping this part) would make it easier to follow the paper's contribution. \n- Table 1, showing that after fine-tune the model is unable to do the original task, is not by itself surprising or interesting. The question is how this is impacting calibration on the classification task, which is not directly related to the quality of the model on the MLM task. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "DIDxIv83xiX", "reviewer": "Reviewer_YFTC", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies the ECE of language models trained using different recipes, including the pretrained model, finetuned model and parameter-efficient finetuned models. They find that the finetuned model has larger ECE and tend to be over confident when making predictions. The paper proposes a baseline method of mixing the finetuning objective and the pretraining objective during finetuning to address this and this method works to lower ECE. ", "summary_of_the_review": "The paper has made some good observations on how LMs training differently would have different ECE and overconfidence behavior. They proporsed a baseline mixture of pretrain and finetune to reduce ECE for finetuning, which was successful. It was not clear why this is an important discovery or innovation that significantly benefits the ICLR community. ", "strength_and_weaknesses": "Strength: the paper studies ECE in language models which is somewhat of an overlooked area.\n\nWeaknesses: \n1, The paper did not provide sufficient motivation for why the broader community should care about a metric like ECE. They have a few references and some vague description of \"disastrous consequences in safety critical applications\". But they did not concretely describe such applications and such consequences or demonstrate that this is the most important issue of language models when applied in such applications. Thus this reviewer is not convinced that they are solving a problem of high enough priority.\n\n2, The paper did not go deep into the reason for why finetuning causes over-confidence in language models. They observed this phenomenon and proposed the most basic solution, which is mixing finetuning with pretraining. The result is also non-surprising. When the finetuning is mixed with the pretraining, then the resulting model behaves a bit more similar to the pretrained model. One might imagine that the overconfidence and the regression of the finetuned model as demonstrated in Table 1 is a result of the lack of diversity in the finetuning dataset. It would be interesting to establish the connection between the diversity of the training dataset and the ECE/overconfidence issue empirically, which would be a meaningful contribution to the community in understanding the response of deep networks to different data distributions.  \n\n3, The discussion about parameter efficient finetunings are interesting. However if one wants to make these model still do MLM, one can just remove the trained small parameter components and keep the pretrained model. The motivation for testing a parameter efficient model that is finetuned on specific downstream task on MLM is not very clear.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "1hzM_OSxbJ6", "reviewer": "Reviewer_eTyr", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work observes that while a pretrained language model is well-calibrated on the pretraining task as measured by ECE, it becomes poorly calibrated on downstream tasks after finetuning. It is further observed that efficient finetuning methods, such as Adapter, LoRA, and Prefix Tuning, result in better calibrated models than full finetuning. This leads the authors to conclude that the source of the miscalibration is the distortion of features learned during pretraining.\n\nThe authors then proposed to include the pretraining objective during finetuning to preserve pretraining features. This improves model calibration as measured by ECE while largely retaining performance on SNLI/MNLI, QQP/TwitterPPDB, and SWAG/HellaSWAG.", "summary_of_the_review": "Great motivation and interesting observations. However, the presentation can be simplified by forgoing the DVAE formalism, which adds little to this work. This paper can be further strengthened by considering other baselines that preserve pretraining features.", "strength_and_weaknesses": "Model calibration is often overlooked in the context of large language models. This paper makes the intriguing observation that efficient finetuning methods yield better calibrated models than full finetuning, likely due to the former’s ability to preserve pretraining features.\n\nThis work leverages this observation to further improve calibration by retaining the pretraining objective during finetuning. However, the denoising variational auto-encoding interpretation seems unnecessary. The referenced paper, Im et al., 2017, is motivated by the ability to obtain a more expressive variational distribution by marginalizing over input corruptions (e.g., to obtain a mixture of Gaussians). This marginalization makes the usual ELBO intractable, which motivated their new DVAE training criterion.\n\nThis work, however, uses a deterministic mapping from x to z, i.e., the variational distribution is the Dirac Delta distribution. By forgoing the stochasticity in the latent space, this work neither shares the core motivation of nor benefits from the DVAE paper. It is perhaps okay to mention Denoising Autoencoder in this context, but it basically describes the MLM objective which is not a contribution of this paper.\n\nThe method proposed in this work is better understood by inspecting Eq. 6 and can be summarized as “performing MLM on the finetuning data in addition to the finetuning objective (plus an L2 regularization in the latent space).” It is also unclear what the role of the L2 regularization is since it is present in neither the usual pretraining nor the usual finetuning objective. Indeed, as shown in Table 8, successful values of beta are many orders of magnitude smaller than those of alpha.\n\nI would encourage the authors to reframe their method without the DVAE framework and draw from the established literature on catastrophic forgetting. Moreover, given the baseline success of efficient finetuning methods, it seems worthwhile to investigate how they can be made better calibrated for scenarios where calibration is much more important than a slight performance drop.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster