On the Calibration of Large Language Models and Alignment
Authors: Chiwei Zhu
Benfeng Xu
Quan Wang
Yongdong Zhang
Zhendong Mao
Venue: EMNLP 2023 Findings (2023)
Keywords: Large Language Models
Calibration

Abstract: As large language models attract increasing attention and find widespread application, concurrent challenges of reliability also arise at the same time. Confidence calibration, an effective analysis method for gauging the reliability of deep models, serves as a crucial tool for assessing and improving their reliability. However, such investigation has been comparatively underexplored. In this work, we conduct a systematic examination of the calibration of aligned language models throughout the entire construction process, including pretraining and alignment training. At each stage, we investigate how different training settings, such as parameter scales and training data, affect model calibration. To thoroughly assess model calibration, we evaluate models on three most concerned aspects: generation, factuality and understanding. Our work sheds light on whether popular LLMs are well-calibrated and how the training process influences model calibration.

On the Calibration of Large Language Models and Alignment
Chiwei Zhu', Benfeng Xu'; Quan Wang', Yongdong Zhang', Zhendong Mao
University of Science and Technology China, Hefei, China MOE Key Laboratory of Trustworthy Distributed Computing and Service Beijing University of Posts and Telecommunications, Beijing, China tanz@mail. ustc. edu. cn, benfeng@mail wstc.uttcoo edu. wangquanébupt. edu. cn, zhyd73@ustc. edu. en, zdmao@ustc. edu. cn
Abstract
Pretraining
Language Modeling
Model Parameters
Training Dynamics
As large language models attract increasing attention and find widespread application, con- current challenges of reliability also arise at the same time. Confidence calibration, an ef- fective analysis method for gauging the relia- bility of deep models, serves as a crucial tool for assessing and improving their reliability. However, such investigation has been compara- tively underexplored. In this work, we conduct a systematic examination of the calibration of aligned language models throughout the entire construction process, including pretraining and alignment training. At each stage, inves- tigate how different training settings, such as parameter scales and training data, affect model calibration. To thoroughly assess model cali- bration, we evaluate models on three most con- cerned aspects: generation, factuality and un- derstanding. Our work sheds light on whether popular LLMs are well-calibrated and how the training process influences model calibration.
Factuality
Alignment
Training
Training Algonithms
Training Dynamics
Language Understanding
Figure I: Scope of investigations in this paper.
imperative ensure their reliability and faithful- ness, particularly in fields such healthcare (Kung et al., 2023) and law (Huang et al., 2023). These are domains where inaccurate predictions can lead to significant, potentially severe challenges. However, due to the intrinsic autoregressive mechanism and complex system structures, the behaviours of these models can not be easily attributed or interpreted.
Confidence calibration is an effective method to estimate model'ss 's awareness of its uncertainty, and helps enhance our understanding and assurance of the trustworthiness of deep models. Generally, associates model output confidence, i.e. probabil- ity, with ground truth correctness likelihood (Guo et al., 2017) and informs the user to what extent the outputs should be trusted, even though they may not always be correct. Intuitively, for ex- ample, given 100 predictions in classification task which produced by classifier and each of them is assigned 0.8 confidence, we expect 80 of them to be correctly classified for a well-calibrated classifier. As a consequence, better calibration of LLMs could significantly extend their usability. early meteorology, calibration was noted as valid- ity (Miller, 1962) reliability (Murphy, 1973), indicating the trustworthiness of forecasters. Well calibrated probabilitics can provide extra informa- tion for users decide whether trust the model's prediction, particularly for modern neural networks whose decisions are harder to interpret (Guo et al., 2017). Studies have also pointed out that calibra- tion is helpful to reduce hallucination in language
1 Introduction
Large Language Models (LLMs) like GPT- 3 (Brown et al., 2020), PaLM (Chowdhery et al., 2022), and GPT4 (OpenAl, 2023), fol- lowed by many open-source replications including LLaMA (Touvron et al., 2023a), Pythia (Biderman al., 2023) are revolutionizing the paradigm and re-shaping the expectation of modern natural lan- guage processing. When further trained with align- ment treatment (Ouyang et al., 2022; Bai et al., 2022), these LLMs further exhibit impressive ca- pability in responding to generalized human in- structions, which implies their potential as general- purpose intelligent assistants and this has since at- tract considerable attention in the field and around the world.
As LLMs find more diverse applications and ex- ert widespread influence, it becomes increasingly
"Coreesooning author: Benfeng Xu.

---

models (Xiao and Wang, 2021; Tian et al., 2020). Previous works have shown that pre-trained lan- guage models can generate well-calibrated predic- tions (Desai and Durrett, 2020; Kadavath et al., 2022). However, these works mainly concentrate on vanilla language models, while the aligned lan- guage models receive less focus. A newly proposed work evaluates calibration of some aligned models by prompting them to verbalize confidence in the token space (Tian al., 2023), but it mainly studies black-box models, whose training process is not available, and thus can not provide insight into how model calibration is affected by different factors in the alignment training process. To conclude, a systematical study on the calibration of aligned lan- guage models is still missing, and our work aims to fill this gap.
arrive the following findings.
For pretraining of LLMs:
Larger Parameter Scales 1 Improve mod- els' calibration.
Longer Training Dynamics 1 Also benefit calibration accuracy.
For alignment of LLMs:
Instruction Tuning +: Deteriorates models' calibration.
Synthetic Data + Exacerbates the harmful effect of instruction tuning.
Paraieter-efficient Fine-tuning > Effec- tive regularization for restraining calibration error.
In this work, we study the calibration of aligned language models in the entire building cycle and provide evidence on how to achieve decent model calibration. An overview of the scheme of out study is at Figure 1. Following the training process of aligned language models, we study model calibra- tion in pre-training stage and alignment training stage respectively. In each stage, we reveal how model calibration changes when using different training settings. For pre-training stage, we ex- amine the effect of parameter scale and training dynamics (steps). For alignment training stage, we study the effect of instruction tuming and RLHF, in which instruction tuning is further scrutinized by changing instruction datasets, training methods and also training dynamies.
RLHF > Help maintaining calibration ac- curacy.
For different tasks:
In pre-training: Improvement in calibration accuracy is more significant on fact genera- tion task or language understanding tasks than language modeling task.
In alignment training: Calibration accuracy evolves consistently across different down- stream tasks including fact generation, lan- guage understanding vanilla language mod- eling.
Besides the understanding and generating abil- ity, factual faithfulness and reasoning capability are two widely considered issues with large lan- guage models (Du et al., 2023). We also follow this path to study models" calibration when applied to different tasks. For this purpose, we design three tasks for each of the stages above. (1) To evalu- ate model calibration on common text generation, we use Causal Language Modeling (CLM) task, which also the objective of pre-training stage. (2) To study model calibration on factuality, we de- signed a facts generation task where the models are asked to generate fact-related content. (3) To study model calibration on reasoning, we use multi-task language understanding task, where questions and possible options are provided and models asked select the most probable one.
We believe these conclusions well detailed experiments can take us a step further towards un- derstanding large language models, especially the intrinsic mechanism of their calibration behaviour. Our experimental results also provide us with some possible solutions to improve calibration, includ- ing increasing model scales and employing param- eter efficient tuning methods. Besides, diversity guided instruction data construction may also be very promising. Hopefully these findings can shed light on future works to construct more factual and trustworthy assistants.
Related Work
Aligned Large Language Models are large lan- guage models that are specially trained to follow human's 's intents or instructions. Large language
Through extensive experiments and analysis, we

---

models are proved have the ability of complet- ing some downstream tasks without any gradient updating (Brown et al., 2020). To better make use such ability, many researches have found that instruction following models can be constructed by fine-tuning models with instruction- instruction-responee pairs, which is called instruction tuning (Weller et al., 2020; Mishra al., 2022; Sanh et al., 2022; Wei al., 2022a; Xu al., 2023b). While these models can understand human instructions and make rea- sonable responses, they often produce unexpected results like lies, madc-up facts, biased or toxic texts and so on. To better align models with human in- tents, reinforcement learning with human feedback is introduced to the training of large language mod- els (Ouyang et al., 2022). Though instruction tun- ing and RLHF significantly improve the mod- els' ability of interacting with humans, how they influence the calibration of large language models have not been researched on.
1.0
Pythia-70M ECE=0.107
perfect calibration
0.8
accuracy
0.4
0.2
0.0
0.2
0.4
0.6
0.8
1.0
confidence
Figure 2: Reliability diagram for Pythia-70m model.
lic in.
Analysis large language models. Understand- ing various aspects of LLMs through theoretical or empirical approaches have long been an impor- tant interests for NLP scholars. Many works have demonstrated the scaling law of LLMs in different model scales, data size and com- putational costs (Kaplan al., 2020; Rae et al., 2022; Xia et al., 2023). Wei et al. (2022b) defines and reveals the emergent abilities of large language models. Liang et al. (2022) proposes holistic eval- uation framework named HELM to analyze large language models on their capabilities, limitations, and potential risks. Hallucination and factuality also draw lot of attention (MeKenna et al., 2023; Zheng al., 2023), but they do not take further step towards the intrinsic mechanism while merely explore the verification on the surface. Differently, this paper provides a formal and systematical anal- ysis on calibration behaviour of LLMs and their alignment treatment.
Confidence calibration is concerned prob- lem for classification models. large amount of works have studied the calibration of statisti- cal machine learning systems and the methods to improve their calibration (DeGroot and Fienberg, 1983; Palmer et al., 2008; Yang and Thompson, 2010). Later, calibration of neural networks have also been researched on (Hendrycks and Gimpel, 2016; Nguyen and O' O'Connor 2015; Nixon et al.,
scenarios
2019; Minderer et al., 2021). Guo al. (2017) points out that modern neural networks are as calibrated as their ancestors and proposes temper- ature scaling methods to calibrate neural networks. natural language processing field, calibration of cansformer-based language models are evaluated among different tasks, including machine trans- lation (Kumar and Sarawagi, 2019), QA (Jiang al., 2021) and selective prediction (Varshney al., 2022). Recently, large-scale generative lan- guage models are receiving growing attention, and some works have examined calibration of these models (Srivastava et al., 2022; Kuhn et al., 2023; Tian et al., 2023). There are also works improving calibration of large language models, for example, Xu et al. (2023a) propose &NN Prompting to effec- tively mitigate calibration emrors in in-context learn- ing. However, as mentioned before, these works either concentrate on vanilla language models or study black-box models. We study models calibra- tion their whole life cycles from pre- pre-training to alignment training, where our main contributions
3 Definitions
In this section we formally define some basic con- cepts in our work, including confidence calibration. reliability diagram and expected calibration error. Confidence calibration is the main objective we are studying this work and the other two are tools we use to evaluate model calibration. Our selection of tools follows previous work (Guo et al., 2017).
Confidence Calibration. Given a supervised multi-class classification scenario, where we have input x, label E y {1, 2...., model predic- tion y' E y and confidence p' E [0, 1]. model is

---

perfectly calibrated, if we can get
metric which can reflect overall calibration level of model. Expected Calibration Error is such a quantitative measurement of calibration (Naeini et al., 2015).
P(y' = ulp'= =)
Vp E [0,
for any input (Guo et al., 2017). In another word, the more confident a model is, the chance of its prediction being the same as ground truth should be higher. It should noted that P(y/ = y/p' - p) can not be calculated with finite number of samples, so calibration is often evaluated by some statisitcal approximations.
For a set of N samples, we also divide confi- dence intervals into M bins and get Acc (m) and Con (Bm) in the same way as we did when draw- ing reliability diagram. Then ECE is calculated as follows:
Bml |Aco( (Ac((m) fon (Bm)
ECE Σ
Reliability Diagram is kind of visualized eval- uation of confidence calibration (DeGroot and Fien- berg. 1983), which plots prediction accuracy as a function of confidence (c.g. Figure 2).
ECE represents confidence error averaged on samples and obviously lower ECE means better calibration. We set 10 when measuring cal- ibration with the tools above following previous works (Guo al., 2017: Desai and Durrett, 2020; He al., 2023).
To evaluate the calibration with a model with a finite set of samples, we divide confidence in- terval [0, 1] into M bins with equal length (1 /M) and group model predictions into these bins ac- cording to their prediction confidence. Let Bm be the set of indices of samples which fall into the interval ( (m-1 mi M M then for each interval bin, we can calculate corresponding accuracy and average confidence as follows:
Calibration Evaluation Tasks and Data
As mentioned before, we evaluate model calibra- tion three tasks considering language models' ability of understanding, factuality and reasoning. In this section, we in detail introduce for each task how the evaluation is conducted and the datasets chosen for evaluation.
Acc( Bm))
Σ ic
1 = mi),
|mml
Causal Language Modeling is the task of pre- dicting the next token for given sequence, which is also the pre-training objective of causal language models. For test sequence, we randomly sample position in the sequence. Then this sequence is fed into models to generate a token corresonding to the position. If the predicted token is the same as the one in original sentence, we count as a true positive. In such way we can get generation accuracy and confidence of test dataset, and then evaluate model calibration with reliability diagram and ECE metric. We use development and test set of the PILE dataset (Gao et al., 2020) in this task. The PILE dataset large-scale English text cor- pus which frequetly used in the pre-training of large language models.
Conf(BB.)
Σ p.:
Bm
i.
where ÿ and y are the prediction class and ground truth of the it sample. 1 is the indicator function which produces if y yi otherwise O. Ps is the prediction confidence (probability) of the ith sample.
Given Acc(Bnn) and ff we can draw the reliability diagram for a model. For per- fectly calibrated model, we will have Acc( Bm.) Con for all m. so its reliability diagram will be y Obviously, the nearer a curve is the di- agonal, the better calibration it represents. Though perfect calibration impossible, we normally hope a model well calibrated. Note that since reliabil- ity diagram do not contain the number of samples, sometimes it could be insufficient to represent true calibration of a model when some bins only contain very few samples.
Facts Generation is the task aimed at evaluating models' memory on factual knowledge. The task is mostly the same causal language modeling in form, except that we utilize entity linking data, in which texts are labeled with entity spans. We let models generate the first token of entities. We only take the first token of entities into account as when the first token is correctly generated, there is high
Expected Calibration Error (ECE). As relia- bility diagram more like a qualitative evaluation which depicts model calibration in different confi- dence intervals, we hope to get quantitative scalar

---

1.0
(al Relabilty Biagram: Sca VILE
17h
1.0
(c) nol abbiityyy blegram: Scale T-txx
17h
1.0
(c) Relabiliy Diogram: Scale KNLL
17h
0.0
0.0
0.0
Parameter Scales
Partamttee Scates
0.6
AccU8000
0.6
0.66
10m
70m
70m
Confidence
Confidence
Curfidencer
[5) ECESRCC Curve: Scalc-FILE
ELE/ACL Curve: Scale-
(1) ECEAAC Curve: scale MHLL
DCE
DCE
DCE
0.4
0.4
0.4
0.2
0.2
0.2
08
70m
160m
430m
1b
1.4b
2400
6.90
320
0.0
70m
160m
410m
3,4b
2.80
6.90
12b
0.0
70m
300m
410m
Ub
1.40
2.80 6.90 12b
Figure 3: Model calibration of different parameter scales.
chance that the whole entity can also be recovered. facts generation task, we use an enitity linking dataset T-REx (Elsahar et al., 2018) 8), which includes entity-labeled texts extracted from Wikipedia pages. For T-REx and the PILE dataset, we randomly draw 100k samples as our evaluation set.
of the scale provides 154 checkpoints including 143 checkpoints saved every 1,000 training steps (i.e. epoch) and 11 checkpoints trained for less than 1,000 steps. All of these models are trained on exactly the same data-the PILE (Gao et al., 2020) dataset in the same order. For parameter scale study, experiment models with all 8 scales. As for training dynamics, we choose Pythia- 1B4 considering time and computational cost, and use 2" 11, 000 2.... steps checkpoints (up to 4tep143,000 for our study. We also include check- points of step256 and step512 in our experiments to observe the behavior of ander-fitted models.
Multi-task Language Understanding is task where models are given questions across different fields with multiple answer options, which de- signed for testing the understanding and reasoning ability of a language model. We mainly focus on questions with a single correct answer. Follow- ing MMLU benchmark (Hendrycks et al., 2021), we concatenate 5 in-context samples ahead of the questions and designed prompts to constrain mod- els to respond with answer options (i.e. AACDD", We choose MMLU benchmark (Hendrycks et al., 2021) as our evaluation data, which covers single- choice questions in 57 subjects across STEM, the humanities, the social sciences and so on.
5.2 Parameter Scales
Figure 3 shows the experimental results of parame- ter scales on three tasks. Generally, larger models produce better calibrated results while the level of such effect is diverse among tasks. We find that models with all parameter scales can produce well-calibrated predictions on CLM task, with ECE lower than 0.1. Also, parameter scales only mildly affect model calibration on the CLM task, where difference is minor between smallest and largest model. This might because CLM task is the same as the pre-training objective, where large scale and diverse corpus makes it hard for models to be over- confident when generating common texts. On facts generation task, model performance on both cali- bration and accuracy shows a stronger positive cor- relation with parameter scales. Results on MMLUU (Figure 3-c and 3-f) seems very messy, but we can still observe some meaningful patterns here. All
5 Calibration in Pre-training Stage
In this section, we study the effect of parameter scales and training dynamics in pre-training stage to models' calibration.
5.1 Experimental Setups
We choose Pythia as our base model (Biderman al., 2023). Pythia is suite of transformer-based. reto-segressive language models designed for sci- entific research. It contains models whose scales range from 70m to 12B parameters and for each

---

1.0
iollaaalty tiggaan Thiiiing bynam
143
1.0
Traiiing Dynemics TREE
1436
(e) lorieaalty tle
Dynomics
143K
0.6
0.4
Confidencee
Conficincee
Confidence
IB3 ELEJACC Lurree Training Dynam
Hd] ECEAADU Curwe Taining DyneMics-TRE:
(1) ECE Carver
Traniing Dynomics
0.5
ECE
ECE
0.5
ECE
ACC
ACC
ACC
0.3
0.3
0.3
0.1
68 512
Or 32k 1286-433
256
512
6k
32k 1288:433
0.9 256 512
88 32k one 1298143
Figure 4: Model calibration of different training dynamics.
models perform poorly on language understand- ing task, with accuracies only slightly better than random choice. As Pythia models have not been trained to follow certain instructions, it is hard for demanding
pointed out (See Figure 4). On CLM task, we can observe apparent improvement in both accu- racy and calibration in the very early stage of pre- training. However, though accuracy keeps growing as the training goes on, ECE stabilize at a low level for the rest of the training process, which means that fntte-fitted models also be well- calibrated. Training dynamics also show stronger impact on facts generation task. It can be seen that models trained for less than epoch behave extremely poor. In this stage, models are near to randomly initialized parameters which can not gen- erate reasonable probability distribution, thus the accuracy is almost zero for all confidence intervals. Results on MMLU dataset is similar that of pa- rameter scales, with accuracy barely grows while calibration keep improving. Note that we observe an increase of ECE step143, 43,000 (see Figure 4-f), which may be an sign of over- weer-fttiing. As Pythia does not provide checkpoints with further steps, we keep this as simple hypothesis.
them to understand these questions in the MMLU dataset. However, while accuracies from all models creases monotonically as the model scale increases, Moreover, we found that as the parameter scale increases, confidence distribution of model output gradually shrinks to a smaller and lower interval (see Appendix B), which might indicate that al- though larger models still can not solve these prob- lems, they are more aware of their own capability than small ones. To further verify our conclusions, we conduct same experiments LLaMA (Touvron al., 2023a), LLaMA-2 (Tou- vron et al., 2023b), FLAN-TS (Chung et al., 2022) and OPT (Zhang et al., 2022). Results are presented in Appendix C. We can see that our conclusions holds most of the time. where LLaMA2, FLAN-T5 and OPT show monotonic improvement with in- creasing model scale while there are outliers in the results of LLaMA. This demonstrates that factors other than scale may influence the trend, and LLMs should be examined using more continuously sam- pled checkpoints to reach more robust conclusion, which we leave for future works.
similar, ECE de-
more models,
Calibration in Alignment Stage
Alignment training is divided into two sub-stages, instruction tuning and RLHF. In this section we study how model calibration changes during these process. As the models are fine-tuned instruc- tions in this stage, add instruction prompt for all three tasks (see Table 1). Figure shows the ECE level of fine-tuned models on three tasks when using different aligmment training settings. We also report accuracy results in Appendix C.2, where the
5.3 Training Dynamics
On the whole, the effect of training dynamics fol- lows the same pattern with that of parameter scales but there are a few unique observations can be

---

PILE
T-REx
MMLU
0.30
0.30
0.30
0.25
0.25
0.25
0.20
0 0.15
0.15
0.15
0.10
0.10
0.10
0:05
0.05
0.05
Alpaca
Alpaca
Alpaca
Alpocn
Epech-1
Rpucle2
Epoch-3
Figure 5: Model calibration using different alignment training settings.
Tasks
Prompts
Causal Language Modeling
Finish this sentence:
Facts Generation
Finish this Wikipedia description:
Multi-task Language Understandign
The following multiple choice questions (with answers) about (...].
Table I: Prompts of different tasks Alignment Training Stage
accuracy performance generally remains stable and only fluctuates with different datasets and training methods.
the same instructions are collected and ranked by human annotators another language model. We open- LM-ranked responses data (Peng et al., 2023), where GPT-4, GPT-3.5 and OPT-IML generate responses to Alpaca instructions and these responses are ranked by GPT-4. We use LLaMA our reward model and conduct PPO training on top of previously instruction tuned model with Alpea data. We also use LoRA in RLHF training process to lower computational costs. We perform RLHF training with Huggingface TRL Library'.
6.1 Experimental Setups
In aligmment training stage, use LLaMA-7B our base model (Touvron al., 2023a). Com- plete training hyper-parameters can be found in Appendix
Instruction Tuning, We leverage open-source GPT-generated and human-labeled data in instruc- tion tuning. Alpaca (Taori et al., 2023) contains 52k pairs of instructions and responses generated in the style of self-instruct (Wang et al., 2022) us- ing OpenAI GPT model. OpenAssistant Conversa- tions', which we denote as OA, is numan-llbeled assistant-style conversation corpus. We extract all single-turn conversations written in English from OA, resulting in 1k pairs of instructions. Consid- ering fairness, we sample the datasets to the same size when comparing effect of instruction tuning. We follow setups of Stanford-Alpact and Alpaca- LoRA3 for direct fine-tuning and LoRA training respectively. For each experiment group, we train the model for epochs.
6.2 Instruction Tuning
We find that instruction tuning generally weakens the calibration of language models. Such impact changes when using different instruction tuning settings.
Training Data. As can be seen in Figure direct fine-tuning with Alpaca does the most harm to cal- ibration while models fine-tuned with OA dataset perform better. Note that in MMLU dataset, the model trained with OA is even better calibrated than LLaMA in the first epoch, which might be- cause of the capability gain in following instruc- tions. However, the ECE level rapidly becomes worse later epochs, which represents that degen- eration of calibration is a result fitting process and OA is less likely to cause such degeneration. We presume such behavior is related to the diver- sity of datasets. Such presumption is mostly based on intuition where samples in OA datasets display strong personal emotional characteristics while
RLHF training contains two parts, reward model training and reinforcement learning. Training of reward models often needs ranked response data, where responses from different language models to
co/datsstss///////
"https://githie / wwwptttwo/ccccctc..oooooooo.ottt comm///oee/alaaaa-lorr
"https: co/ co/docs/trr//nee

---

instructions in Alpaca are much more homoge- neous. To further look into the difference of Alpaca and OA, we compare semantic diversity of their re- sponses. We use MPNET (Song et al., 2020) to ex- tract sentence features of responses both datasets and visualize these features with t-SNE (Van der Maaten and Hinton, 2008), see Figure 6. Results show that semantic features of OA dataset are more evenly distributed while those of Alpaca tend to be dense and clustered, which means the former is more diverse in semantics. We attribute this difference of diversity to how the datasets are con- structed. Alpaca is a synthetic corpus generated in a self-instruct way, where a small set of human- written instruction data containing 175 seed tasks are fed into GPT-3 and augmented to 52k scale. In this case, Alpaca will contain a lot of instruction data with similar format and content for each seed task, and fine-tuning with such clustered dataset consequently leads to a worse calibration. On the other hand, OA created by crowd-sourcing where thousands of volunteers are asked to submit their own instruction data, which makes the dataset more diverse in tasks and text styles, thus do less harm to model calibration.
100
200 300
100
Figure 6: Sentence feature distributions of Alpaca and OA dataset (Each point is response sentence).
Training Dynamics. almost all instruction tun- ing experimental groups, models trained for more steps behave worse in calibration, which indicates that models calibration are affected severely by the small instruction dataset. Note that obtained contradictory observation (model calibration im- proves when trained longer) in the pre-training stage where the model is trained with large-scale and diverse corpora, thus we anticipate that im- prove the scale and diversity instruction data can also help improve calibration.
6.3 RLHF
The last groups of bars in each chart of Figure 5 show ECE level of models trained with RLHF for three epochs. We can see that compared to the instruction-tuned model, i.e. the 3rd epoch model trained with Alpaca, there is no significant degen- eration in ECE after RLHF training. Moreover, models' calibration do not deteriorate RLHF last for more epochs. This indicates that when applied to models that already have been trained with in- struction data, RLHF might not do further harm model calibration.
Training Methods. Parameter efficient tuning is type of training methods that keep the pre-trained weight unchanged and only train small set of extra parameters. Although these methods are originally designed train large models with lower resources, they may be able to improve calibration by reduc- ing catastrophic forgetting (He et al., 2023). We compare calibration of models trained on Alpaca and OA datasets with full fine-tuning and LoRA tuning. Results exhibit that in all three tasks, model trained with LoRA performs better in calibration than those directly fine-tuned. Besides, it can be noticed that deterioration in calibration is slight for model trained with LoRA (often in level of 0.001) when training epochs increase, while the full fine-tuned model becomes visibly worse. Such observations proves that LoRA can mitigate the calibration degeneration in the instruction tuning process. In MMLU dataset we observe that behav- iors of models trained Alpaca with LoRA similar those trained with OA, where calibration improves compared to LLaMA in the first epoch. This may also indicate that LoRA helpful in re- duce the harmful effect of instruction tuning and improve model calibration.
7 Discussion
Confidence calibration helpful for building hon- est large language models in two ways. Firstly, confidence calibration is closely related to the un- certainty of language models, which is leveraged in many approaches like self-consistency to improve model performance (Wang al., 2023). Studying and improving model calibration will provide fur- ther evidenee to these methods and inspire more uncertainty- techniques.
8 Conclusions
In this work we systematically study the calibration of aligned large language models. We designed

---

thorough experiments evaluating the model cali- bration with different training settings and reveal how model calibration affected by pre-training and alignment training process. In pre-traie pre-training, we find that model calibration improves as parameter scales and training dynamics increases. In align- ment training stage, experimental results show that instruction tuning damages model calibration sig- nificantly and ill-distributed synthetic data does more harm. Such harm will increase when fine- tuning process lasts longer while can be remediated by using parameter efficient training methods like LoRA. In the mean time, we surprisingly find that RLHF has little impact on calibration of instruction tuned models.
Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. 2023. Pythia: A suite for analyzing large language models across training and scaling.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam. Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 13:1877-1901.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sehastian Gehrmann, et al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:204.003/11.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Al- bert Webson, Shixiang Shane Gu, Zhuyun Dai. Mirac Suzgun, Xinyun Chen, Aakanksha Chowdh- ery. Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Ja- cob Devlin, Adam Roberts, Denny Zhou, Quoe Le, and Jason Wei. 2022. Scaling instruction-finetuue language models.
Limitations
There are two main limitations in this work. The first one is that we can not carry out fine-grained experiments on larger and better-performing mod- els like LLaMA and LLaMA-2, they only pro- vide limited number of variants scale (e.g. 7B/30B/70B for LLaMA-2) and do not provide checkpoints for different training dynamics. More detailed and rigorous conclusions can be drawn finer-grained model variants are available. The sec- ond one is that our observations and conclusions can be further explored, like mining the relation of these observations to the mathematical theory of confidence calibration and proving our conclusions theoretically. We leave such in-depth exploration for future works.
Morris H DeGroot and Stephen Fienberg. 1983. The comparison and evaluation of forccasters. Journal of the Royal Statistical Society: Series (The Statisti-
cian), (2(1
Shrey Desai and Greg Durrett. 2020. Calibration of pre-trained transformers.
Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. 2023. Improving factuality and reasoning in language models through multiagent debate.
Acknowledgements
We would like to thank all the reviewers sincerely for their valuable advice to improve this work. This research is supported by National Science Fund for Excellent Young Scholars under Grant 62222212 12 and the General Program of National Natural Sci- ence Foundation of China under Grant 62376033.
Hady Elsahar. Pavlos Vougiouklis, Arslen Remaci. Christophe Gravier. Jonathon Hare, Frederique Lafor- est, and Elena Simperl. 2018. T-REx: A large scale alignment of natural language with knowledge base triples. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki. Japan. European Language Resources Association (ELRA).
References
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. 2022. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint 055622...
Leo Gao, Stella Biderman, Sid Black, Laurence Gold- ing, Travis Hoppe, Charles Foster, Jason Phang. Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy, 2020. The pile: An 800gb dataset of diverse text for language modeling.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Wein- berger. 2017. On calibration of modern neural net- works. In International conference on machine learn- ing, pages 1321-1330. PMLR.
Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle Brien, Eric Hallahan, Mo- hammad Aflah Khan, Shivanshu Purohit, USVSN Sai

---

Guande He, Jianfei Chen, and Jun Zhu. 2023. Pre- serving pre-traine trained features helps calibrate fine-tuned language models. In The Eleventh International Con- ference on Learning Representations.
Hudson, Erie Zelikman, Esin Durmus, Faisal Lad- hak, Frieda Rong, Hongyu Ren, Huaxiu Yao, Jue Wang, Keshav Santhanam, Laurel Orr, Lucia Zheng, Mert Yuksekgonul, Mirac Suzgun, Nathan Kim, Neel Guha, Niladri Chatterji. Omar Khattab, Peter Henderson, Qian Huang, Ryan Chi, Sang Michael Xie, Shihani Sanurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas Ieard. Tianyi Zhang, Vishrav Chaudhary, William Wang. Xuechen Li. Yifan Mai, Yuhui Zhang, and Yuta Koreeda. 2022. Holistic eval- uation of language models.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language under- standing.
Dan Hendrycks and Kevin Gimpel. 2016. A baseline for detecting misclassified and out-of-distribution examples in neural networks. arXiv preprint
Nick McKenna, Tianyi Li, Liang Cheng, Moham- mad Javad Hosseini, Mark Johnson, and Mark Steed- man. 2023. Sources of hallucination by large lan- guage models on inference tasks.
160.v:1610.22136
Quzhe Huang, Mingxu Tao, Zhenwei An, Chen Zhang, Cong Jiang, Zhibin Chen, Zirui Wu, and Yansong Feng. 2023. Lawyer llama technical report.
Robert G Miller. 1962. Statistical prediction by discrim- inant analysis. In Statistical prediction by discrimi- nant analysis, pages 1-54. Springer.
Zhengbao Jiang, Jun Araki, Haibo Ding, and Graham Neubig. 2021. How can we know when language models know? on the calibration language models for question answering. Transactions of the Associa- tion Computational Linguistics, 9:962-9777
Matthias Minderer, Josip Djolonga, Rob Romijnders. Frances Hubis, Xiaohua Zhai, Neil Houlsby, Dustin Tran, and Mario Lucic. 2021. Revisiting the calibra- tion of modern neural networks. Advances in Neural Information Processing Systems, 34:15682-1556
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez. Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones. Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny Hernandez, Josh Jacobson, Jackson Kernion, Shauna Kravec. Liane Lovitt, Ka- mal Ndousse, Catherine Olsson, Sam Ringer, Dario Amodei, Tom Brown, Jack Clark, Nicholas Joseph, Ben Mann. Sam McCandlish, Chris Olah, and Jared Kaplan. 2022. Language models (mostly) know what they know.
S Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannanch Hajishirzi. 2022. Cross-task generaliza- tion via natural language crowdsourcing instructions.
Allan H Murphy, 1973. A new vector partition of the probability score. Journal of Applied Meteorology and Climatology, 12())5955-600.
Mahdi Pakdaman Nacini, Gregory Cooper, and Milos Hauskrecht. 2015. Obtaining well calibrated proba- bilities using bayesian binning. In Proceedings of the AAAI conference on artificial intelligence, volume 29,
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models.
Khanh Nguyen and Brendan O' O'Connor. 2015, Pos- terior calibration and exploratory analysis for nat- ural language processing models. arXiv preprint arXiv: 3oXiv:5508.055:5
Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. 2023. Semantic uncertainty: Linguistic invariances for un- certainty estimation in natural language generation.
Jeremy Nixon, Michael W Dusenberry, Linchuan Zhang, Ghassen Jerfel, and Dustin Tran. 2019. Mcasuring calibration in deep learning. In CVPR workshops, volume 2.
Aviral Kumar and Sunita Sarawagi. 2019. Calibration of encoder decoder models for neural machine trans- lation.
OpenAl. 2023. Gpt-4 technical report.
Tiffany H Kung, Morgan Cheatham, Arielle Medenilla, Czarina Sillos, Lorie De Leon, Camille Elepaño, Maria Madriaga. Rimel Aggabao, Giezel Diaz- Candido, James Maningo, et al. 2023. Performance of chatgpt on usmle: Potential for ai-assisted medical education using large language models. PLoS digital health, 22):c00011
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Car- roll L. Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray. John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller. Maddie Simens, Amanda Askell, Peter Welinder. Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback.
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Ku- mar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang. Christian Cosgrove, Christopher D. Man- ning, Christopher Ré, Diana Acosta-Navas, Drew A.
TN Palmer, FJ Reyes, Antje Weisheimer, and MJ Rodwell. 2008. Toward seamless prediction: Cal- ibration of climate change projections using seasonal forecasts. Bulletin of the American Meteorological Society, 89(4):459-470.

---

Baolin Peng, Chunyuan Li. Pengcheng He. Michel Gal- ley, and Jianfeng Gao. 2023. Instruction tuning with gpt-4, arXiv preprint arXiv:2304.03277.
An destruction-oolloiing llama model. https:// gi thub. wwt..ccc//aaaaaeaeeeeeee
Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher D. Manning. 2023. Just ask for cali- bration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback.
Jack W. Rac. Sebastian Borgcaud, Trevor Cai, Katic Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susan- nah Young, Eliza Rutherford, Tom Hennigan, Ja- cob Menick, Albin Cassirer, Richard Powell, George den Driessche, Lisa Anne Hendricks, Mari- beth Rauh, Po-Sen Huang, Amelia Glaese, Jo- hannes Welbl, Sumanth Dathathri, Saffron Huang. Jonathan Uesato, John Mellor, Irina Higgins, Anto- nia Creswell, Nat McAleese, Amy Wu, Erich Elsen, Siddhant Jayakumar, Elena Buchatskaya, David Bud- den, Esme Sutherland, Karen Simonyan, Michela Pa- ganini, Laurent Sifre, Lena Martens, Xiang Lorraine Li, Adhiguna Kuncoro, Aida Nematzadeh, Elena Gribovskaya, Domenic Donato, Angeliki Lazaridou, Arthur Mensch, Jean- can-taptiste Lespiau, Maria Tsim- poukelli, Nikolai Grigorev, Doug Fritz, Thibault Sot- tiaux. Mantas Pajarskas, Toby Pohlen, Zhitao Gong, Daniel Toyama, Cyprien de Masson d' Autume, Yujia Li, Tayfun Terzi, Vladimir Mikulik, Igor Babuschkin, Aidan Clark, Diego de Las Casas, Aurelia Guy, Chris Jones, James Bradbury, Matthew Johnson, Blake Hechtman. Laura Weidinger, lason Gabriel, William Isaac, Ed Lockhart, Simon Osindero, Laura Rimell, Chris Dyer, Oriol Vinyals, Kareem Ayoub, Jeff Stanway, Lorrayne Bennett, Demis Hassabis, Ko- ray Kavukeuoglu, and Geoffrey Irving. 2022. Scaling language models: Methods, analysis & insights from training gopher.
Ran Tian, Shashi Narayan, Thibault Sellam, and Ankur P. Parikh. 2020. Sticking to the facts; Confi- dent decoding for faithful data-to-text generation.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet. Marie-Anne Lachaux. Timothée Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023a. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Al- bert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami. Naman Goyal, An- thony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura. Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Di- ana Liskovich, Yinghai Lu, Yuning Mao, Xavier Mar- tinet, Todor Mihaylov, Pushkar Mishra, Igor Moly- bog. Yixin Nie, Andrew Poulton, Jeremy Reizen- stein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subrama- nian, Xiaoqing Ellen Tan, Binh Tang. Ross Tay- lor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan. Iliyan Zarov, Yuchen Zhang, Angela Fan. Melanie Kambadur, Sharan Narang, Aurelien Ro- driguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023b. Llama 2: Open foundation and fine-tuned chat models.
Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, 7aid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, Manan Dey, Saiful Bari. Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szezechla, Taewoon Kim. Gunjan Chhablani, Nihal Nayak. De- bajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang. Trishala Neeraj. Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Tali Bers, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M. Rush. 2022. Multi- task prompted training enables zero-shot task gener- alization.
Laurens Van der Maaten and Geoffrey Hinton. 2008
Visualizing data using t-sne. Journal of machine learning research, 9(1).
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie- Yan Liu. 2020. Mpnet: Masked and permuted pre- training for language understanding.
Neeraj Varshney, Swaroop Mishra, and Chitta Baral. 2022. Investigating selective prediction approaches across several tasks in iid, ood, and adversarial set- tings.
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, and Adrià Garriga-Alone Alonso et al. 2022. Beyond the imita- tion game: Quantifying and extrapolating the capa- bilities of language models.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le. Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-consistency improves chain of thought reasoning in language models.
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannanch Hajishirzi. 2022. Self-instruct: Aligning language model with self generated instructions.
Rohan Taori, Ishaan Guirajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca:

---

Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, An- drew M. Dai, and Quoc V. Le. 2022a. Finetuned language models zero-shot learners.
Jason Wei. Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunrri Hashimoto, Oriol Vinyals. Percy Liang. Jeff Dean, and William Fedus. 2022b. Emer- gent abilities of large language models.
Orion Weller, Nicholas Lourie. Matt Gardner, and Matthew E. Peters. 2020. Learning from task de- scriptions. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1361-1375, Online. Association for Computational Linguistics.
Mengzhou Xia. Mikel Artetxe, Chunting Zhou, Xi Vic- toria Lin, Ramakanth Pasunuru, Danqi Chen, Luke Zettlemoyer, and Ves Stoyanov. 2023. Training tra- jectories of language models across scales.
Yijun Xiao and William Yang Wang. 2021. On halluci- nation and predictive uncertainty in conditional lan- guage generation. arXiv preprint arXiv:2103. 15025.
Benfeng Xu, Quan Wang. Zhendong Mao, Yajuan Lyu, Qiaoqiao She, and Yongdong Zhang. 2023a. $KSNN prompting: Beyond- context learning with calibration- free nearest neighbor inference. The Eleventh In- termational Conference on Learning Representations.
Benfeng Xu, An Yang, Junyang Lin, Quan Wang, Chang Zhou, Yongdong Zhang, and Zhendong Mao. 2023b. Expertprompting: Instructing large language models to be distinguished experts.
Huiqin Yang and Carl Thompson. 2010. Nurses' risk assessment judgements: confidence calibration study. Journal of Advanced Nursing, 66(12):2751- 2760.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher De- wan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mi- haylov, Myle Ott, Sam Shleifer. Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2022. Opt: Open pre- trained transformer language models.
Shen Zheng, Jie Huang, and Kevin Chen-Chuan Chang 2023. Why does chatgpt fall short in providing truth- ful answers?

---

A Hyper-parameters
We list all used parameetrss in Table and Table 3.
Parameters
Values
Direct Fine-tune nums of gpu epochs
Parameters
Values
Reward Model
nums of gpu epochs
4
batch size per gpu gradient accumulation total batch size max sequence length learning rate
2
8
batch size per gpu gradient accumulation
8
4 8 = 128
2048
total batch size
* 8
32
2e-5
sequence length learning rate
2048
warmup ratio
0.03
2c-5
Ir scheduler
cosine
lora
8
LoRA
Iora alpha
32
nums of gpu epochs
4
lora dropout
0.1
Ir scheduler
cosine
batch size per gpu gradient accumulation total batch size max sequence length learning rate
16
RLHF
nums of gpu epochs
4
16
128
3
2048
batch size
8
3e-4
gradientaccumulation
8
warmup steps
100
output max length
128
Io
8
learning rate
1.4e-5
Iora alpha
32
lora
16
Iora dropout
0.1
Iora alpha
32
lora target modules
[q_proj, v_proj]
lora dropout
0.05
Ir scheduler
linear
Table Hyper-parametee RLH...
Table 2: Hyper-parameters of instruction tuning.

---

B MMLU Confidence Distribution
Figure shows the confidence distribution of out- puts of Pythia models from 70m to 12B. As is explained in Section 5.2, the ranges of the distri- butions tend to become smaller for larger models.

---

70m
160m
10000
10000
7500
7500
frequency
frequency
5000
5000
2500
2500
02
0.1
0.3
077
0.
confidence
confidence
410m
1b
7500
7500
frequency
frequency
6000
5000
2500
2500
0.0
0.2
0.3
0.4
0.6
0.8
1.0
0.1
0.3
0.5
0.7
0.9
confidence
confidence
1.4b
2.8b
10000
10000
7500
frequency
frequency
5000
5000
2500
2600
0.0
0.1
0.2
0.4
0.8
0.8
1.0
0.1
0.3
0.5
0.7
0.9
confidence
confidence
12b
7500
7500
frequency
frequency
2500
2500
00
02
0.4
06
08
1.0
0.1
03
05 confidence
07
09
confidence
Figure Confidence distribution of model outputs of different scales MMLU Dataset.

---

Supplementary Experimental results
C.1 Calibration of Other Models
Figure 8-11 show supplementary results about parameter scales for other 4 models (LLaMA, LLaMA-2, FLAN-T5 and OPT), where can draw similar conclusions with that of Section 5.2.
1.0
Raliabiliy Diagran::
650
1.0
[C] Relustaliy Diegramm: Soatee--ite
1.0
(4) Railabiliy Diagr ame Scale-M0L.))
650
0.8
0.0
0.0
Paaae Sereeoa Comeene Scathe
0.6
0.6
0.6
04
0.4
04
0.2
0.0
0.0
0.0
Contitrtne
Confitence
Contidetio
0.7
(0) ECEIACO Curve: Scale-FiLE
01
0.7
(8) ECEAACC Curve: ScaleTTREx
01
0.7
10 ECEJACC Curve: Scale-MMLL
0.2
0.8
0.0
0.
0.9
0.5
0.
0.4
0.05
0.06
0.4
0.12
E
0.3
03
0.04
Parometer Scale
Parameter Scale
Parometer Scale
Figure 8: Calibration results of I.LMMA
Roliabiity Doogram: ScalePPLE
(c) Reliabiity Diogram: Scale T-RXx,
(c) Reiability om: Scale MILU
1997% argpuu PR
Pkr Parameter Soaae
Plen Cocomnal Scatie
0.0
0.4
0.8
1.0
0.0
0.4
0.6
0.8
3.0
7B
0.0
0.4
0.6
0.8
3.0
78
ECE/RCC PILE
ECENICC Rcx
ECE/ACC NNLU
ACC
ACC
0.7
0.5
0.4
0.04
ECE
0.05
0.01
0.2
0.2
0.94
0.02
011
0.1
0.01
0.0
75
135
705
0.
75
133
705
0.03
0.0
78
130
105
0.0
Figure 9: Calibration results of LLaMA-2.

---

(a) Rolabilty Diagrom: Scale PLL.
(c) Relabiity Blogram: Scale 1-88
(c) Raliabllty Mlogram: scale RWLy
Perametee Scates
Aesamattee Scatae
0.6
0.0
base
0.0
base
0.0
basd
Confidence
Confidencee
Confidence
ELEJACC Lurve: Scalo-ILL
(4) ELEIROC CurvE: Scale-T-RE
011 ECHAALL Curve: Scale NKLU
0.4
ACC
0.20
0.4
ACC
0.25
0.5
ACC
X
0.25 g
g 0.3
0.2
0.13
0.2
0.35
0.0
0350
pare
100
0.1
0.0
0350
large
3
301
911
0.0
bw5
Iarge
301
9.15
Figure 10: Calibration results of FLAN-T5.
Refabliity SiLEe-PILE,
Relabiiity Scale-TREy
Paramater Stales
0.6
2.6
0.4
0.4
0.4
0.2
0.2
0.2
0.0
0.0
0.0
0.0
0.2
3.4
3.6
0,8
1.0
125m
0.0
0.2
0.4
0.6
0.8
1.0
125m
0.0
0.2
0.4
0.6
0.0
100
125m
Cunfidence
Carfidence
Conficence
39
0.02
010
0.3
0.00
0.12
0.3
0.04
0.1
0.02
0.1
0.04
011
1.4mmm0m 267. (67
686
123m50mm
267 067
65b
13mm5mm 163
300 656
Paramete Scale
Parometer Scale
Parameter Scale
Figure II: Calibration results of OPT.

---

C.2 Accuracy Results in Alignment Stage
Figure 12 shows the accuracy of different model outputs on the the three datasets. As can be seen in the figure, accuracy generally keeps stable and only fluctuates with different training data.
PILE
MMLU
0.5
0.5
0.5
0.4
0.4
0.4
0.3
0.3
0.0
Alpace
0A
Ap80 LONA OHA
RLHF
Alpaca
0A
Agprce 10
GALLOMA
RLHF
Alpaca
04
Apaso LONA OLLORA
RLHF
ULMM
[poch-1
Epoch-2
Epech-3
Figure 12: Model accuracy using different alignment training settings.


Reviews:
[{"review_id": "Re4JM5Ef4S", "reviewer": "Reviewer_fbVH", "summary": "", "questions": "A) Does it refer to a understanding or generation?\n\n[110] To evaluate model calibration on common text understanding, we use Causal Language Modeling (CLM) task,\n\nB) What were the criteria for selecting Reliability Diagram and ECE for the analysis? It seems that the reference was the work of Guo el al. (2017). If so, this should be indicated.\n\nC) The T-Rex dataset is mentioned to assess the Facts Generation task. However, PILE is also said to be used as a test (line 305). For what purpose is PILE used?\n\nD) In line 406 it says \"As the models are fine-tuned on instructions in this stage, we add instruction prompt for all three tasks (see Table 1)\". I understand that the models are fine-tuned only on Alpaca and OA. Are these prompts for running the inference on the tests?\n\nE) In the experiments on the alignment phase, why is LlaMA-7B used and not a Pythia model of those used in the pre-training analysis (e.g. Pythia-1B)? In that way, the evolution of accuracies and calibration along the different steps (pre-training, fine-tuning, and RLHF) could be analysed in a more coherent way.\n\nF) Regarding the comparison between Alpaca and OA datasets, it is pointed out that OA implies more calibrated models because of its greater diversity. In addition to ECE, do you have results from any other metrics that point to this fact?\nOn the other hand, it is said that OA is less diverse because it starts from 175 seeds. And in one of the findings (line 132) it is said that \"Instruction tuning deteriorates calibration, and the incorporation of synthetic data exacerbates such effect”. It is therefore assumed that all automatic datasets are inherently non-diverse. Is it safe to say that the automatic methodology determines less diversity in instructions than manual methods?\n\nG) Why have accuracies not been included in the alignment experiments? They would be of great value in order to contextualise and interpret the ECE values in a more nuanced way.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate the reviewer's thorough examination and insightful comments. For missing references and suggestions on writing, we will try our best to improve in our next version. Following are some replies to your questions. \n\n### **About the Limitation of Contribution**\n\n> There is no considerable contribution regarding SOTA ... The contribution of this work seems limited to analyzing the number of pre-training steps and different configurations of fine-tuning and RLHF, in addition to including CLM and fact generation tasks in the analysis. \n\nIn this paper, the comprehensive examination of **SFT** and **RLHF** significantly broadens the scope of existing works by Kadavath et al., which already provides non-trivial contribution. These techniques are widely adopted in recent works or applications, consequently have a profound impact on a broad spectrum of researchers. Thus, it becomes crucial to deeply explore and address the implication of their effect on calibration behavior. In addition, we also evaluate popular efficient fine-tuning methods (e.g., LoRA) and demonstrate their regularization effects on calibration.\n\nFurthermore, we integrate the aforementioned aspects with the pretraining stage under a consistent evaluation protocol. The resulting conclusions cover the entire lifecycle of the LLM, providing a more comprehensive perspective on the emergence and evolution of calibration behavior.\n\n### **Replies to questions**\n\n- **A. Does it refer to a understanding or generation?**\n  - It refers to generation task. Sorry for the words that lead to confusion.\n\n- **B. What were the criteria for selecting Reliability Diagram and ECE for the analysis? ...**\n  - For the selection of analysis tools, we follow most previous works to select Reliability Diagram and ECE. As you pointed out, such selection might be originally from Guo el al. (2017). We do cite this literature in our work, and will further clarify this in this part as well in our next version.\n\n- **C. The T-Rex dataset is mentioned to assess the Facts Generation task. However, PILE is also said to be used as a test (line 305). For what purpose is PILE used?**\n  - Here are the main reasons why we chose T-REx dataset and still kept the PILE. **Pile** covers a wide range of high-quality natural language texts ranging from web corpus to academic papers. It thus measures LM’s essential calibration behaviour under a very generalized distribution.\n  - By contrast, **T-Rex** is constructed from Wikipedia, and is more focused on wiki facts. And the results reflects how well-calibrated LM behaves in factual scenarios, where most of the miscalibration concerns exists.\n\n- **D. About inference time prompts.**\n\n  > In line 406 it says \"As the models are fine-tuned on instructions in this stage, we add instruction prompt for all three tasks (see Table 1)\". I understand that the models are fine-tuned only on Alpaca and OA. Are these prompts for running the inference on the tests?\n\n  - Yes, they are prompts for running the inference on the tests.\n\n- **E. About why Pythia is not used in alignment experiments**\n  - LLaMA is the most prevalent choices for existing alignment studies (e.g., Alpaca, Vicuna, WizardLM, etc.) , so it is selected to address the most common concerns in real application scenarios.\n  - Nonetheless, it indeed would provide more coherent analysis to also include Pythia in the entire alignment phase, and we will definitely add such experiments in our next revised version, e.g., training Pythia-1B with SFT and RLHF.\n\n- **F. About diversity of automatic and manual datasets**\n\n  > Regarding the comparison between Alpaca and OA datasets, it is pointed out that OA implies more calibrated models because of its greater diversity. In addition to ECE, do you have results from any other metrics that point to this fact? ...  Is it safe to say that the automatic methodology determines less diversity in instructions than manual methods?\n\n  - Besides ECE, we also demonstrate the distribution in semantic space of these two instruction-following datasets in **Figure 6**, we can see that Alpaca clearly exhibits more clustered pattern while OA is much more scattered.\n\n  - The discussion about diversity of different datasets is mostly based on intuition at present. Alpaca is constructed via in-context learning where LLMs generate new instruction-response pairs according to demonstration samples. While in manually annotated dataset OA, data are crowdsourced from annotators with different interests and backgrounds. For example, here are some instructions about emails in OA dataset. They display strong personal or emotional characteristics which are beyond the generation framework of Alpaca. By contrast, instructions in Alpaca are much more homogeneous. \n\n    ```python\n    # OA\n    [\n        \"I do not want to work today. Can you imagine a disease which makes me sick for three days and formulate an email for my boss that I cannot work for the next days?\",\n        \"I'm completely overwhelmed with all the work I need to do. So many people expect stuff form me and I just can't handle it all. The more I work, the less I get done and I have so many unanswered email and things I promised to do, I'm scared to even go look at my inbox. What can I do?\",\n        \"write an email for me to a journal that i submitted a paper in it, but the review process is taking too long and the review tracking webpage did not change in a long time. I want to ask them what's wrong or to speed the process up.\"\n    ]\n    \n    # Alpaca\n    [\n        \"Create a response for a customer's email about a refund issue.\",\n        \"Generate an example of a formal email.\",\n        \"Write an email introducing yourself to a professor you will be taking an online course with.\"\n    ]\n    ```\n\n- **G. Missing accuracy results in alignment experiments**\n  - Sorry for the missing information. As the accuracy change is consistent with expectation and it is not the main target of our work, we did not present them in the paper initially. Here are the accuracies results of our alignment experiments (Results are splitted into two tables for better view). We will add them in the revised version appropriately.\n\n    |       | LLaMA | Alpaca_ep1 | Alpaca_ep2 | Alpaca_ep3 | OA_ep1 | OA_ep2 | OA_ep3 |\n    | :---: | :---: | :--------: | :--------: | :--------: | :----: | :----: | :----: |\n    | PILE  | 0.452 |   0.441    |   0.443    |   0.441    | 0.447  | 0.446  | 0.442  |\n    | T-REx | 0.407 |   0.403    |   0.407    |   0.406    |  0.41  | 0.412  | 0.411  |\n    | MMLU  | 0.366 |    0.336    |   0.354    |   0.372    | 0.331  | 0.318  | 0.334  |\n\n    |       | Alpaca_LoRA_ep1 | Alpaca_LoRA_ep2 | Alpaca_LoRA_ep3 | Alpaca_RLHF_ep1 | Alpaca_RLHF_ep2 | Alpaca_RLHF_ep3 |\n    | :---: | :-------------: | :-------------: | :-------------: | :-------------: | :-------------: | :-------------: |\n    | PILE  |      0.444      |      0.444      |      0.445      |      0.441      |      0.441      |      0.441      |\n    | T-REx |      0.415      |      0.409      |      0.411      |      0.405      |      0.405      |      0.405      |\n    | MMLU  |      0.362      |      0.367      |      0.369      |      0.37       |      0.368      |      0.369      |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "• The main problem (calibration of LLMs) raised is important, and has not been extensively addressed in the literature.\n    • Many of the specific aspects discussed in the paper are also unexplored in previous studies.\n    • The proposed experimentation makes it possible to answer, albeit partially, the questions proposed in the analysis.", "reasons_to_reject": "• There is no considerable contribution regarding SOTA.\n        ◦  Kadavath et al., (2022) already point to a positive correlation between the calibration of LLMs and their size according to evaluations on tasks of diverse multiple choice and true/false questions. The contribution of this work seems limited to analyzing the number of pre-training steps and different configurations of fine-tuning and RLHF, in addition to including CLM and fact generation tasks in the analysis.\n    • Some aspects of the experimentation and argumentation are weak, which prevent drawing solid conclusions.\n    • The quality of the paper's writing could be improved in terms of spelling, grammar, and style.", "questions_to_authors": "A) Does it refer to a understanding or generation?\n\n[110] To evaluate model calibration on common text understanding, we use Causal Language Modeling (CLM) task,\n\nB) What were the criteria for selecting Reliability Diagram and ECE for the analysis? It seems that the reference was the work of Guo el al. (2017). If so, this should be indicated.\n\nC) The T-Rex dataset is mentioned to assess the Facts Generation task. However, PILE is also said to be used as a test (line 305). For what purpose is PILE used?\n\nD) In line 406 it says \"As the models are fine-tuned on instructions in this stage, we add instruction prompt for all three tasks (see Table 1)\". I understand that the models are fine-tuned only on Alpaca and OA. Are these prompts for running the inference on the tests?\n\nE) In the experiments on the alignment phase, why is LlaMA-7B used and not a Pythia model of those used in the pre-training analysis (e.g. Pythia-1B)? In that way, the evolution of accuracies and calibration along the different steps (pre-training, fine-tuning, and RLHF) could be analysed in a more coherent way.\n\nF) Regarding the comparison between Alpaca and OA datasets, it is pointed out that OA implies more calibrated models because of its greater diversity. In addition to ECE, do you have results from any other metrics that point to this fact?\nOn the other hand, it is said that OA is less diverse because it starts from 175 seeds. And in one of the findings (line 132) it is said that \"Instruction tuning deteriorates calibration, and the incorporation of synthetic data exacerbates such effect”. It is therefore assumed that all automatic datasets are inherently non-diverse. Is it safe to say that the automatic methodology determines less diversity in instructions than manual methods?\n\nG) Why have accuracies not been included in the alignment experiments? They would be of great value in order to contextualise and interpret the ECE values in a more nuanced way.", "paper_topic_and_main_contributions": "The paper focuses on the problem of calibrating large language models. This is a problem that has already been raised in the literature. The paper proposes an analysis covering the different training phases of current LLMs: a) Pre-training, b) instruction fine-tuning, and c) RLHF (phases b and c correspond to the process of alignment).\n\nThe calibration analysis is performed on three tasks that in some way represent the model's capabilities for language generation, fact generation and reasoning (underderstanding). The tasks are: Causal Language Modeling (PILE dataset), Facts Generation (T-Rex dataset), and Multi-task Language Understanding or multiple choice question (MMLU benchmark).\n\nReliability Diagram and Expected Calibration Error (ECE), already defined and used by Guo et al. (2017), are used as metrics to analyse the calibration of the models.\n\nIn the analysis of calibration in the pre-training phase, the effects produced by the size of the model (parameters) and the number of training steps are studied. For this analysis, the models of the Pythia suite, already created for different sizes and numbers of steps, are used. The results point to better calibration (and also accuracy) as the model size and number of steps increase.\n\nIn the analysis of the alignment phase, LlaMa-7B is used as a model. As datasets for instruction tuning, the use of Alpaca (synthetic) and OpenAssistant (manual) are analysed. For fine-tuning the model, the full fine-tuning and LoRA methods are compared. The results show that the calibration of the model degrades the higher the number of epochs of the fine-tuning process. This degradation is greater with Alpaca, and LoRA produces more calibrated models than full fine-tuning. The RLHF is performed on the Alpaca fine-tuned model using the data from Pen et al. (2023). It is observed that the RLHF hardly degrades the calibration of the model. However, the corresponding accuracyes are not presented in order to contextualise these calibrations.\n\nMain contributions:\n\n    - Analysis on factors influencing the calibration of LLMs throughout their construction (pre-training, fine-tuning and RLHF).\n\nAccording to the results:\n\n    - Calibration improves as the model size and amount of training increase.\n\n    - Fine-tuning tends to deteriorate calibration, although this can be mitigated by using LoRA.\n\n    - RLHF slightly deteriorates model calibration.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "The following reference is proposed if appropriate:\n\nLin, S., Hilton, J., & Evans, O. (2022). Teaching models to express their uncertainty in words. arXiv preprint arXiv:2205.14334."}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "The paper has much room for improvement in terms of writing. The following is an exhaustive list for the introduction only (the other sections require similar improvements).\n\n\n031: A reference is missing.\n\n032: capability at -> capability in\n\n035: and has -> and this has\n\n040: The paper (Kung et al., 2023) does not address reliability and trustworthiness.\n\n043: potentially severe challenges. -> potentially severe, challenges.\n\n046: are not easily attributed or interpreted.->are not easily attributable or interpretable.\n\n053: (Guo et al., 2017). No earlier reference?\n\n053: and inform -> and informs\n\n054: outputs is should ->  outputs should\n\n055: even though not always correct. ->even though they may not always be correct.\n\n059: them are -> them to be\n\n064: which indicates -> indicating\n\n067: model prediction. -> model’s prediction\n\n069: Studies also pointed -> Studies have also pointed\n\n076: while the aligned language models are less focused on. -> while the aligned language models receive less focus.\n\n082: thus can not provide insight on how -> thus can not provide insight into how\n\n086: and our work is aimed to fill this blank. -> our work aims to fill this gap.\n\n090: provide evidence on how to acquire ->  provide evidence on how to achieve\n\n096: how model calibration is changed ->  how model calibration changes\n\n099: For alignment -> For the alignment\n\n106: are two widely concerned problems of -> are two widely considered issues with\n\n108: “when applied to different fields.” It is not understood.\n\n109: “For this purpose, we design three tasks for each of the settings above.” What setting is being talked about?\n\n111: “To evaluate model calibration on common text understanding,” only understanding or also generation?\n\n112: “Causal Language Modeling (CLM)” Subsequently, both the acronym and the full form are used.\n\n115: designed facts generation task -> designed a a facts generation task\n\n116: facts-related content. -> fact-related content. \n\n120: the most possible one. -> the most probable one.\n\n124: Model calibration improves with respect to both -> Model calibration improves with both\n\n127: Improvement of calibration accuracy -> Improvement in calibration accuracy\n\n128: facutality -> factuality\n\n127: “Improvement of calibration accuracy is more significant on facutality or language understanding tasks than language modeling task.”\nThe introduction (lines 110-120) mentions three tasks to assess language comprehension, factuality, and reasoning skills. It would be useful to maintain consistency and clarify what is considered a task and what is considered an ability.\n\n134: exacerbates such effect. -> exacerbates this effect. \n\n135: Learning from reward model -> Learning from a reward model\n\n142: “Calibration accuracy evolves consistently across different downstream tasks including factuality, language understanding or vanilla language modeling.” Same as for line 127.\n\n150: hopefully can shed -> hopefully this can shed \n\n151: trust-worthy -> trustworthy\n\n\nOther comments:\n\n-The design of Figure 1 could be improved.\n\n-292: A short definition of the PILE dataset should be included.\n\n-The captions in Figures 3 and 4 should be more explanatory. For example, the relationship between dataset and task should be mentioned.\n\n-Figure 5 should include the accuracies."}}, {"review_id": "zdG7nFWGAb", "reviewer": "Reviewer_ojDc", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the valuable review and affirmative comments. As for some of the reviewer's concerns, we present our replies and supplemental results here:\n\n### **Experimental Results on Larger Models**\n\n- Actually in the pre-training stage we tested models up to 12B (sorry for the mis-labeled legend in figure 3&4, we will correct them in the revised version). We have further conducted some of our experiments on larger models up to 65B (LLaMA). Results are as follows, the first table contains accuracy results and the second represents ECE. **For larger models, the conclusions made in the paper invariantly hold.** We can see that larger models possess better accuracy and calibration. While the accuracy improves significantly, the difference of ECE is minor as these large models are fairly well-calibrated in our three tasks and it might be difficult to improve calibration further. We will add these results in a revised version.\n\n    **Accuracy Results of LLaMA**\n\n    |       | LLaMA-7B | LLaMA-13B | LLaMA-30B | LLaMA-65B |\n    | :---: | :------: | :-------: | :-------: | :-------: |\n    | PILE  |  0.456   |   0.474   |   0.492   |    0.5    |\n    | T-REx |   0.41   |   0.46    |   0.51    |   0.535   |\n    | MMLU  |  0.367   |   0.478   |   0.579   |   0.635   |\n\n    **ECE  Results of LLaMA**\n\n    |       | LLaMA-7B | LLaMA-13B | LLaMA-30B | LLaMA-65B |\n    | :---: | :------: | :-------: | :-------: | :-------: |\n    | PILE  |  0.038   |   0.032   |   0.032   |   0.034   |\n    | T-REx |   0.06   |   0.048   |   0.042   |   0.042   |\n    | MMLU  |  0.021   |   0.017   |   0.025   |   0.016   |\n\n\n### **On the choice of Benchmarks**\n\n- The motivation of choosing these datasets is that they are very representative, respectively for factuality (T-REx), general language understanding (MMLU), and text generation (PILE).\n- PILE itself is a very generalized high-quality texts, including Wikipedia, web corpus, academic papers or even code. It can already serves as qualified testbed for evaluating the essential calibration behavior of LLMs across various domains. We will consider other available datasets in future works.\n\n\n### **About Writing**\n\n- Thanks for the insightful suggestion to include more introduction about the consequences of bad calibration and the addressing methods. In this work, we have already demonstrated that increasing model scales and employing parameter efficient tuning methods can effectively alleviate the degeneration of calibration. The experimental analysis also implies that **diversity guided instruction data construction** may also be very promising. We will add a detailed discussion in a separate section to shed more lights on effective or promising **solutions** for improving calibration in a revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Calibration is an understudied problem and, as the authors state, is closely linked to factuality (hallucinations) and trustworthiness of systems. They conduct a thorough investigation of each stage of the LLM pipeline from pre-training to instruction tuning to RLHF. They nicely include definitions and related works to help guide readers more unfamiliar with calibration literature.", "reasons_to_reject": "The models tested only go up to 7B, and beyond 7B models behave much differently. While understandably difficult to finetune these larger models, it is possible to use them for inference on smaller GPUs. I think it's essential to add lessons from larger models in, even if the comparison is not apples to apples with the Pythia models. Showing Llama, Falcon, etc. calibration scaling laws would be helpful (for the model sizes -- the training length is not possible to compare).\n\nAnalogously, it's hard to read too much into the results because it's only tested on 3 datasets, and the CLM one does not feel particularly useful.  More effort needs to be done to motivate the choice of benchmarks and possibly why more benchmarks were not chosen.  Given that no benchmark specific tuning is necessary, hopefully this isn't too much effort.\n\nLastly, some of the text in the first pages could be tightened to allow for more of a deeper dive into the consequences of poorly calibrated models and possible ways to address it. Instruction tuning is a necessary step - so what should we conclude from this finding?", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper formally studies the impact of pre-training, instruction-tuning, and RLHF dynamics on the downstream calibration of model outputs across 3 tasks: causal language modeling (dev / test set of PILE), generating facts (T-REx), and NLU (MMLU benchmark).\n\nAmong others, they derive a few new insights on the impact of dynamics on calibration:\n- calibration improves with model size and training times\n- gains occur more dramatically for T-REx and MMLU benchmarks\n- instruction tuning leads to worse calibration yet PEFT and RLHF do not impact it as much", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "UE2t0ueDdS", "reviewer": "Reviewer_kGi5", "summary": "", "questions": "A. Is there any experiment results on OA_LoRA for Section 6.2?", "limitations": "", "rating": 2, "confidence": 5, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the thorough and detailed review. Following are our replies containing further illustration and experimental results to hopefully address the reviewer's concerns.\n\n### 1. The qualification of CLM and Fact Generation for calibration analysis\n\n> Evaluating the calibration of causal language modeling (CLM) does not make sense to me.\n>\n> The evaluation for T-REx is also odd.\n\n- For CLM task, if we look at a specific instance, it indeed “Not having a ground truth label”. But for a generalized distributed subset, the predicting accuracy along with the calibration error can well reflects the capability of LLMs and serves as sufficient metrics. For example, GPT4 ([1](#[1])) employs the accuracy on pretraining loss to predictably measure the overall capability of the model. On the other hand, In terms of the selected dataset, **PILE** contains a very **generalized distribution** of **high-quality** natural texts, including web corpus, academic papers, etc. The overall distribution of such aggregated natural texts can well represent how people use language, thus serves as qualified measurement for model capability as well as calibration behavior.\n- Note that for fact generation task, instead of directly ask factual questions, we performed evaluation in a CLM way where prefixes were given and models were asked to generate the entity token. Thus the problem about fact generation task can actually be induced to the first problem -- whether evaluation in the form of CLM is reasonable for model calibration, which we have illustrated above.\n\n[1] OpenAI (2023). GPT-4 Technical Report. *ArXiv, abs/2303.08774*\n\n### 2. On the performance on MMLU in Figure 3 and Figure 4 \n\n- First, regardless of the performance, we can already observe strong and clear patterns in the ECE results, calibration error **monotonically decreases** w.r.t. increased model scales and training dynamics, which can not be treated as randomness.\n- To further verify that such conclusions are not subjected to specific accuracy, we also apply the experiments of model scales on better-performing models(LLaMA). As can be seen in the following results(where the two tables represent accuracy and ECE respectively), with more accurate LLMs, the conclusions that larger models are more calibrated remain consistent. We will add these results in a revised version to make the arguments and analysis more solid. As for training dynamics experiments, we can not conduct further experiments for now as Pythia does not provide models with better performance on MMLU.\n\n    **Accuracy Results of LLaMA**\n\n    |       | LLaMA-7B | LLaMA-13B | LLaMA-30B | LLaMA-65B |\n    | :---: | :------: | :-------: | :-------: | :-------: |\n    | PILE  |  0.456   |   0.474   |   0.492   |    0.5    |\n    | T-REx |   0.41   |   0.46    |   0.51    |   0.535   |\n    | MMLU  |  0.367   |   0.478   |   0.579   |   0.635   |\n\n    **ECE  Results of LLaMA**\n\n    |       | LLaMA-7B | LLaMA-13B | LLaMA-30B | LLaMA-65B |\n    | :---: | :------: | :-------: | :-------: | :-------: |\n    | PILE  |  0.038   |   0.032   |   0.032   |   0.034   |\n    | T-REx |   0.06   |   0.048   |   0.042   |   0.042   |\n    | MMLU  |  0.021   |   0.017   |   0.025   |   0.016   |\n\n### 3. Missing accuracy results on alignment\n\n- For the question that accuracy/performance in the alignment training is not provided, we apologize for the missing information. We will add the results in the revised version.\n\n- Accuracy results are as follows (Results are splitted into two tables for better view), in which the *LLaMA* column refers to results of the original *LLaMA* model, column labeled with *Alpaca/OA/Alpaca_LoRA* refers to performance of instruction-tuned models with different dataset or methods, and the last three columns are results of RLHF-tuned models.  From the table we can see that accuracy generally remains stable and only fluctuates with different datasets and training methods.  We also want to emphasize that this work mainly focuses on model calibration, and accuracy is just for reference.\n\n    |       | LLaMA | Alpaca_ep1 | Alpaca_ep2 | Alpaca_ep3 | OA_ep1 | OA_ep2 | OA_ep3 |\n    | :---: | :---: | :--------: | :--------: | :--------: | :----: | :----: | :----: |\n    | PILE  | 0.452 |   0.441    |   0.443    |   0.441    | 0.447  | 0.446  | 0.442  |\n    | T-REx | 0.407 |   0.403    |   0.407    |   0.406    |  0.41  | 0.412  | 0.411  |\n    | MMLU  | 0.366 |    0.336    |   0.354    |   0.372    | 0.331  | 0.318  | 0.334  |\n\n    |       | Alpaca_LoRA_ep1 | Alpaca_LoRA_ep2 | Alpaca_LoRA_ep3 | Alpaca_RLHF_ep1 | Alpaca_RLHF_ep2 | Alpaca_RLHF_ep3 |\n    | :---: | :-------------: | :-------------: | :-------------: | :-------------: | :-------------: | :-------------: |\n    | PILE  |      0.444      |      0.444      |      0.445      |      0.441      |      0.441      |      0.441      |\n    | T-REx |      0.415      |      0.409      |      0.411      |      0.405      |      0.405      |      0.405      |\n    | MMLU  |      0.362      |      0.367      |      0.369      |      0.37       |      0.368      |      0.369      |\n\n### 4. Conclusions in this paper are drawn too dogmatically without enough evidence\n\n- Thanks for pointing out the problem of the word overfitting here. “Over-fitting” is indeed not a very rigorous description. The main observation in the paper is that “degraded ECE is clearly associated with training epochs”, longer training (i.e., fitting instead of over-fitting) consistently results in worse ECE error. In strict terms, training on the same dataset for more than one epoch does not necessarily results in overfitting (which is commonly recognized as when test errors starting to get worse while train errors keeps getting better).  We will edit this part in the revised version.\n\n- We agree that observations about diversity at present is correlation but not causation. The discussions in our paper are based on intuition at present. As can be seen in the cases about email below, instructions in OA show more personal characteristics while ones in Alpaca are more homogeneous. To strictly prove the relation of diversity and calibration, more quantative experiments should be conducted. We will re-form our conjectures to make it more rigorous and report any further experimental results in the revised version.\n\n    ```python\n    # OA\n    [\n        \"I do not want to work today. Can you imagine a disease which makes me sick for three days and formulate an email for my boss that I cannot work for the next days?\",\n        \"I'm completely overwhelmed with all the work I need to do. So many people expect stuff form me and I just can't handle it all. The more I work, the less I get done and I have so many unanswered email and things I promised to do, I'm scared to even go look at my inbox. What can I do?\",\n        \"write an email for me to a journal that i submitted a paper in it, but the review process is taking too long and the review tracking webpage did not change in a long time. I want to ask them what's wrong or to speed the process up.\"\n    ]\n\n    # Alpaca\n    [\n        \"Create a response for a customer's email about a refund issue.\",\n        \"Generate an example of a formal email.\",\n        \"Write an email introducing yourself to a professor you will be taking an online course with.\"\n    ]\n    ```\n\n### 5. All experiments are based on a single run. \n\n- First, to prove that our observation is not a randomness, we report results of 5 runs on MMLU dataset using models trained on Alpaca with different random seeds (each cell contains results of three epochs). It appears that conclusions are the same for all runs and also the same with our conclusion in the paper (ECE increases monotonically with the training process). We will include results on other two datasets in a revised version.\n\n    **Accuracy and ECE results of 5 runs on MMLU dataset**\n\n    |          |       Run 1       |       Run2        |       Run3        |       Run4        |       Run5       |\n    | :------: | :---------------: | :---------------: | :---------------: | :---------------: | :--------------: |\n    | Accuracy | 0.366/0.324/0.336 | 0.385/0.377/0.388 | 0.336/0.369/0.38  | 0.379/0.348/0.367 | 0.36/0.396/0.388 |\n    |   ECE    | 0.019/0.085/0.125 | 0.038/0.035/0.07  | 0.066/0.084/0.132 | 0.023/0.137/0.174 | 0.03/0.117/0.165 |\n\n- Second, the main target of this work is to display the possible effect that pre-training and alignment training will bring to model calibration, instead of studying the influence of each hyperparameter. So we chose the training settings to change according to the design space of this work (Figure 1) and leave the others as commonly used or recommended.\n\n- Nevertheless, it is reasonable to experiment on more hyperparameters combinations to provide more solid conclusions. We will try our best to include these results in a revised version.\n\n### 6. Experimental Results on OA_LoRA\n\n> Is there any experiment results on OA_LoRA for Section 6.2?\n\n- The reviewer mentioned that experiment results for OA_LoRA should be provided. Following are the results, from which we can see that **ECE increases in a much slower pace than models directly trained** (results in Figure 5). Such observation proves that our conclusion in previous experiments still holds (LoRA can mitigate the calibration degeneration caused by instruction tuning). We will add these results in the revised version.\n\n    **Accuracy Result of OA_LoRA**\n\n    |       | Epoch 1 | Epoch 2 | Epoch 3 |\n    | :---: | :-----: | :-----: | :-----: |\n    | PILE  |  0.447  |  0.449  |  0.45   |\n    | T-REx |  0.416  |  0.42   |  0.42   |\n    | MMLU  |  0.355  |  0.332  |  0.311  |\n\n    **ECE Results of OA_LoRA**\n\n    |       | Epoch 1 | Epoch 2 | Epoch 3 |\n    | :---: | :-----: | :-----: | :-----: |\n    | PILE  |  0.058  |  0.057  |  0.056  |\n    | T-REx |  0.08   |  0.078  |  0.077  |\n    | MMLU  |  0.017  |  0.028  |  0.069  |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Confidence calibration is vital for building and deploying safer and more helpful LLMs.\n2. The paper is well-written and easy to understand", "reasons_to_reject": "1. **Evaluation methods  are not very convincing**\n    - Evaluating the calibration of causal language modeling (CLM) does not make sense to me. Unlike classification, which has a ground truth answer for a given question and allows one to calculate the accuracy, CLM does not have a ground truth token given a prefix. Given a prefix, the next token can be any token that can make the ongoing continuation a reasonable sentence. In this case, using the token in the original dataset is unreasonable. While prior works study the calibration for NMT, I argue that causal language modeling is different from translation as translation still has some ground truth for the target sentence given the source sentence. This is not the case for CLM. One may argue that when calculating perplexity, we also treat the sentences in the test set as ground truth. This is slightly different since, when calculating perplexity, we are not concerned about confidence and accuracy.\n    - The evaluation for T-REx is also odd. First, I disagree with counting the model to predict accurately when it predicts the first token accurately. This is not how accuracy should be calculated. Next, even when the first token is not the expected token, it is still possible that the model completes the sentence in another way that answers the factual question accurately. Requiring the model to follow the predefined format for calculating the accuracy is too strict and might underestimate the accuracy.\n\n2. **The performance on MMLU in Figure 3 and Figure 4 is too low to discuss the calibration**\n    - The reason why we are interested in confidence calibration is that when we get a model that somewhat works, we want to use its confidence to tell us when it works and when it does not work. However, the performance of MMLU in Figure 3 and 4 is mostly as worse as random. In this case, it seems meaningless to discuss the models' confidence calibration since they are not usable.\n\n3. **Section 6 (Instruction tuning and RLHF) does not show the accuracy/performance of models fine-tuned after instruction tuning and RLHF**\n    - It is unclear if the instruction tuning and RLHF experiments in this paper really yield models that can follow instructions. Chances are the instruction tuning and RLHF is unsuccessful, and the loss diverges, so the tuned models cannot work well on any downstream tasks. Without any experiments testing the performance of instruction-tuned or RLHF-tuned models, it is hard to interpret the results here.\n\n4. **Conclusions in this paper are drawn too dogmatically without enough evidence**\n    - The connection between overfitting on instruction fine-tuning datasets and the ECE on the three tasks is not very clear. I do not see why the rising ECE for the three tasks during instruction tuning can be attributed to overfitting on the training dataset.\n    - The connection between semantic diversity and ECE is not clear. The experiment in Figure 6 only shows a connection between semantic diversity (shown in T-SNE) and the ECE of the three test tasks. This is a correlation, not causation. OA and Alpaca may have differences other than semantic diversity. Without more analysis and careful interpretations, we can not know if semantic diversity is really the cause of lower/higher ECE.\n\n5. **All experiments are based on a single run. It is unclear if the observations hold when changing the hyperparameter of fine-tuning the models**. RLHF is very unstable, and fine-tuning an LLM can have very different results depending on the hyperparameters, optimizers, and parameter-efficient fine-tuning methods used. Without more experiments, it is hard to tell if the results in this paper is a general trend or a random observation.", "questions_to_authors": "A. Is there any experiment results on OA_LoRA for Section 6.2?", "paper_topic_and_main_contributions": "Paper Topic\n===\nThis paper studies the confidence calibration of large language models. They study how the calibration of LLMs changes with respect to the LLM size and the pre-training time on three tasks: causal language modeling, T-REx (factual knowledge), and MMLU (multitask language understanding). They also look into how instruction tuning and RLHF affect the calibration of LLMs. They find that large LLMs and LLM pre-trains for longer calibrates better. They also find that instruction tuning harms calibration ability while RLHF merely changes the calibration ability of LLMs.\n\nContributions\n===\nNLP engineering experiment", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A"}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line 054: is(?) should be trusted\nLine 031: Missing reference\nLine 192: Q A\nLine 592: There is no OPT-130B\nLine 218: `they do not take a further step towards the intrinsic mechanism while merely exploring the verification on the surface`. I feel like this paper also does not explore any intrinsic mechanism and only explores the superficial ECE, so it is hard to use the above sentence to justify the contribution of this paper."}}]

Author rebuttal:

Meta-review: This paper performs an empirical analysis of the calibration of LLMs, after pretraining, instruction fine-tuning (full and LoRA), and RLHF, varying model size and training steps. Reviewers mostly agreed this was an important and informative analysis, though some of the findings (namely that larger models appear to be better calibrated after pretraining) are not new and replicate prior work. However, this work does provide analysis on additional datasets and training settings (after fine-tuning and RLHF). One reviewer questioned the soundness of evaluation of calibration using ECE, but seemed less concerned after the authors provided additional experiments. Overall this paper provides a comprehensive understanding of model calibration that goes beyond what was been studied previously, using existing accepted strategies for evaluating LLMs.
Decision: Accept-Findings