LitCab: Lightweight Language Model Calibration over Short- and Long-form Responses
Authors: Xin Liu
Muhammad Khalifa
Lu Wang
Venue: ICLR 2024 poster (2024)
Keywords: calibration
hallucination
large language model
Primary area: generative models

Abstract: A model is considered well-calibrated when its probability estimate aligns with the actual likelihood of the output being correct. Calibrating language models (LMs) is crucial, as it plays a vital role in detecting and mitigating hallucinations of LMs as well as building more trustworthy models. However, standard calibration techniques may not be suited for LM calibration. For instance, post-processing methods such as temperature scaling do not reorder the candidate generations. On the other hand, training-based methods require fine-tuning the entire model, which is impractical for LMs of large scale. We present LitCab, a lightweight calibration mechanism consisting of a single linear layer that takes the input text representation and predicts a bias term, which is then added to the LM output logits. LitCab improves model calibration by only adding < 2% of the original model parameters. For evaluation, we construct CaT, a benchmark consisting of eight text generation tasks, covering responses ranging from short phrases to paragraphs. We test LitCab with Llama2-7B, where it improves calibration across all tasks, reducing the average ECE score by as large as 30%. We further conduct a comprehensive evaluation with multiple popular open-sourced LMs from GPT and LLaMA families, yielding the following key findings: (i) Larger models within the same family exhibit better calibration on tasks with short generation tasks, but not necessarily for longer ones. (ii) GPT-family models show superior calibration compared to LLaMA, Llama2, and Vicuna models, despite having much fewer parameters. (iii) Fine-tuning pretrained model (e.g., LLaMA) with samples of limited purpose (e.g., conversations) may lead to worse calibration, highlighting the importance of fine-tuning setups for calibrating LMs.

Published at ICLR 2024
LITCAB: LIGHTWEIGHT LANGUAGE MODEL CALI- BRATION OVER SHORT- AND LONG-FORM RESPONSES
Xin Liu, Muhammad Khalifa, Lu Wang Computer Science and Engineering University of Michigan Ann Arbor, MI (liuxincs, khalifam, wangluxyJ@umic wwwoummichheeee
ABSTRACT
A model is considered well-calibrated when its probability estimate aligns with the actual likelihood of the output being correct. Calibrating language models (LMs) is crucial, as it plays a vital role in detecting and mitigating hallucinations of LMs well as building more trustworthy models. However, standard calibration techniques may not be suited for LM calibration. For instance, post-procsssing methods such temperature scaling do not reorder the candidate generations. On the other hand, training-basee methods require fine-tuning the entire model, which is impractical for LMs of large scale. We present LITCAR, lightweight calibration mechanism consisting of single linear layer that takes the input text representation and predicts a bias term, which is then added to the LM output logits. LITCAB improves model calibration by only adding < 2% of the original model parameters. For evaluation, we construct CAT, a benchmark consisting of eight text generation tasks, covering responses ranging from short phrases to para- graphs. We test LITCAB with Llama2-7B, where it improves calibration across all tasks, reducing the average ECE score by large as 30%. We further con- duct comprehensive evaluation with multiple popular open-sourced LMs from GPT and LLaMA families. yielding the following key findings: (i) Larger models within the same family exhibit better calibration on tasks with short generation tasks, but not necessarily for longer ones. (ii) GPT-family models show supe- rior calibration compared to LLaMA, Llama2, and Vicuna models, despite having much fewer parameters. (iii) Fine-tuning pretrained model (e.g., LLaMA) with samples of limited purpose (e.g., conversations) may lead to worse calibration, highlighting the importance of fine-tuning setups for calibrating LMs.'
INTRODUCTION
While modern language models (LMs) exhibit impressive performance across various tasks (Brown al., 2020; OpenAI, 2023), they suffer from hallucination (Lin et al., 2021; Zhang et al., 2023), where they can provide nonfactual responses with high confidence. The issue of hallucination un- dermines user trust and significantly limits the applicability of LMs domains that require a high degree of reliability, such as legal, financial, and educational sectors. While eliminating hallu- cination in LMs altogether is highly challenging, calibrating LMs (Nguyen & O'Connor, 2015) by aligning their confidence with the actual probability of output correctness can certainly help. Specif- ically, a well-calibrated model enables users to gauge the model's confidence and make informed decisions about whether to trust its outputs. Moreover, hallucinated facts can be filtered out when the confidence level is below a certain threshold.
Previous approaches to calibrating neural models mainly fall into two categories: (i) post-processing and (ii) training-based methods, as outlined in Figure 1. Post- Post-processing techniques have the ad- vantage of not changing the model weights by directly manipulating the sequence probabilities. Example techniques from this family are temperature scaling (Liang et al., 2018) and Platt scaling (Niculescu- Miiiesscc-iiill&& Caruana, 2005). Post-processing calibration directly adjusts the sharpness of the
Data and code available https: http//gttttttttg// comm/aunchhn p/ tia..

---

Published at ICLR 2024
output distribution, but they do not alter the relative confidence rankings among the outputs. This makes them ineffective if one wishes to filter out incorrect outputs based on a confidence threshold. Training- training-based methods, on the other hand, update the model weights to produce a boost calibra- tion. This family of methods includes label-smoothing (Szegedy et al., 2016), mix-up (Zhang et al., 2020a), regularization (Pereyra et al., 2017) to mitigate the model's overconfidence. However, training the entire model, which is commonly done for these methods, is impractical for today's LMs with billions of parameters.
Motivated by making training-basee based calibration more efficient, we present LITCAB, lightweight calibration technique for LMs. LITCAB takes as input a sequence of hidden states from the LM's final layer and produces a set of logit biases to ad- just the generation confidence. Specifically, LITCAB trains single linear layer on top of the model's last layer using a con- trastive max-margin objective, to maximize the token proba- bilities for correct generations and lower the likelihood for in- correct ones. As LITCAB can adjust the confidence ranking among outputs, a capability that post-procsssing methods lack, it offers greater flexibility. Moreover, the trainable parameters count of LITCAB is less than 2% of the original LM param- eters, making significantly more computationally efficient than standard full-model training-basee based approaches.
Efficiency
Flexibility
Post-procesing Methods
Training-
Methods
Temperature Staling sam
LirCAn
Label Labot-Smoothing Sopple anl
Mix-up Change 2000
poo aww.maaiiaa 2004
Scaling
Regularication Prrmyat 43771
Calibration techniques for neural models. LITCAB com- bines the benefits of computational efficiency with great flexibility.
Figure 1:
We apply LITCAB to Llama2-7B (Touvron et al., 2023b) and compare it against several competitive baselines, including post-processing, training- verbalization- and consistency- based methods (Kuhn al., 2023; Kadavath al., 2022: Xiong al., 2023). Our experiments demonstrate the effectiveness of LITCAB, which exhibits uniformly superior calibration than baselines across the text generation benchmarks.
During calibration evaluation, we note that existing work mainly studies short answer QA (Tian al., 2023; Xiong al., 2023), little attention has been given to LM calibration over long-form outputs. To address this gap, we construct and release Calibration evalua Benchmark (CAT) consisting eight text generation tasks that cover generations encompassing phrases, sentences, and up to paragraphs. We further conduct extensive evaluation over seven open-source LMs, including GPT (Radford et al., 2019; Wang & Komatsuzaki, 2021). LLaMA (Touvron et al., 2023a), Llama2 (Touvron al., 2023b), and Vicuna (Chiang al., 2023), with sizes ranging from 1.5B to 30B.
Our findings underscore insights regarding the relationships among calibration, model scale, and task performance, well the impact of instruction tuning calibration. First, larger models within the same family demonstrate better calibration for phrase-level tasks, but not necessarily for sentence- and paragraph-level tasks. In addition, when comparing different LM families, despite superior task performance, larger models from various families tend to be worse calibrated compared the smaller GPT-2 XL (1.5B) model. This observation suggests that model performance and calibration should be optimized concurrently rather than treated as separate objectives. Furthermore, we find that instruction tuning negatively impacts model calibration, as evidenced by Vicuna-131 exhibiting poorer calibration compared to its predecessor, LLaMA-13B.
To summarize, our contributions can be summarized as follows:
We propose LITCAB, lightweight calibration mechanism for LMs, which does not need any additional training or fine-tuning of the LM itself and adds < 2% extra parameters. Exper- imental results demonstrate its effectiveness compared to methods that use post-processing, model training, verbalization, and self-consistency.
We construct and release CAT, benchmark designed for evaluating text generation tasks with responses in both short and long forms. CAT is comprised of eight text generation tasks that include responses ranging from short phrases to sentences and to paragraphs.
We formulate an evaluation strategy for assessing the calibration of paragraph-level genera- tions. This approach involves extracting distinct claims from the generated content, estimating the confidence associated with each claim, and then analyzing the calibration at the claim level.
Based on CAT, we conduct an extensive evaluation of the calibration of seven state-of tate-of-the-an the-art open-source LMs and extract three key findings.
2

---

Published at ICLR 2024
2 RELATED WORK
2.1 NEURAL MODEL CALIBRATION
Calibration of neural models was first studied in the context of text classification tasks (Guo et al., 2017; Wenger et al., 2020). Given the goal of strengthening the alignment between model confidence and the likelihood of the output correctness, previous studies can be mainly boiled down to two categories: proccesssssing and training-basee methods. Post-processing methods do not alter the model weights and only adjust the model confidence (i.c., token probabilities). Niculescu-Mizs & Caruana (2005) adopt Platt scaling to calibrate model predicted confidence by fitting a sigmoid function. Similarly, Guo et al. (2017) propose to apply a temperature in the softmax function during model prediction to scale the model predicted probabilities, with the temperature being tuned on the validation data. Zhang al. (2020a) further improves temperature scaling through ensemble. Post- processing techniques cannot alter the relative rankings among different outputs, thus unsuitable for being directly applied for hallucination mitigation.
As for training- based methods, one common technique is label smoothing (Szegedy et al., 2016), which has been shown to be useful for reducing calibration errors. Alternatively, Pereyra et al. (2017) add a negative entropy penalty to the loss function to diminish overconfidence. Zhang et al. (2020a), on the other hand, regularize the model training by mix-up technique. While training- based methods are generally more flexible than post-editing techniques, full model training is often needed but becomes impractical as LMs grow in size. Instead, LITCAB exhibits the expressivity of training-based based methods while eliminating the need for full fine-tuning.
2.2 CONFIDENCE ESTIMATION FOR LANGUAGE MODELS (LMS)
An essential step in evaluating the calibration for LMs involves estimating the confidence from the model. Recent research primarily concentrates on methods classified into three types: logit-based, consistency-based and verbalization-based Logit-based estimation (Guo et al., 2017: Cheng et al., 2023) measures the model confidence based on the model predicted logits. Consistency-basse timation (Wang et al., 2023; Kuhn et al., 2023) relies the intuition that LMs will consistently generate similar outputs when they are confident in responding to a query. A major challenge of consistency-basee methods is deciding which is typically accomplished by utilizing a natural language inference model (Kuhn et al., 2023), BERT (Zhang et al., 2020b), QA-based metrics (Fabbri al., 2022). However, these meth- ods are limited to sentence-length generations, and it is still unclear how to decide on semantic equiv- alence over long-form outputs. More recent studies (Tian al., 2023; Xiong et al., 2023) investigate directly prompting instruction- tuned LMs to verbalize their confidence. While consistency- based and verbalization-ert based methods have demonstrated their effectiveness in recent LMs, their utiliza- tion is largely restricted due to their high computational costs during inference and the requirement for LM's ciiructioo-follwiing capability. In contrast, the LITCAB works only adds minimal com- putational cost and is suitable for use with any LM whose last-layer hidden states are accessible.
es-
the semantic equivalence of multiple long outputs,
EVALUATING CALIBRATION FOR GENERATIONS OF VARYING LENGTHS
Model calibration aims to align the model's 's confidence with the actual likelihood of output correct- ness. Therefore, one of the key concerns when assessing calibration is model confidence estimation. For classification tasks, this can be achieved by directly using the class probabilities (Guo et al., 2017: Cheng et al., 2023). For generation tasks, however, the output is a sequence of token proba- bilities, raising the question of how assess the model's confidence in generated sequence, c.g., a sentence or a paragraph. Below we describe our methods for eliciting the model confidence for generations of different lengths, well evaluating the correctness of model generations.
3.1 EVALUATING CALIBRATION FOR GENERATION AT PHRASE- AND SENTENCE LEVEL
When generations are short, e.g., containing one single phrase sentence, we assume each genera- tion focuses on one idea. Therefore, we aggregate the token-level probabilities into one confidence estimation score at the whole sequence level. Let denote the input sequence comprising both the in-context learning demonstrations and the question, and let y denote model generation, which consists of T. tokens. We determine the corresponding model confidence, denoted p(y|zr),
3

---

Published at ICLR 2024
Reeponee
Damis Regpaardtieeeee wan bor on Delcber 2 in Willambbanng Megiits Heaattnneen Wiggrti 1028 whee Slayedl oal won soctial for Eiatees 1he donteed by Cokklndd roadden in pliged
Clam
Conridence Correctress
Camba Hayaard oua Oclobur 1987
Camua Heypoard Bay was born Octaber 1997
0.47
X
attended Ving Tech
Span
illended Vingini
4.16
Carnectens
Poin any 24, 1900 hee
ho played llgg fos for the Hokios
playys oul age Noutaal for HURIES
0.60
Coneert of Uagiand
ous drafted Oukland 2009
round
was drafked Cukkadd Paadera 2009
0,00
paees spol Cola tillangh
played itaiders from 2009 2013
It plyadd Foalders
2013
0.61
he played for Ihe Bullrrore Rlavers bun 2014 2016.
LCB
Iv: Baltenrr Rayerr fron 2014 2016
0.75
X
6.57
The 4-step process for evaluating calibration over paragraph-level generations by break- ing the text down into individual claims and then estimating confidence and judging correctness for each claim separately. Step 1: Individual claims are extracted using GPT-3.5-turts extracted claims mapped back to the corresponding spans in the paragraph. Step 3: The confi- dence of each claim is estimated by aggregating probabilities over tokens in the corresponding span. Step 4: The correctness of each claim is determined by GPT-4 as whether the claim is supported by the retrieved Wikipedia passages.
Figure 2:
Step 2: The
geometric mean over the sequence of token probabilities:
p(y|x) = V I1 pluel x. Bee).
(I)
Following Tian et al. (2023), we employ GPT-4 to measure the correctness by asking it to decide on the semantic equivalence between the model generations and the references. We consider an output to be correct if there is at least one reference that matches the model output.?
3.2 EVALUATING CALIBRATION FOR GENERATION AT PARAGRAPI LEVEL
In real-world scenarios, LM is typically prompted by users to generate long-form text (e.g., one or more paragraphs.) For instance, for a prompt of "Summarize the events of World War Il", good response will need to include multiple claims and facts (henceforth claims for simplicity). In such a case, coarse-grained aggregation of the token probabilities over the whole sequence is unsuitable since confidence in individual claims should be estimated separately. As such, it is crucial to break down the model response into distinct claims, treating each as a separate unit for evaluation. In order to do this, we propose four-step procedure to assess claim-level confidence and determine the accuracy of each claim. We demonstrate the evaluation process on a task of generating biography in Figure 2. This procedure also described below:
Step 1: Claim Extraction. We prompt GPT-3.5-turbo to extract individual claims from the generated text,' following Min et al. (2023).
Step 2: Span Mapping. Each extracted claim takes the form single sentence. de- termine model confidence for each claim using the method in Equation (I), we need to map the claim to the corresponding span in the generated paragraph. To do that, we prompt GPT- 3.5-turbo once again to identify the span.* 4 Since GPT may rephrase the spans, select the sentence in the paragraph yielding the longest common string (L.CS) with the returned span. Step 3: Confidence Estimation. Once gather the generation span that corresponds each claim, we can calculate the claim-level confidence by aggregating the token probabilities of its corresponding generation span, done in Equation (1).
Step 4: Correctness Estimation. Unlike phrase-leve tasks, there is no reference available for evaluating the correctness of claims. Therefore, we follow the retrieval-basee method used by Min et al. (2023) to assess the correctness of claims. We first retrieve relevant passages from
PPrompts for measuring correctness can be found in Appendix A.4. 'The prompts used for extracting individual claims can be found in appendix A.1 *Prompts can he found in Appendix A.2.
4

---

Published at ICLR 2024
Max-Margin Objective
Token Probubilities
Litcab
Litcab
Are vampires real?
LLM
Yes, they are real
1(y)
LLM
LLM
... F yL
Tokens
Yes, they cxest.
No. vamprres
Yis. setrpiios sil resd
Figure 3: Left: The process of constructing positive and negative samples. Right: LITCAB Train- ing. LITCAB adjusts the LM predicted logits of the last layer's hidden states, with parameters trained using a max-margin objective.
Wikipedia for each generated claim through a Generalizable T5-based Retriever (Ni et al., 2022).² We note that the selection of the information retrieval component does not significantly impact the estimation results, which agrees with Min et al. (2023). We then instruct the more powerful GPT-4 model to determine whether each claim is supported by the retrieved passages. Claims with support are considered correct, otherwise incorrect."
4 LITCAB: LIGHTWEIGHT CALIBRATION OF LANGUAGE MODELS
Fine-tuning the LLM to decrease its confidence in incorrect generations and augment its certainty in correct ones has been shown to improve calibration (Jiang al., 2021). Yet, fine-tuning the full models, especially those of large scale, can be compute-intensive and often require more training samples. In this section, we propose LITCAB for lightweight calibration which trains single linear layer over the LM's last hidden layer representations to predict bias term, which is used to modify the model's logits and subsequently alter the generation confidence.
Formally, given an LM generation y. which consists of L tokens, its last layer hidden states given by the LM can be represented as h[z.z] = [h, b2 .... 1.-1, h2]- At each position t, LITCAB uses single linear layer, built on top of model's last layer, to adjust the LM predicted logits y L.M E R by using yt y: J.M + Wh; +b, where W and b are the trainable parameters. V denotes the vocabulary size. After that, the adjusted sentence confidence p(y(æ) can be derived from the geometric mean
of the updated token probabilities p(y[z) VII. p(u ( vee). Here p( vee) is the updated probability of token y/s computed by the softmax function:
exp (vc)
p(u, Ir, vee)
EV (xp)
where y (2s) is the logit corresponding to token in the vector y
Negative Sample Construction and Max-margin Objective. We gather collection of positive and negative samples to train our model for each task. For phrase- and sentence-level tasks where references are available, we use the references directly as positive samples. We then repeatedly sample model generations, treating incorrect ones negative samples. For the paragraph-level tasks, we directly instruct each LM to generate biographies or descriptions for given people's names entities. Subsequently, ones as negative samples. Let r represent the input and represent the positive sample, while [(y) represents the negative samples. The max-margin objective for training the LITCAB is given by L(x, (1) = Eyy) max(0, + p(y (z) p(y (r). Intuitively, the max-margin objective enables the LM to distinguish correct generations from incorrect ones through LITCAB, thereby yielding enhanced calibration of confidence for both correct and incorrect generations. We collect incorrect samples per question for phrase- and sentence- tasks, and use all generated positive and negative samples for paragraph-leve tasks. The process of constructing positive and negative samples and the training process are illustrated in Figure 3.
categorize the correct claims positive samples and the incorrect
'We use gtr-t5-1 5-11 rge from Huggingface.
PPromp for estimating claim correctness listed Appendix A.3.
5

---

Published at ICLR 2024
5 CAT: A BENCHMARK FOR CALIBRATION EVALUATING
Unlike previous studies that have primarily concentrated on multiple-choice QA settings (Jiang et al., 2021; Cheng et al., 2023), we emphasize on general text generation tasks with responses of vary- ing lengths, which present more challenges to assessing LM calibration. Therefore, for evaluation, we collect tasks with lengths of responses at (i) phrase level, (ii) sentence level. or (iii) paragraph level. For phrase-level generation datasets, use NaturalQuestions (NQ), SciQ, and TriviaQA, all of which include short responses (e.g., named entities). For sentence-level responses, we choose complete sentences. For paragraph-leve
TruthfulQA and WikiQA, where model responses generations, we prompt the LMs to write biographies of different figures (celebrities, scientists, etc.), whose names are sourced from BioGen (Min et al., 2023). As these names originally come from Wikipedia, we can easily verify the model-generated claims by comparing them against the corre- sponding Wikipedia passage. We also design a paragraph-Ievel task called WikiGen (new). where LMs are tasked to generate Wiki-style descriptions for entities gathered from the fact verification dataset FEVER (Thorne et al., 2018). Compared with BioGen which is limited to biographies, Wiki- Gen covers a wider spectrum of subjects beyond individuals, including events, concepts, and objects, resulting in more diverse topics. Additionally, we employ QAMPARI (Amouyal et al., 2022), question-answerti
task with multiple answers. While the answers are not long, the model's re- sponse comprises multiple claims and can be easily extracted. Instead of searching for Wikipedia passage to assess correctness, ground truth. The statistics and examples of the benchmark, along with details on the construction of training and test sets, are listed in the Appendix B.
directly evaluate it by comparing the generated claims with the
EXPERIMENTS
6.1 EVALUATION METRICS
We evaluate the model calibration performance using three metries:
Expected Calibration Error (ECE). Following previous studies (Guo et al., 2017; Lin et al., 2022; Tian et al., 2023), we adopt the expected calibration error, which estimates the gap between the model confidence and the actual accuracy. We divide the confidence from 0 to into 10 intervals with a spacing of 0.1, and assign samples to the interval corresponding to their confidence scores. Within each bin bú we compute the accuracy of model generations acc(b.) and the average model confidence (b.). The ECE is then calculated as ECE =
K [bel |ace(b:) conf(b;;), where N denotes the number of model generations. Lower ECE indicates better calibration, reflecting closer alignment between confidence and accuracy.
Disl
Brier Score directly measures the distance between the model confidence and the binary cor- rectness label of the generation. Given all model generations Y, the Brier score is defined as Brier = Eyer (p, I(v))², where I[yy) denotes the correctness label.
Selective Classification Metrics. Following Tian al. (2023), we also evaluate calibration by employing metrics from the selective classification domain, as model confidence plays a piv- otal role in this field. Specifically, we employ the metrics of accuracy at coverage (acc @q) and coverage at accuracy (cov @). Acc@q quantifies the precision of the model by evaluat- ing the accuracy of the top 9 percent of predictions. Conversely, cov@p gauges the extent to which the model achieves recall by identifying the largest percentage, denoted as c, for which the most confident e percent of predictions exhibit accuracy surpassing the threshold p. Com- pared to AUROC (Bradley, 1997) that focuses on measuring the quality of confidence scores, these two metrics directly evaluate the LM's performance filtering out incorrect generations by setting the thresholds.
6.2 BASELINES
We compare LITCAB with two popular calibration methods for neural networks on the widely-used Llama2-7B*, which is one of the most recent open-source LMs:
We choose the 7 version hecause it can be easily fine-tuned using LoRA, enabling comparison hetween LITCAB and rassedg-hased methods.

---

Published at ICLR 2024
Temperature Scaling (Liang et al., 2018): A temperature constant is used to scale the logits before computing the softmax. We employ gradient descent optimization to search for the optimal temperature the training set.
Label Smoothing (Szegedy et al., 2016): As a training-based method, the entire LM fine- tuned training set with label smoothing. To train Llama2-7B, we adopt LoRA (Hu et al., 2022) to enable the fine-tuning process on single GPU.
Additionally, we also compare LITCAB with recent confidence estimation methods that are specifi- cally designed for LMs, including:
Verbalization prompts the LLM to provide its own confidence in a given output. We directly reuse the prompt provided in Tian al. (2023).
P(IK) (Kadavath et al., 2022): A linear layer is stacked on top of the LM last-layer's hidden state that corresponds to the question's last token. The added layer learns to predict whether the model can accurately answer the question. We keep the parameters of the LM fixed and only fine-tune the linear layer.
Self- Cofff-Coniitenccy (Tian et al., 2023; Xiong et al., 2023) relies on the hypothesis that confident responses will appear more frequently when sampling from the model. As applying majority voting directly is not straightforward over long-form generations, we rely on a natural language inference model. Details about this method are listed in Appendix C.
We note that LM confidence estimation methods, i.e. Verbalization, P(IK) and Self cofff-ssisttnny,, can only yield a single unified score for the entire generation, which cannot be used to assess calibration the claim level since each generation normally contains multiple claims. Therefore, we do not list the results for paragraph-level tasks by these comparison methods. To construct training instances for label smoothing and temperature paragraph-Ievel tasks, we employ the same generation process used for constructing training instances for LITCAB, but only keep the accurate claims. Other generation and training details (e.g. hyperparameters) can be found in Appendix C.
6.3 RESULTS COMPARED WITH TRADITIONAL CALIBRATION METHODS
The results of LITCAB and the two commonly used neural network calibration methods are shown in Table 1. As can be seen, LITCAB consistently improves over the original LM's calibration perfor- mance across all tasks, highlighting the effectiveness of LITCAB. Using the NQ dataset, a dataset of human-originated queries via Google search, 0.171 to 001. Furthermore, when compared to the two baselines, LITCAB achieves the best cali- bration performance, measured by the lowest average ECE (0.093) and Brier score (0.197) across all tasks. Moreover, LITCAB further enhances the model's calibration performance on phrase- and sentence-level tasks when combined with temperature scaling, suggesting its compatibility with other calibration methods. However, the combination of LITCAB with temperature scaling leads to poorer calibration performance on paragraph-level tasks. This occurs because the temperature ad- justed based on the model output samples due to the absence of references, which further intensifies overconfidence.
example, LITCAB reduces ECE by 41%, from
Note that the fine-tuning of the Llama2-7B model with label smoothing does not yield satisfactory calibration performance, possibly due to the small size of training data that leads to model overfit- ting. In contrast, LITCAB can leverage small training sets for improved calibration, suggesting that LITCAB achieves superior data efficiency compared to label smoothing.
6.4 RESULTS COMPARED WITH LM CONFIDENCE ESTIMATION METHODS
As shown in Table 1, LITCAB outperforms all LM confidence estimation methods in phrase- and paragraph-level tasks and attains the lowest average ECE and Brier score, again showcasing its superior effectiveness calibrating LM. Interestingly, we can observe that the model's confidence estimated by verbalization is worse than that of the original model for most tasks. We speculate that this could be attributed to the instruction-followwe 7B lacks. Despite the specific instances where self-consistency surpasses LITCAB in tasks like NQ and TruthfulQA, LITCAB consistently demonstrates competitive performance across a wide range of tasks, measured by its lower averaged ECE (0. (0.125 0.084) and Brier score (0.216 0.195) on
skills that verbalization requires, which Llama2-

---

Published at ICLR 2024
Metric
Original
Traditional Calibration Mathods
Smoothhng
Temp. Scaling
Contidence Fxiimation Sell-Consklenyy
Verhalization
Tomp. Scaling
LIrCAn
aced
0.208
0.288
0.340
NQ
cow S750
ECE
0.171
0.115
0.186
0.061
0.365
0.115
0.158
0.000
0.055
0.516
0.145
0.217
0.105
0.101
(L083
0.105
Brier
0.196
0.212
0.193
0.204
0.468
0,163
0.169
0.164
accdd 50
0,764
0.212
0,764
0.656
0.668
0,744
0,762
0.762
S
BCE
9190
0,094
0.211
0.003
0,991
0.211
0.004
0.117
0.101
0,124
0.221
0,082
0.231
0.302
0.204
0,500
0.446
TrivsQQA
0.181
0.079
Brier
0.203
0.195
0.277
0.409
0.253
Sentence Level
acc
0.314
0.181
0.514
0.207
0.233
0.405
0:114
0.314
TruhfulQA
oow 51-0
0.136
0.000
0.176
0.005
0.224
0,500
0. 1.195
1.195
ECE
0.138
0.134
0.161
0.323
0,510
0.060
0.108
0.108
Brier
0.218
0.175
0.240
0.340
0.474
0.194
0.306
0.208
0.368
0.275
0.388
0.339
0.572
0.628
0.997
0.397
WIKIQA
ECB
0.075
0.012
0.155
0
0.004
0.136
0.621
0.074
01062
0.243
Paragraph
ace
0.347
BioGen
low
0.065
0.059
0.066
0.148
ECR
0.169
0.196
0.346
0.166
0.244
Drier
0.269
0.284
0,313
0.267
0.308
a2012 50
0.876
0.860
0.876
0.872
0.872
Wikifion
ECE
0.045
0,745
0.075
0.733
01.745
0.049
0.837
0.756
0.065
0.756
0,172
0.174
acedd
0.195
0.193
0.207
30
0.260
0.260
0:257
QAMPARI
Brier
0.228
0.208
0.273
0.157
Average
acedl 50
BCE
0.137
0.459
0.191
0.319
0.145
0.459
0.093
0.461
0.104
0.461
0,213
0.226
01:009
Table I: Results of LITCAB and baselines on CAT, with the best score per metric per dataset in bold. We highlight numbers where LITCAB improves over both the original LM and all baselines in blue when LiTCAB outperforms the original LM, it is colored in green The row shows the average metric value over all tasks. LITCAB effectively calibrates the LM on various tasks. The combination of LITCAB and Temp. Scaling further enhances the LM's calibration performance phrase- and sentence-level tasks.
phrase- and sentence-level tasks. This underscores the versatility and effectiveness of our approach, making it a promising choice for various tasks. Additionally, considering that the computational cost of colfisensist during inference is N times" higher than that of LITCAB, our approach computationally more effective.
6.5 INVESTIGATING CALIBRATION OF LMS USING CAT
Here on CAT we assess a broad range of popular open-source LMs, with sizes ranging from 1.5B 30B. Specifically, we evaluate GPT2-XL. (1.5B) (Radford et al., 2019), GPT-J (6B) (Wang & Komatsuzaki, 2021), LLaMA families (7B, 13B, and 30B) (Touvron et al., 2023a), Llama2 families (7B and 13B) (Touvron et al., 2023b), and Vicuna-v1.3-1333 (Chiang et al., 2023). The results are illustrated in Figure 4,9 From the figure, we can readily draw the following conclusions:
Larger models within the same family generally exhibit better calibration on short model re- sponses, but not necessarily for longer ones. Within the models trained using the same data, ic.. LLaMA families and Llama2 families, the large ones of LLaMA-30B and Llama2-13B can achieve not only improved task performance (acc@q, cov @p) but also superior calibration (ECE, Brier) that their smaller counterparts on phrase-level tasks. This echoes the findings in Kadavath al. (2022). However, LLaMA-30B does not demonstrate better calibration (ECE, Brier) in tasks
N denotes the number of model predictions in self-consistency, which is set as 10 in our setting. Detailed results he found in Appendix
8

---

Published at ICLR 2024
Phrase Level
accg30 Va for Difierent Models
ECE Values Offirent Modals
Brier Values for Differnnt Models
0.--
0.00
0.20
2 10
0:15
0.05
Sentence Level
6
8
0:06
Paragraph Level
0.20
124
320
0.18
148
218
Lattt
Unman
Liman Li
0rD-4/2.00
Laaee
Liatte
Limmat-7
Kil
Uieaa
Models
Figure 4: Bar charts of averaged acc 50, ECE, and Brier score of popular LMs computed on CAT. The results for GPT-2 XL in paragraph- tasks are missing due to the prompt's 's length exceeding its context limit. Bars with the same color represent models from the same model family.
involving longer generations (sentence- and paragraph-Ievel tasks). This suggests that the model scaling effect may not hold true for calibration in tasks with longer response lengths.
GPT2-XI. is better calibrated than other larger models. Despite its smaller model size, GPT2- XI. exhibits strong calibration, as evinced by its consistently lower ECE and Brier scores com- parison to the larger LMs, including the largest model, ILLMAA30B. However, note that LLaMA models tend to yield better accuracy, as measured by higher acc @q and cov cov@p scores. We observe that overconfidence infrequently observed by both GPT2-XL and GPT-J, especially the incor- rect generations, thus leading to better calibration. These observations suggest that model accuracy and calibration should not be treated separate objectives to optimize. Given their interconnected- ness, improving them concurrently seems to be far more favorable and efficient approach in future research undertakings.
Vicuna, fine-tuned from LLaMA, gains worse calibration than LLaMA. Vicuna- 13B. which fine-tuned from LLaMA-13B on user-shared conversations, exhibits a much worse calibration. This is highlighted by the increased average ECE and Brier scores. This suggests caution when implementing a second stage of fine-tuning, as it could potentially diminish calibration.
7 CONCLUSION
We presented LITCAB, a lightweight calibration technique for LMs. LITCAB calibrates LMs by predicting a bias term added to the model's predicted logits. An empirical comparison with post- processing, training- verbalization-, and consistency-bar LITCAB and underscores its computational efficiency, considering that it introduces less than 2% of additional parameters. Moreover, we collected CAT, a benchmark specifically designed to evaluate calibration in text generation settings with both short- and long- forg-form responses. We propose an eval- uation methodology dedicated to examining calibration in long-fore generations, which can serve as the basis for future studies. Finally, we use CAT to comprehensively assess seven open-source LMs. Our findings show that larger models within the same family generally exhibit superior calibration phrase-level tasks, but not necessarily tasks at sentence and paragraph levels. Interestingly, despite a lower accuracy. EPL2-XL (1.5B) is better calibrated than larger models. Our observations also reveal that additional fine-tuning may lead to worse calibration.
approaches substantiates the efficacy of

---

Published at ICLR 2024
This work is supported in part by National Science Foundation through grant IIS-2046016 and LG Al Research. We also thank ICLR reviewers for their helpful comments.
Acknowledgments.
REFERENCES
Samuel Joseph Amouyal, Ohad Rubin, Oni Yoran, Tomer Wolfson, Jonathan Herzig, and Jonathan Berant. QAMPARI: An open-domain question answering benchmark for questions with many answers from multiple paragraphs. CoRR, abs/2205. 2022. doi: 10.48550/ARXIV.2205.. 12665. URL https: /doi. woo.orgg///... 4 48550/ /00/55//xxX0 arxiv 2205. 12665.
Andrew P. Bradley. The use of the area under the ROC curve in the evaluation of machine learning algorithms. Pattern Recognit,, 30(7): 1997. doi: URL https:
100 00.0011//00011 3203 (96) 00142-2.
Tom B. Brown, Benjamin Mann, Nick Ryder. Melanic Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert-Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, lla Sutskever, and Dario Amodei. Language models are few-shot learners. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tice Lin (eds.), Advances in Neural Information Processing Systems 33: Anmual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL https: http:////occeeeeneeeeeeeee neurg..n.eeirpe. iarrips. ps. hac//aper/Da 2020//220//00 / 1 4.577dd6fcb44 96741 86 fww.ff8811142 ca 42 Abst html.
Silei Cheng, Zhe Gan, Zhengyuan Yang, Shuohang Wang, Jianfeng Wang, Jordan Boyd-Graber, and Lijuan Wang. Prompting gpt-3 to be reliable. In International Conference Learning Represen- tations (ICLR 23), May 2023. URL https: / /wwww www.:mmiccooooooooomoomoo cwm/ comm/nn-se us- research/ iroolicatio/ppromppeee
gpt 3-to-bo rew-bbe-eelbee......
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https:
/
Alexander R. Fabbri, Chien-Sheng Wu, Wenhao Liu, and Caiming Xiong. Qafacteval: Improved qa-based factual consistency evaluation for summarization. In Marine Carpuat, Marie-Catherine de Marneffe, and Iván Vladimir Meza Ruíz (eds.), Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2022, Seattle, WA, United States, July 10-15, 2022, pp. 2587-2601. As- sociation for Computational Linguisties, 2022. doi: 00.18653/v1/002..1 wwwcl-aainnnnnaa 187. URL https: / //w.i//ooii/// oooiiiorg/1000 wtt.o/////.111//////0000001 111638611/2000
nwac maaacl-maiin 187.
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. GPTQ: accurate post-training quantization for generative pre-trained transformers. CoRR, abs/2210.17325,, 2022. doi: 48550/azXiw.2210.11733 URL https::///// www.oooorg......o wtt.l//.////..../......
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning. ICML 2017, Sydney, NSW, Australia, 6- 6-11 August 2017, vol- ume 70 of Proceedings of Muchine Learning Research, pp. 1321-1330. PMLR, 2017. URL. http:/ / http://pooeeeeceeeee ngs...dnnng.....mo ngs .l mll..press/ve v 7a html.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. The Tenth Inter- national Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. URL https: httpttss///peeeeiiiieee opw.revie.eeeeeeeeeeeeee.......m net. don.um/iineeeeeeerr
Zhengbao Jiang, Jun Araki, Haibo Ding, and Graham Neubig. How can we know When language models know? the calibration of language models for question answering. Trans. Assoc.
10

---

Published at ICLR 2024
Comput, Linguisties, 9:962-977, 2021. doi: 10 -00000077 URL https://doi. htg//0.1162/taan a_00477.
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Doder Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny Hernandez, Josh Jacobson, Jackson Kernion, Shauna Kravec, Liane Lovitt, Kamal Ndousse, Catherine Olsson, Sam Ringer, Dario Amodei, Tom Brown, Jack Clark, Nicholas Joseph, Ben Mann, Sam McCandlish, Chris Olah, and Jared Kaplan. Language models (mostly) know what they know. CoRR, abs/2207.05222 2022. doi: 10.48550/arXiv.2207 .05221. htts::::: www.doo.......gooooooooooo...e.. 4e0pg//.....5.0000 48550/ arxiv 2207. 05221.
Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. In The Eleventh International Confer- ence on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https: /owpr//eeeeeeeeeeeeeeeeeeeeeeeee net net/ podffidd---AAA---
Rémi Lebret, David Grangier, and Michael Auli. Generating text from structured data with appli- cation to the biography domain. CoRR, abs/1603.07771, 2016. URL http:://axxiiiii
abs/1603. 16t..111..7777.
Shiyu Liang, Yixuan Li, and R. Srikant. Enhancing the reliability of out-of-distributiin image de- tection in neural networks. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenRe- view.net, 2018. URL https:://///./...... /ppneeeeiivi eww net.
TickIxxR x
Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint vrXiv:2109.09955 2021.
Stephanie Lin, Jacob Hilton, and Owain Evans. Teaching models to express their uncertainty in words. Trans. Mach. Learn. Res., 2022, 2022. URL https: httpr///e/eeeeee..eettt.e. wtt.orrummee/eeommmm forum?
1d=8s8K2UZGTZ.
Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Wei Koh, Mohit lyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. Factscore: Fine-grained atomic evaluation of factual precision in long form text generation. CoRR, abs/2305. 2023. doi: 10.48550/axXiv23305. 14251. URL https::////. /oii. wrgglll../.oooooooooooooooo.o..o 005448550///1 www.///rrvvvvv.. 2300.14255
Khanh Nguyen and Brendan O'Connor. Posterior calibration and exploratory analysis for natural language processing models. In Lluís Mårquez, Chris Callison-Burch Jian Su, Daniele Pighin, and Yuval Marton (eds.), Proceedings of the 2015 Conference Empirical Methods in Natu- ral Language Processing, EMNLP 2015, Lisbon, Portugal, September 17-21, 2015, pp. 1587- 1598. The Association for Computational Linguistics, 2015. doi: 10.18653/w1/d15-1113 URL https: //l:://doo /oii .org/r//g... /trg/1/ 11. 1el.18653/// /v13/vll. /d1 5-1182.
Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hernández Ábrego, Ji Ma, Vincent Y. Zhao, Yi Luan, Keith B. Hall, Ming- Wei Chang, and Yinfei Yang. Large dual encoders are generalizable retrievers. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pp. 9844-9851 9855. Association for Computational 669. URL https://dol org/10.
Linguistics, 2022. doi: 10. 10.18653/w1/2022.mmm//. 18653/v1/2022 wwweeennmmmmeeeeee ma maii.. 669.
Alexandru Niculescu-Mizil and Rich Caruana. Predicting good probabilities with supervised learning. In Luc De Raedt and Stefan Wrobel (eds.), Machine Learning, Proceedings of the Tivent-Secoond International Conference (ICML 2005), Bonn, Germany, August 7-11, 2005, vol- ume 119 of ACM International Conference Proceeding Series, pp. 625-632, ACM, 2005. doi: 10. 10.1145/1 141351/112235511444433 URL https:
1ttp///10..11111111311..11010000 11024430.
OpenAl. GPT-4 technical report. CoRR, abs/2303.08774, 2023. doi: 10.48550/aXii.2203.07744 URL https::tp/// /ooi .ariiorgg/11 0ee.//....5555111

---

Published at ICLR 2024
Gabriel Pereyra, George Tucker, Jan Chorowski, Lukasz Kaiser, and Geoffrey E. Hinton. Regu- larizing neural networks by penalizing confident output distributions. In 5th International Con- ference Learning Representations, ICLR 2017. Toulon, France, April 24-26, 2017, Workshop Track Proceedings. OpenReview.net, 2017. BRRLhttps:/ /ttp::///ppeeeiiieeeee wwwreeiiie....... ew net. forum? i=d= HyhbYrGYe.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Dlya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jonathon Shlens, and Zbigniew Wojna. Re- thinking the inception architecture for computer vision. In 2016 IEEE Conference on Com- puter Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 2818-2826. IEEE Computer Society, 2016. doi: 10.1109/CVR..206.3300 URL https: //ddi. orr/ 10. 09/CVPR. 20166300
James Thorne, Andreas Vlachos, Christos Christodoollpooulos, and Arpit Mittal. FEVER: large-scale dataset for fact extraction and VERification. In Proceedings of the 2018 Confer- ence of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, Volume / (Long Papers), pp. 809-819, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10.18653/v1/N18-100 URL httpp::/aclaattaeeit http//aaaaanttoooaccoeeea
Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher D Manning. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback. arXiv preprint 3.Xi::2305.1497.0 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Ar- mand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. CoRR, 0ts//302.11371,, 2023a. doi: 10.48550VarXi.2200..333 13971. URL https://don http//ii..rgg/...1 www.o0.../... 0111//10.85550/ 48550/ 23022 13971.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee. Diana Liskovich, Yinghai Lu. Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, lliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine- tuned chat models. CoRR, abs/2307.092288 2023b. doi: 10.48550/axxiv223377. 092888. URL https:/ /oii. www.oo.............o..oo........ 48550/
09288.
Ben Wang and Aran Komatsuzaki. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. https:://// / thub. www.:ingool.o.cccc/ooooocc z/ htt////sshtttee weeeshhttrasffooo warransformer-jje May 2021.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023, URL httph:ttp///ccmm / net/ DdfFidd1PLINIMME.
Jonathan Wenger, Hedvig Kjellström, and Rudolph Triebel. con-parametric calibration for classifi- cation. In International Conference on Artificial Intelligence and Statistics, pp. 178-190. PMLR, 2020.
12

---

Published at ICLR 2024
Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li. Jie Fu, Junxian He, and Bryan Hooi. Can llms express their uncertainty? an empirical evaluation confidence elicitation in llms. arXiv preprint arXiv:2306.13063, 2023.
Jize Zhang, Bhavya Kailkhura, and Thomas Yong-Jin Han. Mix-n-match Ensemble and com- positional methods for uncertainty calibration in deep learning. In Proceedings of the 37th In- ternational Conference on Machine Learning, ICML. 2020, 13-18 July 2020, Virtual Event, vol- ume 119 of Proceedings of Machine Learning Research, pp. 11117-11128, PMLR, 2020a. URL http:/ http://pocceedddeeee www.m malr.press/vlin 2/whgg22222g00 0k. html.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. Bertscore: Eval- uating text generation with BERT. In 8th International Conference on Learning Representa- tions, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020b. URL https: https://opeereeiee net htwprr/eweeeeeett.oooooioeeeeeee =or.urii---sseeeeevVr
Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, al. Siren's song in the ai ocean: A survey hallucination in large language models. arXiv preprini urXiv:2309.01219, 2023.
A PROMPTS
A.I PROMPT FOR BREAKING DOWN PARAGRAPHIS
Please break down the following paragraph into independent facts. You should ONLY present the independent facts (one in a row), no other words explanation.
-> [Biography]
A.2 PROMPT FOR MAPPING CLAIMS WITH SENTENCE SEGMENTS
which segment in the following paragraph reflects the claim [claim]] "? The segment doesn't need to be complete sentence and should be short as possible. (Biography]
A.3 PROMPT FOR ASSESSING CORRECTNESS OF CLAIMS
Answer the question about [Name] based on the given context.
Title:
[Passage Title]
Text:
[Passage]
Input:
[claim] True or False?
Output:
A.4 PROMPT FOR ASSESSING CORRECTNESS OF MODEL GENERATION
Are the following two answers my question "[Question]" semantically equivalent? (Answer "Yes" or "No" first, and then explain your answer.)
1. [Reference]
2. [Mode] Response]
13

---

Published at ICLR 2024
NQ
SeiQ
TrivinQA
TruthfulQ4
WIRIQA
BioGoon
WikiGen
QAMPARI
0 TTain
2K
2K
2K
397
1040
500
5000
2K
Test
IK
IK
IK
420
293
183
100
IK
Avg. Response
Mox. # Ressonne tnkens
12
42
24
106
318
197
458
Min. Respoese tokens
1
21
33
Table 2: Statistics of tasks in CAT. The number response tokens for phrase- and sentence-level tasks is estimated based references. For BioGen and WikiGen, the number of response tokens is calculated based on the generations of I Llama2-7b, due to the lack of references.
Task
Example
Phrase Level
Question: what is the smallest prime number that is greater than 30
NQ
Response: 31
Question: About how tall can mid-ocean ridges be?
SciQ
Response: about km
Question: In which decade did sterco records first go on sale?
TriviaQA
Response: 1930's
Sentence Level
Question: Which language do all people Europe speak?
TruthfulQA
Response: There's 's no single language that all Europeans speak
Question: bow much is I tablespoon of water
WikiQA
Response: Australia one tablespoon (measurement unit) is 20 ml..
Puragraph Level
Question: Write a paragraph for Bill Tobin's biography.
BioGien
Response: Ron Meagher (bom October 2. 1941. Oakland. California. USA) is best known us the bassist of the American rock band The Beau Brummels, When guituriss- songwriter Ron Elliott was putting the...
Question: Write a paragraph about The Beatles.
WikiGen
Response: The Beutles were English rock band formed in Liverpool in 1960, comprising John Lennon, Paul McCartney, George Harrison, and Ringo Starr They are regarded as the most influential band
time...
Question: What fictional character had their debut in Mega Man X?
QAMIPARI
Response: Hlaee Mammoth; Spark Mandrill; Launch Octopus; Chill Penguin; Sigma: Storm Eagle; Zero; Boomer Kuwanger; Sting Chamelcon; Armored Ar- madillo
Table Examples of question response pairs in the benchmark.
B DETAILS OF CAT
We list the statistics of tasks from CAT in Table 2 and show some examples of CAT in Table 3. For phrase- and sentence-level tasks, we randomly select 1K samples as test set (if the original test data size exceeds IK) and 2K samples for training (if the original training data has more than 2K samples). Given that there is no official training set for TruthfulQA, we randomly select 397 instances from the original test set for training and use the rest for testing in our experiments. For BioGen, we collect 683 people's names provided by Min et al. (2023) in total. Of these names, 183 utilized for evaluation purposes, while the remaining 500 employed to generate both correct and incorrect claims for training LITCAR. Similarly, for the WikiGen task, we randomly select 600 entities from the FEVER dataset, each linking to a specific Wikipedia passage. Among these entities, 100 are designated for evaluation, while the remaining 500 are used for training LITCAB. Regarding QAMPARI, we randomly selected 1K samples for testing and 2K samples for training.
C IMPLEMENTATION DETAILS
To ensure consistency across tasks, we use in-context learning since not all LMs exhibit good zero- shot capabilities. We use 15 demonstrations for NQ, SciQ. TriviaQA, TruthfulQA, and WikiQA, and 5 for BioGen and WikiGen because their demonstrations we select biographies from WikiBio (Lebret et al., 2016). For WikiGen, we manually gather demonstrations by extracting the initial paragraphs from Wikipedia passages. For the remaining tasks, random sampling from the training set used. The query for BioGen formed as "Write paragraph for [Name] 's biography", and for WikiGen, it is "Write paragraph about [Entity]""
much longer. In the case of BioGen,
14

---

Published at ICLR 2024
GPT2-X121.58) GPT-](6)
L.MMA-7B LLaMA-138
LLaMA-30R
Llama2-7B
1Jama2-138
Vicuna-138
Phruse Level
NQ
cove550
0.001
0.057
0.271
0.347
0.445
0.115
0.407
0.113
ECE
0,045
0.059
0.144
0.123
0.169
0.171
0.139
0.204
wce 50
0.258
0.620
0.756
0.796
0.874
0.764
0.844
0.678
0.361
0.277
0.142
ECE
0.059
0.133
0.126
0.117
0.107
0,094
0.102
0.214
Brier
0.137
0.209
0.210
0.306
0.186
0.203
0.197
0.318
TriviaQA
cov 80.00
0.000
0.128
0.029
0.426
0.156
0.111
0.169
0.266
ECE
0.063
0,133
0.052
0.112
0.087
0.186
0.213
0.203
0.190
Sentence Level
TruthfuQQA
cov
0.005
0.040
0.117
0.331
0.648
0.136
0.350
0.998
ECE
0.041
0.120
0,121
0,110
0.158
0.132
0.200
0.223
0.218
Mce 50
0.099
0.240
0.322
0.347
0.264
0.388
0.455
0.421
WikIQA
0.358
ECE
0.063
0.045
0.108
0.114
0,142
0.075
0.064
0.211
Brier
0.125
0.149
0.190
0.223
0.182
0.212
0.192
0 0.304
Paragraph
acedd 50
0.228
0.230
0.275
0.313
0.347
0.485
0.380
BioCien
65:0
0.250
0.066
0.999
ECE
0.159
0.128
0.105
0.169
0.114
0.229
Brier
0.182
0.153
0.162
0.173
0.269
0.260
0.255
WikiGen
covir30
0.001
0.220
0.400
0.354
0.745
0.914
0.675
ECE
0.102
0.102
0,124
0.165
0.045
0.048
0.168
0.236
wce 50
0.115
0.229
0.293
0.267
0.193
0.253
0.305
0L580
0.501
0.260
ECE Brier
0.372
0.257
0.291
0.268
0.290
0.323
0.268
0.306
0.225
0.250
0.230
0.228
0.265
0.230
Table 4: Different calibration metrics of popular LMs computed over CAT.
Please note that we use LITCAB exclusively to evaluate the confidence of predictions made by the original LM. LITCAB does not affect the generation process of the LM.
For searching the optimal hyperparameter of LITCAB and comparison methods, use 20% of the training samples for validation and train LITCAB on the remaining samples. We use a training batch size of 128 and learning rate of le-5, We train LITCAB for epochs with early stopping. To prevent excessive adjustment of the LM's predicted logits, we initialize LITCAB's 's weights to zero. All LMS run on a single GPU with 48GB. We use fp16 for 13B-sized models. For models with more apply quantization technique, called GPT-Q algorithm (Frantar
than 13 billion parameters, et al., 2022), it also enables mixed precision of int4 and float16 during inference. To implement the over-smoothing comparison method, train the entire LLM, set the LoRA rank to with learning rate of 3e-4, utilize cross-entropy loss with the Sabol_smoothing attribute set to 0.1, and train the model for 10 epochs. For the self-consistency method, we consider model generations as semantically equivalent if they entail each other. We sample 10 generations for each question and use the largest cluster of semantically equivalent generations estimate the LM confidence.
D RESULTS OF LMS ON CAT
The detailed results of LMs on CAT can found Table
15


Reviews:
[{"review_id": "qdkpyyrXWe", "reviewer": "Reviewer_7atx", "summary": "The paper proposes a new calibration technique for LLMs: LITCAB, a lightweight calibration mechanism that adjusts the generation confidence of large language models (LLMs) by adding and training a single linear layer over the last hidden states of the LLM. LITCAB aims to improve the alignment between the model confidence and the output correctness and reduce the issue of hallucination in LLMs. The authors construct CAT, a calibration evaluation benchmark consisting of six open-ended question answering (QA) tasks that cover answers ranging from phrases to paragraphs. The authors propose a four-step procedure to assess the calibration and correctness of paragraph-level generations, which involves extracting individual claims, mapping them to spans, estimating their confidence and verifying their accuracy using GPT-4 and Wikipedia passages. The authors conduct experiments on LITCAB and several baselines using Llama2-7B and other popular open-source LLMs. They find that LITCAB consistently improves the calibration performance across all tasks, and that larger models within the same family tend to exhibit better calibration. They also observe that fine-tuning may lead to worse calibration in some cases.", "questions": "- How did you select/generate incorrect answers for the QA tasks?\n- The confidence of a sentence is the geometric mean of all its tokens. However, different tokens contribute differently to semantics. For instance, think about \"The 2023 Oscar best actress is Michelle Yeoh\" as the answer to \"Who wins the 2023 Oscar best actress?\". Only the last two tokens decide the correctness of the sentence. How did you handle the different importance of tokens?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Originality: The authors propose LITCAB, a lightweight calibration mechanism that only adds and trains a single linear layer on top of the LLMs12. The authors also construct CAT, a new benchmark for evaluating calibration across different answer lengths.\n- Quality: The paper compares several baselines of calibration on datasets of different output length - phrase, sentence, and paragraph.\n- Clarity: The paper is easy to follow and understand.\n- Significance: The paper addresses the calibration problem for LLM and improves upon SOTA open-sourced LLMs on open-ended QA tasks.", "weaknesses": "- It is not clear to me how to select the incorrect answers. This could be important as different selection strategies may affect the performance of the final model.\n- It is not clear why adding a linear layer on the last hidden state is the best option. Other options include LoRA or prefix tunning, which also only requires a fraction of parameters to tune. More evaluation results should be presented to justify the model choice of LITCAB.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pFwPY0J91l", "reviewer": "Reviewer_Y9cb", "summary": "The paper comprises two main contributions focusing on language model calibration. Firstly, they propose Litcab, which is a lightweight LM calibration mechanism that accepts the sentence representation as input and predicts a bias term that will be added to the output logits. Secondly, they construct CaT, which is a benchmark consisting of six open-ended question-answering tasks covering text lengths of different granularity. The paper presents experiments showing that Litcub is an effective calibration mechanism, as well as a comprehensive evaluation of LLMs in terms of how well-calibrated they are using the CaT benchmark.", "questions": "* How would the use of Litcab fare on arithmetic and commonsense reasoning benchmarks?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "* A lightweight alternative to supervised model calibration is important given that our models are increasing in size.\n\n* Calibration benchmarks for long-form responses have not yet been introduced before.", "weaknesses": "* Presentation-wise, parts of the paper have coherence issues because it is convoluted with two different ideas that are not properly tied up together. The paper starts to be read as having Litcab as the major contribution and curating the CaT benchmark, a minor contribution, to evaluate Litcab. However, the final page (Section 6.6 and Table 3) is dedicated to using CaT to benchmark how well-calibrated the LLMs are. Litcab is not used in this section, which makes it somewhat irrelevant to answering the main hypothesis of the paper.\n\n* The CaT benchmark has two major shortcomings. Firstly, the motivation for creating CaT is to be able to evaluate calibration for long-form responses. However, CaT consists of five datasets with short-form responses and only one with long-form responses. There are a couple of datasets that could have been included, such as ELI5, WebGPT, ASQA, and QAMPARI (which is not passage-level but contains multiple claims). Secondly, it is important for a proposed benchmark to have both proposed (a) baselines and (b) evaluation metrics. However, the proposed baselines could not be evaluated in all cases (specifically passage-level) using their metrics. This makes it difficult to assess the calibration capability of Litcab at the passage level.\n\n* I think it would help the paper if there were extrinsic evaluations to support Litcab's usage in a real-world task setup. Self-consistency is now commonly used because the original paper showed promising results on arithmetic and commonsense reasoning benchmarks. How would the use of Litcab fare on those datasets?\n\n* The paper mentions that Litcab has the \"advantage of requiring a reduced number of data samples for training, leading to enhanced computational and data efficiency\". There is no evidence shown in the paper that supports this statement.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "O0udQPqDfu", "reviewer": "Reviewer_vYMP", "summary": "This paper present LITCAB, a lightweight calibration mechanism to calibrate the LLMs by only adding <2% of the original model parameters. This paper also introduces a new collection of datasets for assessing calibration of LLMs in phrase-, sentence-, and paragraph-level. Experimental results show that the proposed method outperforms a series of baselines by using four metrics. Other key findings include: (i) Larger models within the same family exhibit better calibration. (ii) GPT-family models show superior calibration compared to LLaMA, Llama-2 and Vicuna models despite having much fewer parameters. (iii) Fine-tuning pretrained model (e.g., LLaMA) with samples of focused purpose (e.g., conversations) may lead to worse calibration, highlighting the importance of fine-tuning setups.", "questions": "1. It appears that your method involves using GPT-4 to generate training data, which has also been widely used in recent research. Does this introduce potential unfairness in the comparison? If the training data cannot be generated using GPT-4 or if its quality is poor, would the method still be effective?\n\n2. The proposed method seems to have minimal differences compared to the original LM, and the experimental results also indicate that the original LM performs not very badly. Therefore, what are the main advantages of the proposed method? How does it improve LLMs qualitatively or quantitatively?\n\n3. Why do you use different thresholds (e.g., cov@50, cov@90, cov@60) for different datasets?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The model calibration is highly meaningful, especially for nowadays LLMs, and the proposed method can be used for most of the open-source models.\n\n2. The proposed method is simple yet effective, although it needs some training examples collected by ChatGPT and GPT-4.\n\n3. The collected datasets CAT can also be useful for evaluating the methods for LLMs calibration.", "weaknesses": "1. The paper lacks discussion and comparison of some related works [1][2].\n\n- [1] Inference-Time Intervention: Eliciting Truthful Answers from a Language Model. arXiv 2023.\n\n- [2] Eliciting Latent Predictions from Transformers with the Tuned Lens. arXiv 2023.\n\n2. There are still some efficient fine-tuning methods (e.g. LoRA) that can leverage additional training data, but the paper lacks results compared with such methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)