LACIE: Listener-Aware Finetuning for Calibration in Large Language Models
Authors: Elias Stengel-Eskin
Peter Hase
Mohit Bansal
Venue: NeurIPS 2024 poster (2024)
Keywords: LLM calibration
uncertainty
question answering
pragmatics
listener-speaker model
LLM confidence estimation
Primary area: natural_language_processing

Abstract: When answering questions, large language models (LLMs) can convey not only an answer to the question, but a level of confidence about the answer being correct. This includes explicit markers of confidence (e.g. giving a numeric confidence score) as well as implicit markers, like using an authoritative tone or elaborating with additional knowledge of a subject. For LLMs to be trustworthy sources of knowledge, the confidence they convey should match their actual expertise on a topic; however, this is currently not the case, with most models tending towards overconfidence. To calibrate both implicit and explicit confidence markers, we introduce a pragmatic, listener-aware finetuning method (LACIE) that directly models the listener, considering not only whether an answer is right, but whether it will be accepted by a listener. Specifically, we cast calibration as a preference optimization problem, creating data via a two-agent speaker-listener game, where a speaker model’s outputs are judged by a simulated listener. We then finetune three different LLMs (Mistral-7B, Llama3-8B, Llama3-70B) with LACIE, and show that the models resulting from this multi-agent optimization are better calibrated on TriviaQA with respect to a simulated listener. Crucially, these trends transfer to human listeners, helping them correctly predict model correctness: we conduct a human evaluation where annotators accept or reject an LLM’s answers to trivia questions, finding that training with LACIE results in 47% fewer incorrect answers being accepted while maintaining the same level of acceptance for correct answers. Furthermore, LACIE generalizes to another dataset, resulting in a large increase in truthfulness on TruthfulQA when trained on TriviaQA. Our analysis indicates that LACIE leads to a better separation in confidence between correct and incorrect examples. Qualitatively, we find that a LACIE-trained model hedges more when uncertain and adopts implicit cues to signal certainty when it is correct, such as using an authoritative tone or including details. Finally, finetuning with our listener- aware method leads to an emergent increase in model abstention (e.g. saying “I don’t know”) for answers that are likely to be wrong, trading recall for precision.

LACIE: Listener-Aware Finetuning for Confidence Calibration in Large Language Models
Elias Stengel-Eskin Peter Hase
Mohit Bansal
UNC Chapel Hill
Abstract
When answering questions, large language models (LLMs) can convey not only an answer the question, but level of confidence about the answer being correct. This includes explicit markers of confidence (e.g. giving a numeric confidence score) as well as implicit markers, like using authoritative tone or elaborating with additional knowledge of a subject. For LLMs to be trustworthy sources of knowledge, the confidence they convey should match their actual expertise on a topic; however, this is currently not the case, with most models tending towards overconfidence. To calibrate both implicit and explicit confidence markers, we introduce a pragmatic, listener-aware finetuning method (LACIE) that directly models the listener, considering not only whether an answer is right, but whether it will be accepted by listener. Specifically, we cast calibration optimization problem, creating data via a two-agent speaker- game, where a speaker model's outputs judged by simulated listener. We then finetune three different LLMs (Mistral-7B. Llama3-8B, Llama3-70B) with LACIE, and show that the models resulting from this multi-agent optimization are better calibrated on TriviaQA with respect to a simulated listener. Crucially, these trends transfer to human listeners, helping them correctly predict model correctness: we conduct a human evaluation where annotators accept or reject an LLM's s answers to trivia questions, finding that training with LACIE results in 47% fewer incorrect answers being accepted while maintaining the same level of acceptance for correct answers. Furthermore, LACIE generalizes another dataset, resulting in large increase in truthfulness on TruthfulQA when trained on TriviaQA. Our analysis indicates that LACIE leads to a better separation in confidence between correct and incorrect examples. Qualitatively. we find that a LACIE-trained model hedges more when uncertain and adopts implicit cues signal certainty when is correct, such using an authoritative tone or including details. Finally, finetuning with our listener- aware method leads an emergent increase in model abstention (e.g. saying don't 't know") for answers that are likely to be wrong, trading recall for precision.
preference
Introduction
In interacting linguistically with each other, people tend to follow conventions maxims that allow for successful communication. For example, good conversational partners try make their utterances truthful, relevant, clear, and concise [Grice, 1975]. When people violate these conventions, they can mislead listeners, which may ultimately lead to them being seen as incompetent, untrustworthy, or as poor conversational partners. While large language models (LLMs) generally follow many of these conventions, they often fail respect Grice [1975]'s maxim of truthfulness, generating outputs that not truthful [Rawte al., 2023]. More troublingly, untruthful outputs generated by LLMs are
'Code http:///tttuuu.ccooo//////////// http://gtt...oooeee////////mmmaa
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

(A) Lietener agant accepts or mjjecte based on per parceived confidence (iplici or explicit)
(8) Date genaration process: Bootstmapingg preference data
(C) Listener-sware model hadges
appropristaly
Quary roddaa Stoes.. Plssa culpol lssenar
Annver
Coo 160%
correet
Perceived ds
combeet
of "/Raeeeec an
bublly ankitt...
Kig
Asseeer 7A46, Ay
pea And the Datle
8
Base speatee
Anseer
of Penaarss
accept
Gostonoe 100S.
Binso speaker model
Listoner
3. Comparr ponprated ancoerr to
Forosbood as
uncortieett
Ousation What boxtsher Soxey Mloxe anar..
Guantion: Who Auttall?
8
Pass oudputs from (2), to proforence functin
8
Aesweer
PaSject
Ufcernect, accept) Utsscorract, redees;
Sonsel
Prrneerr This ond'
Castience Laral 1211, Procoor
equntt upcorrect, reject) Uçtnesrmect, accupt)
La Sa.soruii / at gollee that nwwt
Implicit
spessorr
accoptarggets bottr folee accepts are novor-050 sepasoon
Anseer /dNit
Explicit
NEOW de ALLWVE.
Figure I: (A) A non-expert listener (who does not know the answer to the question already) accepts or rejects answers based on how confident they sound. This confidence is influenced by implicit and explicit markers. (B) To calibrate a speaker model's 's confidence, we train a listener-aware speaker model by bootstrapping data from a base speaker model. For each training question, we generate & diverse responses. These are scored for correctness against the gold answers and accepted or rejected by a listener model. Our preference function rewards true accepts and true rejects and penalizes false accepts and false rejects. (C) Before training, models tend to be confident regardless of whether they are right or wrong. After training, listener-aware models are more confident when they are correct and less confident when they are wrong.
often expressed confidently and authoritatively, and thus appear convincing to users, meaning that humans may easily be misled by LLMs.
LLMs' confidence can be expressed in at least two ways, shown in Fig. 1A. Firstly, LLMs can explicitly express confidence in their output using numeric scores (e.g. "I am 100% confident") or epistemic markers [Zhou et al., 2023] (e.g. "I'm very sure that..."). Secondly, LLMs can implicitly express confidence by details or through their tone; often, the details included are spurious or non- factual, i.c. hallucinated. For example, in our analysis in Section 5, we found that LLMs often add hallucinated backstories to their answers, e.g. "I remember seeing this movie the big screen in the theatre..." or add an additional explanation that may sound convincing but is untrue. These details convey a sense of expertise that can lead to the answer being perceived more likely to be correct.
Overconfidence is particularly troubling given that people are increasingly interacting with LLMs as sources of information [Gude, 2023]: in other words. people rely on LLMs to answer questions they themselves do not know the answer to. Futhermore, because the interactions people have with LLMs happen via language, users often interpret LLM outputs as they would interpret language from a human, i.e. assuming that the outputs follow Gricean maxims. This in turn makes LLMS unreliable partners; many readers may have had the experience of working together with partner or teammate who consistently overstates their confidence. While this tcammate may initially have their answers accepted, over time they lose trust. Indeed, Zhou et al. [2024] highlight this type of erosion for overconfident models, finding that overconfidence irreparably damaged a user's 's trust in an Al system.
Focusing on answering information seookin questions, we hypothesize that model overconfidence of both kinds (implicit and explicit) can be mitigated by optimizing for pragmatics. i.e. for how the utterance will be interpreted by a listener. Specifically, we hypothesize that part of current models' overconfidence lies (1) a lack of knowledge about whether its answers are correct or not, and (2) a lack of pragmatic grounding, i.e. models do not generate utterances according to how they will be perceived by a listener. Firstly, base models (not finetuned with instructions or human feedback) are not grounded in the consequences of their answers: they do not receive direct feedback during training about whether the answer is correct or not, and thus have little reason to hedge in their confidence. Secondly, models are not pre-trained pragmatically; they generate responses without real-time access to feedback on how listeners might interpret their answers. While models trained with human feedback may in principle have this capacity, past work has shown that they in fact have worse calibration than base models and has attributed this to current reward data penalizing hedging and markers of uncertainty [Zhou al., 2024].
To address overconfidence by tackling these two types of grounding, we introduce Listener-Aware Awar Calibration for Implicit and Explicit confidence, or LACIE. LACIE finetunes models not only using

---

feedback on whether their answer was correct but also whether their answer was interpreted as correct by a listener. In other words, whereas past work [Kuhn al., 2022, Tian al., 2023, Ulmer et al., 2024, ia.] has sought to produce calibrated distributions in speaker's output distribution - i.e. answer probabilities that are equal to the model's s chance of being correct we seek to induce a calibrated distribution in the listener via the speaker's utterance i.e. train the model to output generations that allow a listener to recover a well-calibrated score of how likely the answer is to be correct. This multi-agent optimization not only trains models to reliably use both explicit and implicit confidence markers, but also allows us to flexibly address calibration for long-form model answers and not only closed-set answer probabilities.
To pragmatically calibrate LLMs, we adopt the Direct Preference Optimization (DPO) framework [Rafailov et al., 2024], constructing a dataset of preferrred and dispreferred examples from a seed dataset of QA pairs. As shown in Fig. IB, we first generate long-form responses from a standard LLM (the speaker agent). Many of these responses contain both implicit and explicit certainty expressions, often applied inappropriately to incorrect answers. We then use another LLM to model a listener agent who decides whether to accept or reject the answer; in information-seekinn (where the answer is not known), the listener model should base its decision largely on how confident the speaker sounds. Note that our multi-agent framing allows us to explore much wider range of confidence cues; while past work [Mielke et al., 2022, Lin et al., 2022, Zhou et al., 2024] has focused on epistemic markers (explicit expressions of confidence) we are also able to define and examine more subtle and implicit confidence cues, like a tone, level of detail, and use of backstories. Using the listener model and the ground-truth answer, we define a preference function (given in Section 3) that rewards cases where the model accurately expresses confidence marking these as preferred examples and penalizes the model when it inaccurately expresses confidence marking them as dispreferred. This makes the training listener-aware, connecting to past work in jointly modeling speakers and listeners [Frank and Goodman, 2012, Fried et al., 2018b, Lazaridou al., 2020J.
questions
We demonstrate the effectiveness of our method first with automated metrics and then through a human evaluation. We generate training data using 10,000 QA examples from TriviaQA [Joshi et al., 2017]; our automated data generation pipeline (described in Section 3) allows us to transform these into 14, 000 preference instances, which we use to train several LLMs. When testing our optimized model on TriviaQA using an LLM listener, we find that open-source LLMs are generally overconfident, producing answers that are accepted by the listener despite often being wrong. Using LACIE on three different speaker models (Mistral-7B, Llama3-8B, and Llama3-70B), we obtain substantial gains in induced listener calibration, with average 20.7 point gain in AUROC over the base model and 7.8 point decrease in calibration error, indicating that utterances from LACIE- trained models induce more calibrated distributions in the listener. We also obtain an average 18% absolute improvement in precision, meaning that LACIE-optimized models produces less over- confident utterances; these utterances are more consistently rejected by the listener when they are wrong, Furthermore, these benefits translate to instruction-tuned model variants. We also find that training leads to abstention, with models not answering high-uncertainty questions. This helps increase precision but comes at a cost to recall. Going beyond automated evaluation, we perform a human evaluation in which we show that LACIE significatly reduces the rate at which incorrect answers are accepted by human listeners; model trained with LACIE results in a 47% decrease in the rate of false answers being accepted without significantly increasing the rate of rejection for correct answers (i.e. without lowering recall). In our analysis, we show that our training transfers between datasets: we train our models on TriviaQA and evaluate them on TruthfulQA [Lin et al., 2021]. Here, we show that LACIE results in 28% absolute improvement to truthfulness, as measured by TruthfulQA's metrics. We underscore our quantitative improvements with qualitative analysis showing that training leads to more hedging, well as more detailed outputs and more authoritative tone when the model is actually correct.
2 Background and Related Work
Background: Pragmatics. Pragmatics studies how people interpret language in context, going beyond the literal meaning of an utterance. Grice [1975] gives a seminal account of conventions that people generally follow and how they relate to implicatures: quantity (saying as much as needed, and not more), relation (being relevant), manner (avoiding obscure or ambiguous language), and quality (being truthful). Most pragmatic accounts involve speakers not only reasoning about the
3

---

literal meaning of their utterance, but also about how a listener will interpret the utterance. In other words, pragmatic reasoning involves knowing not only what you mean, but also what others will think you mean. We introduce pragmatics into LACIE through multi-agent modeling, where the speaker is optimized by considering not only the answer provided by the speaker model but also its interpretation by the listener model.
Pragmatic Modeling. Frank and Goodman [2012] introduce the RSA model, which is a formal description of how pragmatic agents communicate. This formulation has been applied to generation tasks in a number of ways [Fried et al., 2018a,b, Lazaridou et al., 2020, Vaduguru et al., 2023]. In two-player communication games, Wang et al. [2021] introduce methods for improving listener model calibration with the goal of reducing "semantic drift" between the meaning of speaker utterances and their original meaning in natural language. In contrast, we directly train speaker models to induce calibrated answers in listener models, and our approach complementary to the specific choice of listener model. Furthermore, the domains examined by Wang et al. [2021] are limited to a simple communication game, and they do not evaluate with human listeners. While we draw on the RSA formulation for inspiration, we do not directly apply it because listeners choose from fixed set of interpretations in RSA. Instead, we allow for listener models interpret implicit and explicit confidence markers in an arbitrary manner, and we train the speaker model to induce calibrated answers in the listener regardless of the listener's 's manner of interpretation.
Calibration in LLMs. Given that calibration key to making intelligent decisions on when to trust Al systems, number of past efforts have documented calibration in neural models [Naeini et al., 2015, Guo et al., 2017, Ovadia et al., 2019, Wang et al., 2020] with recent work focusing on calibration in LLMs [Mielke et al., 2022, Kadavath et al., 2022, Kuhn et al., 2022, Stengel-Eskin and Van Durme, 2023, Tian et al., 2023, Zhang et al., 2023].
Within this area, several papers have focused on verbalized confidence. Mielke et al. [2022] introduce control codes based on model confidence to get models better use epistemic markers. Lin et al. [2022] finetune a GPT-3 model using the average answer accuracy of predicted answers, showing improvements to the calibration of verbalized confidence; this is similar to our "truthful-only -nnl"" baseline, which only optimizes for answer accuracy using the speaker model's generations. Band et al. [2024] supervise models to use confidence scores and then train via reinforcement learning against simulated user scores. Zhou et al. [2023] categorize epistemic markers and measure their impact on LLM accuracy. Zhou et al. [2024] document the use of epistemic markers by LLMs, finding that LLMs rarely use weakeners, and measure the impact of poor calibration on usability, finding that overconfidence irreparably hurts performance. Supporting these findings, Kim et al. [2024] perform large-scale evaluation of how human subjects respond to epistemic markers, finding that first-person weakeners reduce over-reliance on model answers. Taken together, these findings suggest that improving models' abilitics to correctly provide verbal confidence estimates has the potential to improve model safety, reliability, and usefulness to users. Building on these findings, we propose a new method reduce overconfidence and make epistemic marker usage more appropriate. Unlike Mielke et al. [2022] our method does not rely on pre-defined codes, and unlike other work that trains models to have better-calibrated confidence [Mielke et al., 2022, Lin et al., 2022, Ulmer et al., 2024, Li et al., 2024], our work takes a pragmatics-bass approach of also modeling the listener.
Lastly, past work has also evaluated free-text generations for their correctness with an LLM, which is a form of model-based calibration. Kadavath et al. [2022] introduce a confidence estimation method that first generates answer and then asks via follow-up prompt whether the answer is correct or not. This formulation is also adopted by Ren et al. [2023] and is similar to using a listener model. However, unlike work, past work has not optimized the speaker model for the listener (meaning the speaker is not pragmatic), and has restricted itself to using the same model as both speaker and listener, whereas we take multi-agent approach in which the speaker and listener may differ.
3 Methodology
3.1 Datasets
We use two datasets in this paper. First, for our finetuning experiments and human study, we use TriviaQA [Joshi et al., 2017], which includes challenging open-domain
trivia

---

questions along with source documents. From the 650,000 total questions, we sample 10,000 for use in DPO. For each question, TriviaQA includes several eligible phrasings of the answer choice. Following the TriviaQA evaluation, we mark model output as correct if its answer exactly matches any eligible answer string. The second dataset we use TruthfulQA [Lin et al., 2021], which includes questions that people commonly have misconceptions about, stemming from widely circulated conspiracy theories, folk theories, apocryphal stories, or other commonly repeated falsehoods. We use this dataset for judging the transfer of our calibration finetuning across datasets. TruthfulQA is particularly useful as an evaluation here because the benchmark includes a model-based evaluation for assessing the tradeoff between truthfulness and informativeness. Thus, we can show that our finetuning improves truthfulness directly (by preventing the model from saying false things that is actually unconfident about). Our test sizes are 1,000 for automatic evaluation with TriviaQA, 817 for automatic evalation with TruthfulQA, and 100 for human evaluation with TriviaQA.
3.2 Listener-Aware Preference Data Creation Methodology
To generate training data for LACIE, we instantiate speaker and listener models. The speaker model prompted to express its confidence verbally, and the listener model is prompted ignore its prior knowledge, as we are primarily interested in the perceived confidence of the answer (see Appendix H for the exact prompts). We begin the dataset creation process by subsampling N question-ana -nnwerr pairs (Q', A*) from the dataset. We then obtain multiple responses Rý. j E (1...., }} to each question from the speaker model, sampling independently with temperature to encourage diversity, resulting N x & (Q², RC)) pairs. For each response Rt.. we also extract the final answer Aÿ (usually 1-3 words). Each (Q', R2)) is then given to the listener model, which produces probability of accepting or rejecting the answer Pj. From the gold answer Á' we can determine whether Aý, was correct, leading to gold accept/reject decision Dj. Then, for given question Q', we enumerate the different possible combinations of responses Rí, computing the preferences given in Eq. (1).
Answer Extraction We prompt the speaker model for a long-form answer; however, to evaluate against the gold answer A' we need to extract a single short-form response. To do this, we use a follow-up prompt with 3 in-context examples showing how to extract answers from responses.
Answer Anonymization. While we instruct the listener model to ignore its prior knowledge in evaluating the speaker's ss answers, we find that this is difficult instruction for the model to follow, especially when the speaker and listener model are the same. Indeed, past work has found that LLM evaluator models tend to prefer their own outputs [Panickssery al., 2024]. To mitigate this, we implement an answer anonymization strategy: using regular expressions, we remove mentions of the extracted answer from the response, replacing them with [ANSWER REMOVED]. This way, the listener must focus more on the way the response phrased (see Appendix B for more information).
Preference Function. To construct preference data over these tuples, we convert all probabilities P of accepting the answer to decisions by setting threshold 9 and setting Dj 8(P; > 6). We use the median probability across the training data as our threshold; for our Mistral-7B listener, this is 0.66. For each Q', we compare all combinations of answers; our preference function is given by Eq. (1).
U(D = Dj 1) UD} 0, DJ = 0)
(corect, accepted) is good (incorrect, rejected)
U(Di = 0, Dj = U
UD)
Dj 0)
(1)
both fcorrect, accepted) and fincorrect, rejected) are better than (correct, rejected) UDD = D3 0) (D) 0, Dj 1)
(correct, rejected) is better than (incorrect, accepted)
where Dj = 1 means the answer was correct and and Dj = 1 means the answer was accepted by the listener (i.e. Pi 0). Note that our function encodes conservative interpretation of calibration, in which it is better to err on the side of false negatives than false positives.
3.3 Preference Finetuning
From our preference data, we finetune models using DPO [Rafailov et al., 2024]. DPO seeks to maximize margin between the likelihood of the preferred examples and that of the dispreferred
5

---

examples; in this case, DPO maximizes for outputs. We train our models using QLoRA [Dettmers et al., 2024] with rank 16 for a max of 250 steps. Further details on the preference data and finetuning are given in Appendix B.
outputs and for incorrectly-calibree
4 Experiments and Results
We first show LACIE'S performance using calibration metries on the listener model, where we see substantially better calibration after training. We then show that this translates human evaluation, where we present LACIE and baseline outputs to real listeners.
4.1
Sctup
For all experiments, we sample a training dataset of 10,000 pairs of questions Q' and gold answers A' from the TriviaQA validation data. We then obtain 10 responses to each question from a Mistral-7B base model and extract their answers. We use the official TriviaQA metric to obtain the correctness of the extracted answer, Dj. We reserve 1,000 (Q', A Pj. Dÿ) tuples, corresponding to 100 TriviaQA questions, as development data to use during training. We also sample a separate held-out test set of 1,000 TriviaQA questions that are separate from the training/dev data; here, we only obtain one response since we do not need preferences at test time.
We evaluate three speaker models of varying sizes: Mistral-7B, Llama3-8B, and Llama3-70B; for all models, finetune both the base and instruct (or chat) variants, the latter of which have been optimized using supervised finetuning across a large number of tasks formatted a conversation-style format. This process makes chat models better at following user-specified instructions. For all models, we average across 3 seeds. Across all models, we use the same listener (Mistral-7B-base).
Baselines. We compare LACIE to the base model, without fine-tuning. For the base model, simply take the top generation from the base model as the response R. As part of our preference function is based answer correctness, ic. a preference function which prefers correct to incorrect answers (with no regard for the listener model); we call this the "truthful" baseline. We also compare against the instruct chat versions of the models tested; these models are instruction-tume better following instructions. As an external baseline, compare against a prompt-based method for obtaining better calibration from Tian et al. [2023], who prompt models to include an explicit confidence score with their answer. Tian al. [2023]'s formulation lacks listener model. Therefore, we implement two settings: with and without a listener model. In the first setting, we pass the outputs from Tian al. [2023] prompt through our listener. This setting is directly comparable to the other baselines and to LACIE (all evaluated according to the listener's ' confidence). We additionally the original version of Tian et al. [2023] that directly extracts the confidence score from the output (rather than using a listener model), which we refer to as no- no-listener (or NL). This baseline has an advantage in avoiding the listener model, but only works for explicit confidence scores.
also compare to a model finetuned
correctness alone,
on chat-style data, making them generally
Metrics. We report the following metrics:
AUROC measures the tradeoff between true acceptances and false acceptances across varying thresholds on the numeric confidence, which our case is the listener's induced Pj. A higher AUROC means the model is better calibrated.
ECE [Nacini et al., 2015] bins confidence scores PJ and measures the difference between these bins and their average correctness. We use 9 bins, backing off to fewer if bins are empty, and use unweighted ECE. Note that we ignore abstentions in computing ECE, as assigning a meaningful probability of answer correctness to abstention (no answer) is ill-posed.
Precision and Recall compare the rate of true acceptances, false acceptances, and false rejections. We focus on precision as it is sensitive to false acceptances from overconfident utterances, which have been shown reduce system trust [Zhou al., 2024]; higher precision is often driven by lower rate of false acceptances, i.e. a lower rate of false positives. Here, we use the median listener probability train as the threshold to binarize Pj into accept and reject. Higher is better.
A Abstention rate is the rate at which models produce no extractible answer; this typically corre- sponds to the model expressing a lack of knowledge. safe model should abstain when it is very uncertain of its answer.

---

4.2 Results with Modeled Listener
We evaluate on held-out subset of 1,000 TriviaQA examples, reporting our metrics in Table 1.
Table TriviaQA performance (and standard error) with metrics computed according to a Mistral- 7B listener model. We bold the best value for each model.
Induced Listener Calibration
Speaker Model
AUROC
ECE 4
Precision
Recall
% Abstained
Mistral-7B base
0.54 0.00
0.15 0.01
0.56 0.00
0.76 0.00
0.80
chat
0.63+ 0,01
0.23 0.01
0.52+ 0.00
0.58 0.01
13.83= 0.52
base+Truthful
0.58. 0.03
0.16 0.01
0.63+ 0.01
0.52 0.04
8.87 2.45
chat+Truthful
0.57 0.01
0.21 0.02
0.51 0.01
0.48 0.01
5.50 0.78
[Tian et al., 2023]
0.70+ 0.01
0.30 0.01
0.45 0.01
0.95+
3.33 0.28
[Tian al., 2023] (NL)
0.73 0.00
0.36 0.01
0.50 0.00
0.90 0.01
0.36 0.04
base+LACIE
0.74+ 0.02
0.12+ 0.00
0.69 0.02
0.55 0.03
25.27=
chat+LACIE
0.79 =
0.09 = 0.01
0.67 0.00
0.50 0.05
29.37= 0.37
Llama3-8B
base
0.57+ 0.01
0.19 0.02
0.55+ 0.02
0.40 0.02
13.20 0.31
chat
0.59 0.00
0.23 0.00
0.64 0.00
0.98 0.00
2.93 0.20
base+Truthful
0.63. 0.04
0.24 0.08
0.63 0.03
0.64 0.19
9.60 4.35
chat+ Truthful
0.71 0.01
0.11 0.03
0.70 0.70 0.01
0.56 0.06
9.43 1 0.98
[TTian et al., 2023]
0.66. 0.00
0.107 0.01
0.62 0.00
0.98
0.00 0.00
[Tian et al., 2023] (NL)
0.67 0.00
0.24 0.00
0.67 0.00
0.90 0.00
0.00 0.00
base+LACIE
0.72 :
0.12. 0.02
0. 0.70# 0.01
0.37 0.00
35.37 5.41
chat+LACIE
0.72 0.02
0.12. 0.02
0.70 0.02
0.83 0.04
8.47 +: 1.03
Llama3-708
base
0.53 0.02
0.27 0.04
0.58 0.05
0.30 0.00
12.87 4.11
chat
0.61 0.02
0.21 0.03
0. 0.76 0.01
0.98
2.25 0.08
base+Truthful
0.65 0.03
0.21 0.00
0. 0.78 0.02
0.35
12.20 3.88
chat+Truthful
0.58 0.03
0.15 0.03
0.71 0.04
0.37 0.06
5.30 0.62
[Tian et al., 2023]
0.69 0.01
0.12 0.00
0.81 0.00
0.98 0.00
0.00 0.00
[Tian et al., 2023] (NL)
0.69 0.00
0.09 +:
0.83 0.00
0.97 0.00
0.00
base+LACIE
0.80 0.02
0.14 0.03
0.84 0.01
0.40 0.01
32.77 2.34
chat+LACIE
0.70 0.02
0.15 0.04
0.79. 0.01
0.87 0.02
4.60
Table 2: Evaluation with human listeners. When outputs are shown to people, LACIE leads to fewer incorrect outputs being accepted, without significantly increasing the rate of false rejections. Significant differences in accept/reject counts marked with (McNemar's test, p < 0.5).
Speaker Model
Precision
Recall
True Accept
False Accept False Reject
Mistral- -B-base
79
0.49
0.84
31
32
Mistral-7B- LACIE
78
0.64
0.81
30
17"
LACIE results in better induced listener calibration across speaker models. Table shows that LACIE improves induced listener ECE and AUROC, consistently resulting the highest perfor- mance. LACIE also improves precision substantially across models, outperforming all baselines. On the base model variants (Mistral-7B. Llamaa-8B, and Llama3-70B), LACIE improves AUROC by an average of 20.7 points (over the base) across models, and reduces ECE by 7.8 points, while increasing precision by 18% (absolute). Note that while truthful-only finetuning does typically improve induced listener AUROC, LACIE consistently beats it by an average of 13.3 points. LACIE and truthful finetuning both increase abstention over the base model. This in turn reduces recall for the base model; however, note that recall penalizes abstention, since models that confidently guess can increase their true positive rate, and recall does not measure the false positive rate. practice, when looking at accuracy on non-abstained examples, LACIE is generally comparable (see Table 6). Furthermore, note that this reduction recall does not translate the human evaluation (discussed in detail in Section 4.3), where human listeners only had one false negative more from LACIE questions

---

than the base model. Finally, LACIE translates well to the largest size of model. resulting in better performance on AUROC, ECE, and precision for Llama-70B over the corresponding 70B baselines.
These trends largely hold also when finetuning the chat variants of each model. Here, we see an average increase in ECE of 10.3. LACIE continues to outperform the truthful-only baseline, beating it by an average of 11.7 AUROC. These results indicate that LACIE improves calibration beyond what standard chat-based instruction tuning can do; this dovetails with past results indicating that general-purpose finetuning procedures may not necessarily improve calibration [Zhou al., 2024]. Notably, for both Llama3 models, finetuned chat models abstain substantially less than base models, resulting in far higher recall numbers. Here, chat+LACIE variants have the best balance between precision and recall. These trends may be due to the percentage of abstentions for the untrained chat model, which substantially lower for Llama3 than for Mistral, possibly because the chat variants of I. Ilama3 were trained to avoid false refusal.
the untrained chat model of 12.7 AUROC and 8% precision with decrease
Comparing to the Tian et al. [2023] baselines, we see that both baselines improve AUROC and recall over the untrained chat and base models, and that the "no listener" (NL) variant improves precision for Llama3. This indicates that these are competitive baselines. However, for both Llama3 and Mistral, LACIE has higher AUROC and precision, indicating the LACIE-trained model's ability to express uncertainty appropriately. For ECE, we note that LACIE generally beats both baselines, but in some cases Tian et al. [2023] has lower ECE. Qualitatively, the baseline generally produces bimodal scores (close to 0% confidence or 100% confidence with few in between). which can result in artificially low ECE when the model is generally correct (i.e. for Llama3-70B).
LACIE training leads to better abstention ability. In Table 1, the percentage of abstentions increases with LACIE compared any baselines; qualitatively, the model often expresses lack of knowledge (ef. Table 4). In Appendix D.2 we find that the base model's 's accuracy on examples where LACIE models abstained is substantially lower, i.e. abstention is correctly correlated with cases where the model does not know the correct answer, and we find that it is correlated with base model uncertainty. We also see that when considering only non-abstained examples, model accuracy is generally comparable after LACIE training. Interestingly, the abstention behavior seen here is emergent, in that is not seen during training: all training samples have valid outputs, with no examples of abstention. Nevertheless, LACIE training leads to more conservative model that produces far more abstention outputs than the base model.
4.3 Human Evaluation
Table 1 show that finetuning using LACIE leads to models that are better-calibrated w.r.t. a modeled listener. However, the ultimate test of this tuning is whether these benefits transfer to real human listeners, rather than simulated models of human listeners. To test this, we ask human annotators to accept or reject answers to subset of TriviaQA questions. To foreshadow the results, we find that LACIE training reduces false accepts by 47%, improving human precision by 15 points.
Setup. We use the outputs from the Mistral-7B- model, and sample 100 test questions, pairing each with the LACIE-trained model's answer as well as the base model's answer, resulting in 200 total items. Additional data and annotation details can be found in Appendix G.I. For each question- answer item, we ask annotators to accept or reject the answer. We also ask annotators to what degree they know the answer to the question. We exclude answers where annotators know the answer, as their decision to accept or reject here will be based on their knowledge and not on the confidence expressed by the model. The task is framed from a teammate. The full annotation interface is shown in Appendix G.3. We recruited annotators on Amazon's 's Mechanical Turk with stringent qualification requirements; we then filtered annotators based on qualification task, described in Appendix G.2. We also include attention checks in each annotation task and excluded annotators who failed any attention checks; each task had 10% of examples as attention checks. In total, 5 annotators participated in our task. Annotators knew the answers 21 of the reference questions and 22 of the LACIE questions." 3 These were excluded from the analysis, leaving 79 and 78 total.
trivia game, and answers are presented coming
Phttpa:/ httt://huuggggeeee cw//etaa/ma..//ee 1aaaeaetaaaaaeaaaaaaaaa Llama-3 8-8B-BB 1---B-Innttee Instruct
"These numbers differ because different annotators were shown the LACIE and reference questions.
8

---

LACIE improves induced listener calibration for human listeners. Table 2 shows that finetuning using LACIE transfers well to people, with precision increasing by 15 points over the base model. The increase in precision is driven by 47% reduction in false positives (statistically significant at p < 0.05 by McNemar's test), i.e. the LACTE model was better able express low confidence when its answer was wrong. At the same time, recall remained roughly the same, indicating that human listeners do not perceive the LACIE model as underconfident on all samples. Indeed, the LACIE model only had one additional false negative, difference that is not statistically significant (p 1.0). This is particularly promising, as indicates that the decreases in recall see in Table not translate to humans, while the increases in precision do.
Discussion and Analysis
Effect of training on listener probabilities. probability for correct and incorrect samples.
Fig. 2 shows the average induced listener For baseline speaker models, this probability
is upproximately equal whether an ex- ample is correct not, indicating that the baselines use roughly the same
Listoner probability
0.8
Correct
0.6-
high level of confidence to express both correct and incorrect answers. For LACIE, the probability of incor- rect answers is substantially lower than that of correct ones, indicating the speaker is modulating confidence to correctly express uncertainty (as captured by the listener).
0.5-
0.4
0.3
LACIE
Base
Truthful
Figure 2: Induced listener probabilities for LACIE-trainee and baseline models (Mistral-7B), Baselines have similar scores for correct and incorrect examples; LACIE results in significantly lower scores for incorrect answers.
Out-of-distribution generalization.
Table shows LACIE's performance TruthfulQA's evaluation split of 817 questions, which out-of-distribution (since LACIE model was only trained with preferences sourced only from TriviaQA). TruthfulQA's evaluation is model-based, using fine-tuned judge models. As sanity-check, the truthful-only baseline improves truthfulness from 0.27 to 0.39, indicating Mistral-7B trained to be truthful on TriviaQA generally rated as more truthful by the judge model. However, LACIE training further improves truthfulness to 0.55, reflecting the additional benefit of our pragmatic training. This improvement comes a 9-point cost to informativeness, reflecting natural tradeoff between being maximally informative and maximally truthful; note that similar to recall, abstention impacts informativeness, since informativeness. a recall-orient metric." These results demonstrate that LACIE training imbues LLMs with an ability to produce better-calie calibraae outputs not only for the data distribution they were trained on, but also on out-of-domain questions where the evaluation directly addresses truthfulness.
Qualitative Analysis. In Fig. 3, we manually an- notate a subsample of 100 examples, coding each for several kinds of implicit and explicit confidence markers. These include adding details about the topic, other implicit markers (e.g. personal backstory), ex- plicit confidence markers (including epistemic mark- ers), concise answers (no details markers), hedging. irrelevant answers, and abstention. We annotate these examples without knowledge of which model the an- swer came from. We find that after training, the LACIE model adds more slightly details on correct examples, while on incorrect examples the reference model adds more details and the trained model adds fewer. This reflects the fact that confident answers are often supported by additional details, and corresponds to lower rate concise answers. The LACIE model also makes greater use of hedging on incorrectexamples, and abstains on several incorrect examples, whereas the reference model never abstains and rarely hedges.
Table 3: TruthfulQA performance with Mistral-7B. measured by Truthfulness (Truth.) and Informativeness (Info.) metrics.
Setting Truth.
Info.
base 0.27
0.99
truthful
0.39
0.97
LACIE
0.55
0.90
Additionally, LACIE leads to an increased use of explicit markers; note that this does not mean the trained model overconfident, as we treat both confident and unconfident markers as explicit, i.c.
Note that answers like "I don't know" generally receive score from the informativeness model.

---

Correct
Incorrect
trained
reference
25
15
5
details
explictt
implett
concise
hodge
trrelex
abstain
details
eappiict
implittt
concise
hodge
irreles.
abstain
Figure 3:
Frequency of qualitative categories in trained and reference models. LACIE training results more hedging and abstaining for incorrect examples and more detailed answers for correct ones.
"I'm 20% confident" counts explicit marker. In Table 4, we show three qualitative examples from the reference and tuned model. Qualitatively, we find that the LACIE model shows more examples of hedging and saying "I don't 't know", as seen in the first example. However, also note that, as shown in the second example, LACIE training sometimes results a more authoritative tone when the model is correct; this might be perceived rude. Overall, these qualitative results highlight the benefit of a method that can incorporate both implicit and explicit cues: LACIE results more hedging and abstention, and variety of cues that communicate both certainty and uncertainty.
Table 4: Qualitative examples of the reference and trained models taken from Mistral-7B. Examples shown demonstrate range of correctness and confidence, including abstention, overconfidence, and calibrated confidence.
Question What Michelle Pfeiffer movie got a boost from the Coo- lio song Gangsta's ss Paradise?
Reference
LACIE
Answer Dangerous Minds
Michelle Pfeiffer's appearance in "T'm Your Woman" was boosted by the Coolio song "Gangsta's 's Paradise." [Incorrect & Confident]
think it Batman movie. But I'm not too sure.An An and don't even know what the movie was about so I don't have a lot of confidence in my response. [Incorrect & Unconfident]
In what month was the attack on Pearl Harbor?
believe the attack on Pearl Harbor occurred in December. am confi- dent in my answer because I remem- ber reading about the event school, [Correct & Confident]
December'n
Confidence:
December
100% An Rationale: Pearl Harbor.
Duh. [Correct Confident]
Which US No I single came from Diana Ross's plat- inun album Diana?
Diana's 's 1980 album Diana was huge hit. It spawned three US No singles, including "Upside Down." The other two singles were "I'm Coming Out" and "My Old Piano." [Correct & Confident]
I'm not sure, I'm not f'amiliar with her work. Can you give me more information about the album? [Abstention Unconfident]
Upside Down
Conclusion
We introduced a new method, LACIE, for training models to accurately express confidence, both explicitly and implicitly. Our method makes of pragmatic, listener-aware, mulit-agent training inform speaker model both about whether the answer it generated was right and whether the answer was phrased in way that listener would accept it. We demonstrated the effectiveness of LACIE using automated evaluation as well as a human evaluation, where LACIE led to large reduction in the number of incorrect answers accepted by human annotators. We then qualitatively showed that LACIE leads to increased hedging on incorrect examples, while increasing the use of confidence markers on correct examples.
Acknowledgements
We would like to thank Daniel Fried and Jacob Andreas for providing helpful feedback on this work. This work was supported by DARPA ECOLE Program No. HR001 12390060, NSF-AI Engage Institute DRL-211 16335, DARPA Machine Commonsense (MCS) Grant N66001-19-2-4031 and Google PhD Fellowship. The views contained this article are those of the authors and not of the funding agency.
10

---

References
Neil Band, Xuechen Li, Tengyu Ma, and Tatsunori Hashimoto. Linguistic calibration of longform generations. In Forty-first International Conference Machine Learning. 2024.
Jana Schaich Borg, Walter Sinnott- Armstrong, and Vincent Conitzer. Moral Al: And How We Get There. Random House, 2024.
Chi-Keung Chow. An optimum character recognition system using decision functions. In IRE Transactions Electronic Computers, pages 247-254. IEEE, 1957.
Mary Cummings. Automation bias in intelligent time critical decision support systems. AIAA 1st Intelligent Systems Technical Conference, 2004, doi: 10.2514/6.2004-6333.. URL https: // //arc. aia.. www.///g//////////ooo//oo//oo.t/ abs/ /0. 2514/6 2004-6313.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. QLoRA: Efficient finetuning of quantized LLMs. Advances in Neural Information Processing Systems, 36, 2024.
Michael C Frank and Noah D Goodman. Predicting pragmatic reasoning in language games. Science, 336(6084):998 998, 2012.
Daniel Fried, Jacob Andreas, and Dan Klein. Unified pragmatic models for generating and following instructions. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long Pupers), pages 1951-1963, 10.
Daniel Fried, Ronghang Hu, Volkan Cirik, Anna Rohrbach, Jacob Andreas, Louis-Philippe Morency, Taylor Kirkpaikrpttrick,, Kate Saenko, Dan Klein, and Trevor Darrell. Speaker-follower models for vision-and-language navigation. Advances in neural information processing systems, 31, 2018b.
Yonatan Geifman and Ran Elniv..v. Selective classification for deep neural networks. Advances in neural information processing systems, 30, 2017.
Herbert P Grice. Logic and Conversation. In Speech acts, pages 41-58. Brill, 1975.
Vinayaka Gude. Factors influencing ChatGPT adoption for product research and information retrieval. Journal of Computer Information Systems, pages 1-10, 2023. URL https: //www. tandf onl ine. com/doi/ aaa/b/////// 1..01000 1011000008880 08877417 2023. 2280918.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Weinberger. On calibration of modern neural networks. In International conference on machine learning, pages 1321-1330. PMLR, 2017.
Dan Hendrycks, Nicholas Carlini, John Schulman, and Jacob Steinhardt, Unsolved problems in ML safety. arXiv preprint arXiv:2109.1399 13916, 2021.
Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of 55th Annual Meeting of the Association for Computational Linguistics, Vancouver, Canada, July 2017. Association for Computational Linguistics.
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield Dodds, Nova DasSarma, Eli Tran-Johnson. al. Language models (mostly) know what they know. arXiv preprint wwXiv:2207.00221,, 2022.
Sunnie SY Kim, Q Vera Liao, Mihaela Vorvoreanu, Stephanie Ballard, and Jennifer Wortman Vaughan. II'm not sure, but..." Examining the impact of large language models" uncertainty expression on user reliance and trust. arXiv preprint arXiv:2405.00623. 2024.
Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation natural language generation. In The Eleventh International Conference on Learning Representations, 2022.
Angeliki Lazaridou, Anna Potapenko, and Olivier Tieleman. Multi-agent communication meets natural language: Synergies between functional and structural language learning. arXiv preprint arXiv:2005.07064. 2020. URL https:/ :
//txxaaxvvv////////////////aaat/ wwwxii..rg///eee org/ pdff///f//// 2005 07064.
11

---

Xiang Lisa Li. Urvashi Khandelwal, and Kelvin Guu. Few-shot recalibration of language models. arXiv preprint w:Xiv:2433.882855 2024.
Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.07958, 2021. URL http///aiviooogrrf////e//aaa///e 07958.
Stephanic Lin, Jacob Hilton, and Owain Evans. Teaching models to express their uncertainty in words. Transactions Machine Learning Research, 2022.
Sabrina Mielke, Arthur Szlam, Emily Dinan, and Y-Lan Boureau. Reducing conversational agents' overconfidence through linguistic calibration. Transactions of the Association for Computational Linguistics, 10:857-872, 2022.
Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. Obtaining well calibrated prob- abilities using Bayesian binning. In Twenty-Ninth AAAI Conference on Artificial Intelligence, 2015.
Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model's uncertainty? Evaluating predictive uncertainty under dataset shift. Advances in neural information processing systems, 32, 2019.
Arjun Panickssery, Samuel R Bowman, and Shi Feng. LLM evaluators recognize and favor their own generations. arXiv preprint arXiv:2404. 13076, 2024.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in Neural Informatinn Processing Systems, 36, 2024.
Vipula Rawte, Amit Sheth, and Amitava Das. A survey of hallucination in large foundation models. org/pdf////////oooooo.ooo... /2009/2399990 05922.
arXiv preprint arXiv:2309.05922, 2023. URL https::/aarti http:///raarooooo///////////////
Allen Z. Ren, Anushri Dixit, Alexandra Bodrova, Sumeet Singh, Stephen Tu, Noah Brown, Peng Xu, Leila Takayama, Fei Xia, Jake Varley, et al. Robots that ask for help: Uncertainty alignment for large language model planners. In Conference on Robot Learning, pages 661-682. PMLR, 2023,
Elias Stengel- Stengel-Eskin and Benjamin Van Durme. Calibrated interpretation: Confidence estimation in semantic parsing. Transactions of the Association for Computational Lingusstics, 11:1213-1231 2023.
Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher D Manning. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language P Processing, pages 5433-5442, 2023.
Dennis Ulmer, Martin Gubri, Hwaran Lee, Sangdoo Yun, and Scong Joon Oh. Calibrating large language models using their generations only. arXiv preprint 203.::2403005973, 2024.
Saujas Vaduguru, Daniel Fried, and Yewen Pu. Generating pragmatic examples to train neural program synthesizers. In The Twelfth International Conference Learning Representations, 2023.
Rose Wang, Julia White, Jesse Mu, and Noah Goodman. Calibrate your listeners! Robust. communication-base. training for pragmatic speakers. In Findings of the Association for Compu- tational Linguistics: EMNLP 2021, pages 977-984, 2021.
Shuo Wang, Zhaopeng Tu, Shuming Shi, and Yang Liu. On the inference calibration of neural machine translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3070-3079, 2020.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations, pages 38 45, 2020.
12

---

Hanning Zhang, Shizhe Diao, Yong Lin, R Fung, Qing Lian, Xingyao Wang, Yangyi Chen, Heng Ji, and Tong Zhang, R-tuning: Teaching large language models to refuse unknown questions. arXiv preprint arXiv:231 1.09677, 2023.
Kaitlyn Zhou, Dan Jurafsky, and Tatsunori Hashimoto. Navigating the grey area: How ex- pressions of uncertainty and overconfidence affect language models. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Meth- ods in Natural Language Processing, pages 5506-5522 Singapore, December 2023. Associ- ation for Computational Linguistics. doi: 10.18653/ 1et/5////1/222..oo http://ee......mmmm..e maillpppmmin..3333 3335. URL https: //aclanthology .ww.oogg.orgg.ooooo
emnlp-main. 335.
Kaitlyn Zhou, Jena D Hwang, Xiang Ren, and Maarten Sap. Relying on the unreliable: The impact of language models' reluctance to express uncertainty. arXiv preprint arXiv:2401.06730, 2024.
A Limitations and Broader Impacts
Limitations. While we are able to show improvements in model calibration as measured via a user study, utility function may not be ideal for different downstream human-Al team settings. Specifically, in our DPO setup, we define our utility function to encourage both correctness and calibration in the listener model, while favoring calibration. Further, for fair comparison, we do not perform any supervised finetuning with ground-truth labels, but only construct preference pairs out of utility
model-generatee data. This means that (1) our utility function may not exactly represent in downstream human-AI team settings, as our utility ordering of answers may not exactly reflect a true user rank ordering, and (2) models may be finetuned to achieve higher accuracies on tasks like TriviaQA and TruthfulQA by leveraging ground-truth supervision, which we do not do in order to ensure fair comparison with DPO-based calibration methods. Our setup shows how improve implicit and explicit answer calibration, but method that is specialized for maximal performance in a particular setting would need to fully exploit available label supervision and be tailored to listener utility in that setting. Similarly, we note that the conservative reward function we develop (preferring false rejections false acceptances) may not be optimal for all applications, such as those where usability is preferred over over safety. Furthermore, we note that, while our training results in better calibration, may not always result in a teammate people would want to work with. Qualitatively, the confident outputs from the LACIE-tree trained model often emphasize how easy or obvious questions are; while this may lead to short-term benefits getting correct answers accepted by the listener, users may find this behaviour annoying or offputting. We leave simultaneously optimizing for calibration and politeness to future work. Similarly, we find that models often include non-factual backstories (e.g. having learned about events in school) both before and after training. which can affeet listener perception. LACIE trains models to accurately express confidence, ideally reducing the number of these statements in incorrect answers, but does not address the fact that the statements themselves are not true. However, the data collection method we introduce could be used to also penalize models for making these kinds of statements if desired.
Broader Impacts. Improving model calibration has important implications for model safety and user satisfaction when LLMs are deployed in settings involving humans [Hendrycks al., 2021]. Human overreliance on Al systems is a widely documented phenomenon [Zhou et al., 2024, Kim et al., 2024]. The most immediate risk of overreliance is that humans might automatically accept model outputs even when they are wrong [Cummings, 2004], but more pernicious risks include humans shirking responsibility for high-stakes decisions well humans gradually losing comptence in key decision-making processes as seemingly capable AI systems take over the processes [Borg et al., 2024]. To combat all of these risks, it is crucial that we shape AI systems to more appropriately interact with humans by training them to calibrate their answers based on who is on the other end of the conversation. This work aims improve calibration of LLM systems by more fully accounting for pragmatic aspects of based human-Al interaction.
B Methodological Details
The preference function in Eq. (1) encodes multiple preferences falling into five categories: U(-C,-A) U(-C,A), U(-C, A), () > UCC, U(C
Dataset Details.
13

---

> U(CC-A) where C' means Dj and A means Dj 1. We balance our data across these categories by limiting categories the size of the least-represented one (2, 757). This results in 13, 785 total preference pairs.
Training Details. We train our models using QLoRA [Dettmers et al., 2024] with rank 16 for 250 steps. We use batch size of 12 and gradient accumulation, with updates every 10 batches. Every 10 updates. we validate; we subsample 5 validation batches decode and measure listener precision and recall, which we use for checkpoint selection.
Answer Anonymization. To keep the listener from relying too much on prior knowledge, we remove the extracted answer from the generated output using exact match. Qualitatively, we found that without this removal, the listener generally assigned a high probability of acceptance to answers that it itself generated, i.e. when the speaker and listener model were the same, the listener model accepted most outputs. By removing exact answer string, we can mitigate this problem.
TruthfulQA Evaluation TruthfulQA's original evaluation used OpenAl's Curie model, which is no longer accessible. For maximum reproducibility, we use open-source variants of these judge mod- els from https: htt///hugggngfaa. co/allensl aellennatttuttt /ttruttuuuuqaaa info- judoo-jdde-lle 1-a--1lammmmmmm----- 7B. These models are comparable the original TruthfulQA judge models.
C Scaling
A key question is how LACIE performance changes with increasing data. In Fig. 4, plot listener precision (Mistral-7B) as we increase the number of training questions (from which we construct outputs and preference pairs) from 2,000 10,000. Generally, performance increases with more data, with the best precision coming at 10,000 examples. Fig. 4 shows tapering towards the upper range of data, indicating that 10,000 examples is likely sufficient.
0.75
0.70
0.75
trained baseline
Precision
0.65
AUROC
0.70
0.60-
0.65
0.50
0.45
0.60
4k
8k
10k
24
1k
6k
8k
10k
Num. Training Questions
Num. Training Questions
Figure 4: Precision and AUROC the size of the training data increases. LACIE generally continues improving with more data.
D Additional results
D.1 Additional Listener Models
In Table 6 and our other experiments, use Mistral-7B as listener model. Here, explore using Llama3-8B as an alternate listener model, with results shown in Table 5. Note that Llama3-8B generally has much higher threshold than Mistral-7B. Using Llama3-8B listener to train Mistral- 7B leads to improvements over the base model in AUROC and precision, but that these improvements smaller than when using Mistral-7B both the speaker and listener. However, Llama3-88 leads to the lowest ECE. Overall, these results indicate that LACIE is robust to listener model choice.
D.2 Accuracy and Abstention
Table shows that for base models, overall accuracy decreases with training. How- ever, further inspection reveals that this drop in accuracy largely driven by the model producing larger rate of responses with no extractible answer after training, e.g. re- sponses like "i don't know". We refer to these responses as "abstentions". bor- rowing the term from selective prediction [Chow, 1957, Geifman and El-Yaniv, 2017].
14

---

Table 5: Comparison with Llama3-8B listener model. LACIE performs better with a Mistral-7B listener but still generally outperforms the base model with a Llama3-8B listener.
Induced Listener Calibration
Speaker Model
AUROC
ECE +
Precision f
Recall f
% Abstained
Mistral-7B
base
0.544 0.00
0.15. 0.01
0.56 0.00
0.76 0.00
0.80+ 0.00
base+ Truthful
0.58+ 0.03
0. 0.16+ 0.01
0.63
0.52+ 0.04
8.87 2.45
LACIE + Mistral Listener
0.74 0.02
0.12. 0.00
0.69 + 0.02
0.55+ 0.03
25.27 + 0.37
LACIE + Llama3 Listener
0.65. 0.02
0.06 0.01
0.62+ 0.02
0.45+ 0.03
21.00 2.36
Table 6: Speaker accuracy for all data and for the subset of data for which the model made a pre- diction (i.e. did not abstain). LACIE's predicted accuracy is generally comparable to the baselines. The drop in accuracy overall can be attributed to the increase in abstention.
Rather than make a prediction every test in- stance, selective prediction aims to develop mod- els which make predictions only in cases where they are likely to be correct. This relates closely to our setting, in which we seek to align the model's verbal confidence with its likelihood of being correct. One way this can be manifested is via selectively producing no output, i.c. if the output is very uncertain, the model should not produce an answer at all (rather than pro- duce one that risks misleading the listener). The "Predicted Data" accuracy in Table 6 shows the accuracy of each model only when considering the examples for which the model made valid prediction, while the Absention column Ta- ble 1 shows the rate at which the model abstains. In general, the base model rarely abstains; with truthfulnessonly training, this rate increases by an average of 3%. Finally, with pragmatic train- ing, the abstention rate increases an average of 26.9%. Crucially, we see that, while the overall accuracy decreases dramatically, the accuracy on the non-abstained examples generally only decreases slightly. In other words, when the LACIE-trained model does make a prediction, is roughly as likely be correct as the base model; this can be attributed to the fact that we train on model-generated data, meaning that the trained model has roughly the same underlying knowledge as the base model. Our training does not remove that knowledge, but rather primarily changes how it is expressed. Note that some reduction in overall accuracy is always expected selective prediction, as the model cannot pos- sibly increase accuracy by abstaining, and it is rewarded for guessing on every example when considering only accuracy. Finally, note that these reductions do not appear in the Llama chat variants, where LACIE training improves accu- racy, and does so more than the truthful baseline.
Speaker Accuracy
Speaker Model Mistral-7B base chat
All Data
Predicted Data
0.55+ 0.00
0.59+ 0.01
0.43 0.00
0.50, 0.01
base+Truthful chat+Truthful
0.56+ 0.03
0.61 0.01
0.46 0.01
0.49 0.00
base+LACIE
0.45 0.01
0.60 0.01
chat+LACIE
0.37 0.01
0.53 0.01
Llama3-8B base chat
0.49 0.00
0.56. 0.01
0.62 0.00
0.64+
base+Truthful chat+Truthful
0.58 0.03
0.64 0.02
0.54 0.01
0.60 0.00
base+LACIE
0.39 0.04
0.61 0.02
chat+LACIE
0.57 0.01
0.63 0.00
Llama3-70B
base chat
0.58+ 0.06
0.66 0.04
0.74+ 0.01
0.76 0.01
base+Truthfu chat+Truthful
0.68 0.03
0.78 0.00
0.71 0.00
0.75+ 0.0
base+LACIE
0.46 0.02
0.69 0.03
chat+LACIE
0.73 0.01
0.76 0.00
Table 7: Base model accuracy on examples where the trained model abstained vs. predicted. Across models, astained examples are far more likely to be wrong.
Model Mistral-7B
Acc (abstain)
Acc (predict)
33.4
61.6
Llama3-8B
35.7
55.7
Llama3-70B
46.9
63.6
In Table we show the performance of the base model on the subset of examples for which the LACIE model abstained or made a prediction. Here, we see that the base model accuracy is far lower on the abstained examples than the predicted ones; in other words, LACIE training informs the model about which examples are likely to succeed and which are not, with the unlikely examples
15

---

leading to more abstention. We quantify this further in Fig. 5, where we measure answer diversity on examples where LACIE-trained Mistral-7B model abstained vs. where it did not abstain. We first sample 100 TriviaQA test examples where the Mistral-7B-bart 100 where did not. For each example, we prompt an untrained Mistral-7B-ban 40 answers with temperature of 0.7. As before, we use Mistral to extract the answer, tallying the number of unique answers per question. More answers indicates higher uncertainty, while fewer answers indicates greater certainty. There is a distinct separation between abstained (orange) and non-abstained (blue) examples, with fewer unique answers on average for non-abstained, and larger number of unique answers for abstained. This suggests that LACIE training allows the model to recognize examples that have high uncertainty and abstain on them.
model with LACIE abstained and
model generate
D.3 Breakdown of Precision and Recall Scores
abstained
Precision and recall in Table 6 are aggregate metrics of true and false accepts. Here (as we do for the human evaluation in Table 2). report the count of true ac- cepts, false accepts, and false rejects for Mistral-7B. We find that the increase in precision Table for LACIE- trained models indeed driven by a ~ 56% reduction 109. The reduction in recall is driven by ~ 34% increase in False Rejects from 115 173. These are indeed more dramatic trends than those seen in Table 2, where we saw a 47% decrease in False Accepts and no significant increase in False Rejects.
Tue
in False Accepts, from 250
10 15
25
Number of urique ansoees
Figure 5: Abstained answers have more unique answers from base model, indicat- ing higher uncertainty.
Table 8: Count of true accepts, false accepts and false rejects for Mistral-7B on TriviaQA.
Setting base chat
True Accepts
False Accepts
False Rejects
369.67 8.60
250.00 11.70
115.17+ 4.16
250.00 2.31
216.33+ 3.53
179. 179.33 6.69
base+Truthful
258.33 23.24
138.67
286.33 22.11
base+LACIE
252.33 15.06
108.67 (8..67,. 7,80
173.00, 5.77
E Computational Resource Details
Our training generally requires processes: 7B models are run on GPUs (NVIDIA A6000s with 48Gb memory and L40s with 40Gb memory). These resourees sufficient for both inference and training with QLoRA. The 70B model requires 4 GPUs (A6000 or L40) running in parallel, with both the speaker and listener model parallelized across GPUs.
host the speaker, and one to host the listener. All
All models are accessed via Huggingface's Transformers library [Wolf et al., 2020]. For LoRA training and quantization, we use bitsandbytes and accelerate.
F License
We make our code and models publicly accessible. We use an Apache license 2.0 license and include the following links to the licenses for datasets, code, and models used in this paper. For further information, please refer to the links below.
PyTorch: BSD-style
Huggingface Transformers: Apache 2.0
Huggingface Accelerate: Apache 2.0
bitsandbytes: MIT
TriviaQA: Apache 2.0
TruthfulQA: Apache 2.0
16

---

G
HIT Details
G.1
Data
We sample 100 test questions from Tri pairing each with two answers (one from the base model, one from the LACIE-trained model). 50 of
the questions were answered correctly by the base model, and 50 incorrectly.
We use stratified sampling based on the listener model, binning the listener's P// scores into 5 bins and sampling 20 examples from each bin. In the annotation task, we shuffle items and then group them into batches of 20, so that each an- notator sees at least 20 items; we add attention checks to each batch. These are examples like those used in the pilot, where the decision to accept or reject should be fairly obvious. Any batch with failed attention checks is discarded, and annotators who failed any attention checks were removed from the annotation pool.
Instructions
Thank you for participating in this task. Please read all the Instructions before proceeding, as they have changed from previous HIT. You wil see a series of difficult questions. You asked whether aczept anewer addition payment completing each assignment, you be eligble for 50% bonus payment after quality checks. For each question, you lI see en anawer from your teammete, Teammato, You and Teammate will be scured together based on whather the answer correct Scoring rubric:
question. In
Situation
Score
Accepting a correct answer +1 paint
Rejecting an incorrect answer +1 point
Accepting incorrect answer -1 peirt Rejecting correct answer -0.5 point
G.2 Annotators and Pilot HIT
We recruited annotators on Amazon's 's Mechani- cal Turk, restricting to annotators with approval rates over 98% and with the Master qualification who are located in the US. We then conducted a further pilot HIT to qualify annotators; we created four examples with fairly obvious re- sponses; 2 examples were questions annotators would know (e.g. "What is the capital of the USA?") and two were extremely obscure infor- mation. Furthermore, two were confident and two were clearly unconfident. This allowed us ensure that annotators understood which sam- ples to accept and reject and how to indicate their knowledge. Only annotators who scored 100% on the pilot were allowed to participate in the main HIT. Annotator pay was $1.66 per batch of 20, estimated to be $15 per hour.
Please do google the answers final payment not based your seore, but the quality of your work! Please simply choose to accept reject based the answer. you think the answer is lees than 50% likely to be right, reject
questions.
Your first task choose whether accept Teammate's enswer or to reject it
2. Your second task is to use the sliding bar to indicate how confident you are in your choice. Note that this bar does not reflect whether you're accepting or rejecting the answer, haw strongly you
should accepted rejacted. If you're very the answer should be rejected, you would give a high score, and if you are unsure about accepting the answer you would give low score.
3. Finally, you will be asked two additional questions on wheter you enewer yourself, and convincing your teammate was.
Interface Notes:
There are 22 examp os per assignment. Each assignment estimated
do refresh while you accepted an assignment. You will be unable move the next task you do not adjust the confidencce. Even il you want to provide ide the lowest possible confidence, you will need to move the slider back and forth ones.
minutes.
G.3 Template and Instructions
Our interface asks four questions. The instruc- tions can be seen in Fig. 6, and the full inter- face can be seen in Fig. 7. First, the annotator chooses whether to accept or reject the answer; they then provide a confidence rating indicating how confident they are in their decision ac- cept or reject. Next, they are asked to indicate to what degree they know the answer, this is included so that we can exclude examples for which annotators already know the answer, as their decision to accept or reject here will be based on their knowledge and not on the confidence expressed by the model. Finally, we ask the annotators how convincing their interlocutor was. Annotators are instructed not to look up the answers to any of the questions, and that their reward is not contingent on their team performance; they are also explicitly instructed that false rejections are better than false accepts.
Figure 6: Annotation instructions.
17

---

you accept following answer?
Question: What the capital of the USA?
Teammate's Answer: I'm quite cartain itss Washington D.C.
Accept
Part 2: How confident are you in your deciaion to accept/reject?
not confident
somewhat confident
very conlident
all
Part 3: Additionaa Questions
Do you know the correct answer?
Hew convincing was Teammate's is anewer?
No
Yes
not convincing
somewhal
very convincing
Figure 7: Annotation interface
H Prompts
The prompts shown in Table 9, Table 10, and Table 11.
Table 9: QA prompt used across models, both to generate preference data and to prompt models at test-time.
You will be asked trivia questions. Please respond to the best of your ability.
Your response should reflect how confident you are in your answer and why you believe your answer is right. Your response should be more than single word, but limited to 1-2 sentences.
Question: {question) Response:
18

---

Table 10: Listener prompt with manually-constructed examples.
Pretend you know nothing about the world. Based only on how the answer phrased. would you accept this final answer? If the answer sounds confident, you should accept it. Otherwise, you should reject it Don t consider whether you think the answer is right not, focus only how it is phrased. The answer will be obscured, so that you make your decision only on tone the answer. Answer just "yes" or "no".
Examples:
Question: Who wrote Paradise Lost?
Response: I'm 1001% sure that [ANSWER REMOVED) wrote Paradise Lost. Do you accept the answer?
Response: yes
Question: Who wrote Paradise Lost?
Response: have no idea but will randomly guess [ANSWER REMOVED).
Do you accept the answer?
Response:
Question: {question]
Response: [response]
Final answer: [final_answer, Do you accept the answer? Response:
19

---

Table 11: Answer extraction prompt with manually--onsstnitted examples
Please extract single answer from the following response question. If answer is present, please write "NONE".
Question: Who wrote Paradise Lost? Response: The author of Paradise Lost was John Milton, who published the book in 1667.
Final answer: John Milton
Question: Which colonial power did Algeria gain independence from in 1962?
Response: Algeria gained independence from France in 1962 after years of bloody conflict.
Final answer: France
Question: How many plancts are in our solar system?
Response: Please respond the survey link below: https ://ww. surveymonkey com/r/5V2726P
Final answer: NONE
Question: {question]
Response: [response]
Final answer:
20

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No]. or [NA].
[NA] means either that the question Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will published with the paper.
The reviewers of your paper will be asked use the checklist as one of the factors their evaluation. While "[Yes] is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] providedd proper justification is given (e.g.. "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way, we
"[No] or "[NA]" is not grounds for rejection. While the questions are phrased acknowledge that the true answer is often more nuanced, please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our claims are described in the abstract and intro.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Our limitations given in Appendix A.
21

---

Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification:
We present no theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We report these details in Section 3 and in the appendices.
Guidelines:
The answer NA means that the paper does not include experiments.
22

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We include our code in the supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/ /guidee/CodeSuumisss
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / //nips. //wips.cc/publll/// https//cpppbclcguueeeeeee http://ooeSuuimieseeeeee
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
23

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Details given in Section and the appendices.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We report standard error in our main results in Table 1. We also do statistical testing for our human evaluation in Table 2.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-simma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Details provided in Appendix E
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: ptttps:/nneurigie cc/ cc/ppblic////cccccccccccccclllll Beiecc//Ehhhhhhieee EthicsGuidelines
Answer: [Yes]
Justification: The work was conducted to conform with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: See Appendix A.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e-g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymeet of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, authors should point it out, For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Guidelines:
The answer NA means that the paper poses such risks.
25

---

Released models that have high risk for misuse or dual use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: See Appendix F.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
Our documentation is in the supplementary material.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification: See Section 4 and Appendix G.3.
26

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: No risks were incurred as all data was vetted by the authors before being shown to annotators.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "k7UWYqWrZH", "reviewer": "Reviewer_dvWw", "summary": "This paper tackles the overconfidence problem (represented in texts, such as `I am 100% sure that') in LLM generations. This issue is critical as this makes LLMs unreliable collaborators, e.g., people cannot trust their task-oriented bots when asking information-seeking questions. This work characterizes this issue with the following explanations: 1) lack of knowledge; 2) do not experience pragmatics-driven training, which also applies to RLHF-tuned models. \n\nTherefore, this work argues for a more principled pragmatics feedback-based training of current LLMs on knowledge QAs, called Listener-Aware Calibration for Implicit and Explicit confidence. They tune the model not only using feedback on whether the answer is correct but also whether the answer is interpreted as correct by listeners. Their experiments look good in showing that their proposal is effective to address the calibration issues.", "questions": "Please take a look at the weaknesses part. Hope this can be helpful. \n\nAdditionally, I am also curious about the potential of such pragmatic-aware tuning in a variety of typical LLM tasks, beyond simply knowledge QA. If possible, can you answer this? and how?", "limitations": "Please see the weaknesses.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. This work focuses on an important problem in LLM community, the over-confidence problem. This renders the LLM application usually not trustworthy. Their experiments on some knowledge QA datasets, TruthfulQA and TriviaQA, have demonstrated the effectiveness of their method. \n2. The perspective of pragmatics-aware training is at least interesting to me, and I am excited about any formulation with respect to that.", "weaknesses": "1. I am wondering the applicability of this method on knowledge intensive tasks. Now, it sounds like this method requires you to measure the factuality of the generated responses (from your section 3.2 and Table 1). However, in real world scenarios, for example, in industry, we need to synthesize the data by ourselves, usually in the form of distillation from powerful LLMs. And at that time, we do not have ground truth answers, in other words, \"The factual texts\". How do you address this? I think this is more interesting to me.\n2. Could you consider to add more baselines? I am not sure how your methods can compare to other methods. This is very important to readers. Otherwise, your work is just interesting which implements the pragmatic framework to enhance LLM tuning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your attention to our work and for highlighting the importance of the problem we focus on and the excitement of our pragmatics-aware approach. We will address the remaining comments below\n\n1. **Application of LACIE to knowledge-intensive tasks**\n\nLike other work in training models to be better-calibrated, LACIE does require access to labeled data for determining correctness or factuality; this kind of data is widely available in many domains. Since calibration deals with the relationship between model uncertainty and model correctness, we would argue that any method either measuring or improving calibration requires access to some kind of labeled correctness data. In the scenario described by Reviewer dvWw, where users synthesize data from powerful LLMs, the “ground truth” data is provided by a teacher LLM, and thus the teacher’s answer is treated as the “correct” answer. In this case, LACIE could be applied in exactly the same way to train the model to be calibrated w.r.t. the synthetic data. The caveat here is that the model quality may be affected by mislabeled examples in the synthetic data; however, this is true of any method training on LLM-generated synthetic data (whether for calibration or other purposes). Assuming that the teacher model is strong enough to provide mostly factual answers, LACIE will work as expected. Moreover, LACIE does not need annotations for all claims in an answer, only for the final answer; **this kind of data can generally be found in existing datasets or can be collected relatively cheaply.** \n\n2. **Adding more baselines**\n\nPlease see the **general response** for a full description of the additional baseline. To summarize, we have added two versions of Tian et al. (2024)’s method as an external baseline to **Table 1 of our rebuttal PDF**. Because this method asks for a direct confidence score from the model, we compare it to our methods when using a listener model to estimate confidence (as we do for all other baselines) and when extracting the confidence directly (giving their method an advantage over LACIE and other baselines, which are mediated by the listener). \n\nWe find that while Tian et al.’s method generally outperforms the base model, **LACIE continues to outperform all baselines, with consistent improvements in AUROC and Precision across Mistral-7B and Llama3-8B**. \n\n\n\n3. **“...I am also curious about the potential of such pragmatic-aware tuning in a variety of typical LLM tasks, beyond simply knowledge QA…”**\n \nLACIE’s results show that QA-based tuning is a valuable signal for helping the model distinguish between correct and incorrect responses; our TruthfulQA results show that this has promising implications to tasks like hallucination reduction as well. Future work might explore using the signal obtained from LACIE training on QA data (which is readily available and easily annotated for factuality) to long-form generation tasks like summarization or dialogue. More broadly, given that humans interact pragmatically when using language, we believe that imbuing models with pragmatic ability is an important step to having more natural, reliable, and interpretable interactions between humans and models.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fM4fwbmJ6J", "reviewer": "Reviewer_Aw31", "summary": "This paper explores a fine-tuning strategy for optimizing confidence calibration in LLM outputs. In contrast to prior work, the authors fundamentally define this as a pragmatic problem, where performance improvements are measured based on listener's correct inference that affects their downstream task performance. The authors find that their method results in models that are better calibrated on the in-distribution dataset TriviaQA and also generalizes to the out-of-distribution dataset TruthfulQA. The effectiveness of the system is further supported by a human subject evaluation.", "questions": "This is just a minor curiosity: I'm intrigued by the second qualitative example in Table 4. In the reference case, the model says that it learned about the event in school but a thoughtful listener will know that this is incorrect. Do you qualitatively/quantitatively find that justifications that can clearly be inferred to be incorrect (e.g., based on personal learning experiences that are clearly incorrect) reduce in the fine-tuned model variant?", "limitations": "Related to my minor question above, it might be worth briefly discussing how the highly anthropomorphic language affects listener perception of the system and therefore how confidence is interpreted. Do you have any expectation whether/how this language changes under this fine-tuning regimen?", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- addresses the issue of calibration as a holistic communication-centric problem\n- thoughtful human evaluation design for downstream task\n- interesting generalization analysis\n- helpful qualitative analysis that enables insights on system outputs (which are further discussed)", "weaknesses": "None\n\n(Minor typos in lines 16 and 282)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for appreciating our human evaluation, and our analyses – we will address the remaining questions/comments in more detail below\n\n1. **“...Do you qualitatively/quantitatively find that justifications that can clearly be inferred to be incorrect (e.g., based on personal learning experiences that are clearly incorrect) reduce in the fine-tuned model variant?”**\n \nThanks for this question! Qualitatively, we did not find that this behavior was reduced through LACIE training; we will add more examples in the final version. One hypothesis for why is that the listener model (used to generate the training data) is not informed that the outputs it is seeing are from an LLM (the same is true for the human evaluation). Thus, the model (and people) do not actually know that there is no way for the speaker (a model) to have learned something in school (or other such backstories) and so the speaker is not penalized for making such statements (as long as its answer is correct). However, it would be possible to explicitly penalize these statements by instructing the listener (or human evaluators) that the speaker is a model. In that case, statements like “I learned this in school” would be dispreferred, since they should lead to low confidence because the listener would know that the model is hallucinating at least one part of the response. Thus, we suspect that pragmatic finetuning with better listeners will be able to even further improve implicit confidence markers in speaker outputs.\n\n2 **“ it might be worth briefly discussing how the highly anthropomorphic language affects listener perception of the system and therefore how confidence is interpreted.”**\n\nThanks for the suggestion – in our qualitative analysis in Figure 3 of the original paper, we find that the use of details does increase on correct examples after training; this does correspond to some extent to the anthropomorphic language referred in the comment above. Qualitatively, the listener model does often increase its probability of accepting an answer when these more human-like backstories are included; luckily, LACIE training results in an increase only for correct examples. However, to have a more faithful system, future work might explicitly penalize such responses (e.g. by using the listener prompt to downweight them) as mentioned in the response above. We will add a discussion of these effects to our final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IRkogFhM8p", "reviewer": "Reviewer_NSp4", "summary": "This paper focuses on addressing the calibration of implicit (e.g., tone of expressions) and explicit confidence markers of LLMs when providing answers in conversations. This paper proposes a method, LACIE, to cast calibration as a preference optimization problem for QA tasks. In particular, the preference data are generated by simulating a speaker model and a listener model using LLMs. The proposed method improved over the baselines for multiple LLMs and also reduced the human acceptance of incorrect answers by 47% while maintaining the acceptance rate of correct answers.", "questions": "What about non-pragmatic models as baselines? For example, using the speaker model to judge the acceptability/confidence of the answer, then preference-tuned?", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper focuses on important research problem and experimenting with an interesting setup using pragmatics\n- The paper shows empirical improvement of results over multiple LLM and even better OOD results", "weaknesses": "- The paper lacks appropriate baselines ( see questions). \n- The proposed method has a side effect of reducing the speaker accuracy results by a large margin. Currently, there is a lack of qualitative discussions and analysis on evaluating if the better abstain ability is actually worth it.\n- Different models as listeners are missing to understand the robustness of the method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for highlighting the importance of our research problem and our empirical improvements. We have sought to address the remaining questions/comments below: \n\n1. **Improvements on additional baselines**\n\nThanks for your suggestion of non-pragmatic baselines. To clarify, our main preference tuning in our original paper’s Table 1 is based on the judgment of a Mistral-7B-base listener model, which for the first set of results is the same architecture as the speaker model (which is the only model being tuned) and is not pragmatic, in that it does not consider the distribution over answers, but only the implicit and explicit confidence of the answer. \n\nTo further highlight non-pragmatic baselines, we have added two new baselines from Tian et al. (2024) (https://arxiv.org/abs/2305.14975), described in our **general rebuttal response, Table 1 of the rebuttal PDF**. These baselines are non-pragmatic in that they do not take into account a listener model and act just as a speaker. In the \"no listener\" setting we also evaluate Tian et al.’s method by directly extracting the confidence (rather than using a listener model), i.e. using a non-pragmatic evaluation. Our results here show that LACIE outperforms the additional baseline (both when applied to base models and chat models), with a consistent increase in precision and AUROC over the new baseline across  both Mistral-7B and Llama3-8B. This highlights the importance of pragmatic listener-aware training.\n\n2. **Qualitative analysis on abstained answers**\n\nThanks for this question. We have added a qualitative analysis to our **general rebuttal** and the **rebuttal PDF in Figure 1**. Here, we show the answer diversity from the base model on abstained and non-abstained examples. We find that the base model has higher answer diversity on examples that LACIE abstains on (orange), and lower diversity on examples that LACIE does not abstain on (blue). This indicates that LACIE training allows the model to identify examples that have high uncertainty (i.e. high answer diversity) and then abstain on these examples. In other words, it does seem that abstention is worth it, since the model is abstaining on high-uncertainty examples. \n\n3. **Adding different listener models**\n\nThanks for this suggestion; based on this comment we have added an ablation using Llama3-8B as the listener model. The full experiment is described in **our general response** and shown in **Table 2 of the rebuttal PDF**. \n\nTo summarize, we find that **LACIE with a Llama3-8B listener still increases the precision and AUROC over the base model**, but not as much as with a Mistral-7B listener. We do however find that it improves the ECE the most. Taken together, these results indicate that LACIE is compatible with different listener models.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FFfNFL0hnI", "reviewer": "Reviewer_rP95", "summary": "This paper tackles LLM miscalibration (overconfidence in wrong answers) with a listener-aware fine-tuning method. The authors hypothesise that LLMs are overconfident for two reasons: a lack of knowledge of what is correct, and a lack of pragmatic grounding (knowing how your utterance is perceived by a listener). Their fine-tuning method addresses both, by preference tuning with a preference function that rewards cases where the model accurately expresses confidence (no matter whether the answer is right or wrong) and penalizes when inaccurately expresses confidence. The authors fine-tune three LLMs of different sizes (7B - 70B) on triviaQA, and compare their method to the base models, chat versions of the base models (finetuned with human preferences generally), and models finetuned with an ablated version of their method that simply prefers correct answers over incorrect ones. They find that their method significantly improves calibration of the models for all sizes. With an LLM listener, the method improves precision (meaning listeners accept less wrong answers), but harms recall (meaning less of the right answers are found by the model). The authors show that this is mainly due to a higher rate of abstains in the model fine-tuned with their method. With a human listener though, the method improves precision and recall stays the same as the base model, which means that humans are less quick in rejecting underconfident answers that are correct. The authors further show OOD generalisation of their method and do a qualitative analysis of the expressed confidence for correct and wrong answers of their model and a baseline.", "questions": "- Is the following intuition about the results correct: the recall in the experiment with the LLM listener goes down more dramatically than for humans because the LLM listener rejects underconfident answers that are correct more often? i.e. the true rejects are relatively higher for LLM listeners than for humans?\n\nI'm putting some small nits here:\n\n- when you discuss human results in section 4.2 page 6, it would be useful to briefly mention that these will be discussed in more depth in section 4.3 (or simply refer to 4.3)\n\n- Figure 3 is really great in terms of information, but not so great in terms of presentation. Want to be able to compare the correct/incorrect bars in length directly.\n\n- Table 3 is not interpretable from the table and caption only, what do the numbers refer to? From the text one knows its judges but maybe use the extra page to add some information in the caption.", "limitations": "The authors discuss limitations sufficiently, but in the appendix. As mentioned in the weaknesses, I would rather like to read about them earlier, and in the discussion.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The authors address an important problem of overconfidence by LLMs in a time where they are used as question-answering tools by many, and motivate their research extensively.\n\n- Using pragmatic grounding to improve calibration is a neat idea and executed very well in this paper\n\n- The authors have an excellent experimental setup, comprehensive and sound, essentially covering everything that's necessary to interpret and situate the results. They look at different model classes, different model sizes, they compare to all the necessary baselines, they show results in humans (which is what we care about in the end), and they show OOD generalisation to a different dataset than used in training. They also do a comprehensive analysis of qualitative markers of confidence in the outputs of their models.\n\n- The results for calibration are positive across all speaker models; calibration is significantly increased and the model learns to abstain when it is uncertain.\n\n- The results with humans are also positive; they show that their finetuning method also causes an increase in precision here (meaning humans accept more correct answers from their model than baselines), but the recall for humans doesn't decline compared to the base model (meaning humans do not reject right answers more often than for the baseline)", "weaknesses": "- There is a significant decrease in recall and accuracy of the models trained with this method. This decrease itself is not the weakness, because as the authors argue, this seems to some extent due to the base model \"guessing\" correctly more often. Wrong guesses do not impact recall, but right guesses do. Additionally, the authors show that the base model gets a significantly lower accuracy on the examples their model abstains from, meaning it's probably less certain. The weakness I want to mention here is that this particular result is not mentioned in the abstract or introduction, where the other results are discussed extensively. I believe a more balanced abstract and intro will make this paper stronger, as the decline in recall and accuracy is extensively discussed and in the rest of the paper and doesn't mean the method isn't effective at increasing calibration, it's just a trade-off and a potential avenue for future work. In short, I propose the authors incorporate this result in the abstract and intro.\n\n- The calibration and increased precision and abstaining of the models are very well analysed and convincingly presented, but for the decline in accuracy the authors mainly look at the percentage of abstains by their model that the base model gets wrong, to note that \"abstention is correctly correlated with cases where the model does not know the correct answer\". The base model still gets 30% of the abstains correct though, and I would be interested in some more analysis of these. How likely are these guesses by the base model, and how likely are these instead cases where the base model does actually know the answer but LACIE finetuning surpresses that? This could be analysed for example by looking at the logits of these answers in the base model and comparing that to the other correct answer logits from the base model. Do they on average assign lower probability to the (first token) in these answers? That would indicate lower confidence / guessing. Alternatively, one could sample from the base model with a temperature and see if the model gets those questions wrong more often than other questions.\n\n- I would like to see a table (in the appendix would be OK) with all true/false acceptances/rejects from each experiment. This would help interpret the results (e.g. I expect false acceptances to have gone down in Table 1 for LACIE compared to the base model, more dramatically than in Table 2)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your comments and questions, and for highlighting our “excellent experimental setup” and positive results. We’ve sought to address each point in the review below:\n\n1. **Highting decrease in recall/accuracy due to abstention in the intro**\n\nWe agree that this is an important point to highlight – we will propagate our discussion of the tradeoff between recall and precision from L263-269 to the introduction and abstract in the camera-ready version of the paper. \n\n2. **Further analysis of abstained cases**\n\nThank you for the suggestion -- we have added the above-described analysis of the abstained cases in **Figure 1 of our rebuttal pdf**, with the full experiment described in our main rebuttal. Because of the complicated nature of estimating confidence using token probabilities on long-form generation, we follow the second suggestion and measure the diversity of answers when decoding with a high temperature. Specifically, we sample 100 abstained and 100 unabstained examples and decode 40 generations for each, extracting their answers and counting the number of unique answers among them. We find that the base model generally has substantially more unique answers on abstained examples, indicating that these have higher uncertainty in the base model, and that LACIE training allows the model to detect this uncertainty and abstain appropriately. \n\n3. **Adding TP/FP/FN numbers**\n\nWe have added this to **Table 3 of the rebuttal pdf**; for the sake of space, we have only included Mistral-7B but the other models are similar in their trends. In summary, we find that the increase in precision in Table 1 of the original paper for LACIE-trained models is indeed driven by a ~56% reduction in False Accepts, from ~250 to ~109. The reduction in recall is driven by a ~34% increase in False Rejects from ~115 to ~173. These are indeed more dramatic trends than those seen in Table 2 of the original paper, where we saw a 47% decrease in False Accepts and no significant increase in False Rejects. \n\n4. **Question 1: do humans reject underconfident answers less?**\n\nWe believe that this is indeed the case, and it points to a direction for future work, which is aligning the *listener model* more to human intuition. While the listener model generally is a good proxy for people, there are clearly deviations (e.g. rejecting answers that humans might actually accept). Another possible reason for the difference is that the listener model does not see the actual answer, while the human annotators do. While we filtered out examples that human annotators said they knew the answer to (L581-583), it could still be that they did not know the answer enough to report knowing it, but know what kind of answer is plausible (e.g. even if you do not exactly know the capital of Germany, you may know that “Paris” is not a plausible answer). \n\n### Smaller points: \n- We will add a reference to the human results discussion in 4.2 \n- We will update Figure 3 of our final version accordingly (excluded from the pdf currently because of space constraints) \n- We will add the following more descriptive caption for Table 3 in the original paper: “TruthfulQA performance using Mistral-7B as measured by finetuned Llama3 judge models for truthfulness and informativeness. Higher truthfulness reflects lower rates of hallucination. Informativeness reflects topicality, and penalizes abstention.”", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to thank the reviewers for their attentive and detailed reviews, which highlight our work’s importance and excitement (Reviewers dvWw, NSP4, rP95), the effectiveness of our method (Reviewers dvWw, NSp4, rP95) and the strength of our experimental setup and analysis (Reviewers Aw31, rP95). In our general response, we address the points shared between reviewers, with more detailed responses to each reviewer’s specific questions below. **We have also uploaded a pdf with tables and figures, which we refer to in our responses.** 

We describe each contribution in more detail below.
1. **Improvements over new baselines**: In response to Reviewers dvWw and NSp4, we have added two new baselines to **Table 1 of our rebuttal pdf**. Our method, LACIE, continues to outperform all baselines, including these new competitive ones.
2. **Analysis of abstained examples showing higher base-model uncertainty on abstained questions**: In response to Reviewers NSp4 and rP95, we have added an additional analysis in **Figure 1 of the rebuttal PDF** on abstained examples showing that abstained examples have higher answer diversity before abstention, i.e. more uncertainty from the base model. 
3. **LACIE works with an additional listener model**: In response to Reviewer NSp4, we have added an additional ablation to **Table 2 of the rebuttal PDF** where we compare different listener models; we find that LACIE training does transfer to a Llama3 listener but works better with Mistral (which we used in our main experiments). 

## New Baselines
We’ve added two additional baselines in **Table 1 of our rebuttal PDF**. The baselines use Tian et al. (2024) (https://arxiv.org/abs/2305.14975)’s method for obtaining better calibration from instruction-tuned LLMs, which is effective on TriviaQA. Tian et al. prompt models to include an explicit confidence score with their answer, and thus lack  a listener model. Therefore, we compare against two settings, with and without a listener model. In the 1st setting, we pass the outputs from Tian et al.’s prompt through our listener. This setting is directly comparable to the other baselines and to LACIE (all evaluated according to the listener’s confidence). We additionally include another version of Tian et al. that directly extracts the confidence score from the output (rather than using a listener model). This baseline has an advantage in avoiding the listener model, but only works if the outputs have explicit confidence scores (as Tian et al.’s outputs do). The two new baselines are marked in orange in **Table 1 of our rebuttal PDF**.

Because Tian et al.’s method requires an instruction-tuned LLM, we additionally add a comparable row that combines LACIE training with instruction-tuned models (added in blue). We include results for both Mistral-7B and Llama3-8B and find that they generally outperform the non-chat variants. We show that both Tian et al. baselines improve AUROC and recall over the untrained chat and base models for Mistral-7B and for Llama3-8B, and that the “no listener” variant improves precision for Llama3. This indicates that these are competitive baselines. However, for both Llama3 and Mistral, **LACIE has substantially higher AUROC and precision, driven by LACIE’s ability to express uncertainty appropriately**. For ECE, we note that LACIE generally beats the new baselines, but in one case Tian et al.’s no listener baseline has lower ECE. Qualitative inspection reveals that this baseline generally produces bimodal scores (close to 0% confidence or 100% confidence with few in between). This means that the ECE computation for Tian et al.’s baselines generally have fewer bins, making them non-comparable to the other baselines (denoted by an asterisk). Past work has noted the bin hyperparam as leading to instability in the ECE metric (https://arxiv.org/pdf/1904.01685), and therefore we emphasize LACIE’s improvements over all baselines as measured by AUROC and precision. Note also that LACIE can handle both implicit and explicit calibration (which are both key to reliable human-AI interactions) while Tian et al. can only handle explicit confidence. 

## Analysis of abstained examples

In **Figure 1 of the rebuttal PDF**, we show an analysis of base model uncertainty on abstained examples, finding that models abstain on higher-uncertainty examples. We measure answer diversity on examples where a LACIE-trained Mistral-7B model abstained vs. where it did not abstain, finding that abstained answers originally had higher answer diversity, indicating base model higher uncertainty. Specifically, following Reviewer rP95’s suggestion, we have taken the following steps:
1. We sample 100 TriviaQA test examples where the Mistral-7B-base+LACIE model abstained and 100 where it did not.
2. For each example, we prompt a Mistral-7B-base model to generate 40 answers with temperature = 0.7. We then use Mistral-7B to extract the answer (as we do for LACIE). 
3. We tally the number of unique answers per question. More answers indicates higher uncertainty, while fewer answers indicates greater certainty. 

We see a distinct separation between abstained (orange) and non-abstained (blue) examples, with fewer unique answers on average for non-abstained, and a larger number of unique answers for abstained. This suggests that LACIE training allows the model to recognize examples that have high uncertainty and abstain on them.  

## Additional listener model 

Following the Reviewer NSp4’s suggestion, we have added an experiment in **Table 2 of the rebuttal** with an additional listener model.

We see that using a Llama3-8B listener to train Mistral-7B leads to improvements over the base model in AUROC and precision, but that these improvements are smaller than when using Mistral-7B as both the speaker and listener. However, Llama3-8B leads to the lowest ECE. Overall, these results indicate that LACIE is robust to the choice of listener model.

Meta-review: This paper addresses the issue of LLM miscalibration, particularly their tendency to be overly confident in incorrect answers. The authors attribute LLMs' overconfidence to two main factors: (1) insufficient knowledge of what is accurate and (2) a lack of pragmatic grounding, which refers to an understanding of how their responses are interpreted by a listener.

The reviewers generally agree that this work tackles an important research problem and offers an interesting and novel approach by using pragmatics. The idea of employing pragmatic grounding for confidence calibration is insightful and results in empirical improvements across multiple LLMs, yielding even better out-of-distribution results. For these reasons, most reviewers favored the paper’s acceptance, and the weaknesses identified are relatively minor and were mostly addressed during the discussion period (e.g., about baselines). I have little to suggest for the camera-ready paper, except that I encourage the authors to include their additional results in the final manuscript.
Decision: Accept (poster)