Mind the Confidence Gap: Overconfidence, Calibration, and Distractor Effects in Large Language Models
Authors: Prateek Chhikara
Venue: Accepted by TMLR ()

Abstract: Large Language Models (LLMs) show remarkable proficiency in natural language tasks, yet their frequent overconfidence—misalignment between predicted confidence and true correctness—poses significant risks in critical decision-making applications. We present a comprehensive analysis on calibration in LLMs across nine LLMs and three factual Question-Answering (QA) datasets, systematically comparing standard free-generation settings against structured distractor-augmented prompts. Our evaluation reveals that explicitly incorporating distractors can substantially mitigate miscalibration, achieving relative accuracy improvements up to 460% and ECE reductions up to 90%. Despite general trends, we uncover nuanced findings: large RLHF-tuned models display inherent calibration strengths but can paradoxically suffer increased miscalibration on easier queries, whereas smaller models benefit disproportionately from distractor prompts but remain significantly miscalibrated. Through detailed analyses across question types, we identify persistent calibration failures, particularly in person-based queries. We conclude with concrete recommendations—targeted fine-tuning, structured prompting, and strategic model choice—to ensure reliable, trustworthy LLM deployments.

Published in Transactions on Machine Learning Research (12/2025)
Mind the Confidence Gap: Overconfidence, Calibration, and Distractor Effects in Large Language Models
Prateek Chhikara
pchhikar@uus.eddu
University of Southern California, Los Angeles, USA
Reviewed on OpenReview: https: // openreview. net/ farmm id= LyathiDdZZ
Abstract
Large Language Models (LLMs) show remarkable proficiency in natural language tasks, yet their frequent overconfidence meesaiieeneeemiiellggnnnn between predicted confidence and true correctness- significant risks in critical decision-making applications. We present comprehensive analysis calibration in LLMs across nine LLMs and three factual OneestonnAnswerinng (QA) datasets, systematically comparing standard free- generation set- tings against structured distractor-augmented prompts. Our evaluation reveals that ex- plicitly incorporating distractors can substantially mitigate miscalibration, achieving rel- ative accuracy improvements up to 460% and ECF reductions up to 90%. Despite gen- eral trends, we uncover nuanced findings: large RLHF-tuned models display inherent cal- ibration strengths but can paradoxically suffer increased miscalibration on casier queries, whereas smaller models benefit. disproportionately from distractor prompts but remain sig- nificantly miscalibrated. Through detailed analyses across question types, we identify per- sistent calibration failures, particularly in person-based queries. We conclude with con- crete recommendations- targetee fine-tuning, structured prompting, and strategic model choice to ensure reliable, trustworthy LLM deployments. Code is publicly available at: http://hhhhtccoooeeeeeeeeaaaaaaa
Perfectly Calibrated Model
Poorly Calibrated Model
1.0
Actual Accurucy
0.8
Actual Accuracy
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
50
100
0
50
Predicted Confidence (%)
Predicted Confidence (%)
overconfidence
received Award 2010?
Rosenblatt
I am 93% sure that it is Geoffrey Hinton.
User
Correct Answer: Michio Sugeno
LLM
Figure 1: An instance from SimpleQA dataset where an LLM assigns high confidence an incorrect answer.

---

Published in Transactions on Machine Learning Research (12/2025)
1 Introduction
Large Language Models (LLMss) have significantly advanced natural language understanding, achieving state- of-thc-art results across tasks including conversational AI (Skjuve al. 2024 Zhang 2024 ; scientific dis- covery [Kumar 2024), and multimodal systems (Zhang el al. 2023 Chhikara et al. 2024; Zhang et al.) As LLMs increasingly guide critical decisions in sensitive domains such as healthcare, finance, and law-thr the reliability of their confidence estimates bocomes paramount. Misalignment between model confidence and ac- tual correctness, known as miscalibration, poses severe risks, potentially eroding user trust and causing costly or hazardous errors (Dhuliawala et al. 2023 Geng et al.) 2024). For example, as illustrated in Figure] 1 when asked "Who received the IEEE Frank Rosenblatt Award in 2010P", a leading LLM confidently but incorrectly answers "Geoffrey Hinton" " with a confidence of 93%, despite the correct answer being "Michio Sugeno". " Such pronounced overconfidence can lead users to mistakenly trust erroneous cesppuss-pprttclarryy prob- lematic in high-stakes applications such as medical diagnoses or financial decisions, Well-calibrated models, on the other hand, report confidence scores accurately reflecting their true reliability, thus enabling systems to flag uncertain predictions for human oversight and significantly mitigating real-world risks.
format whether through retrieval-augmennted candidate spans, knowledge-graph sibling entities, sell-consistency checks, or ensemble methods to improve answer selection. However, these structured, distractor-ricl settings introduce novel challenges for confidence scaling, Platt scaling, isotonic regression (Guo et al. 2017)) were not originally designed to address. Although such methods have proven effective on small- to medium-scale neural networks, their applicability to today's large-scale LLMs under real-world, distractor-heavy conditions remains unclear. Prior work has examined isolated factors-mode scale, architecture (dense vs. mixture of experts (MoE)), and finc-tuning regime (supervised fine-tuning (SFT) vs. reinforcement learning from human feedback (RLHF)) (Leng et al.) 2024: Li et al. 2024) but has not extensively examined how explicit distractors, ubiquitous in deployed QA systems, affect calibration accuracy and confidence ranking for modern LLMs.
Modern Quostion-Answering (QA) pipelines adopt
challlnnsss that classical post-hoc calibration methods (temperature
Mitigating overconfidence through distractors Research in cognitive psychology demonstrates that human overconfidence can be reduced by explicitly considering alternative answers before making decisions (Lord et al.] 1984 Mussweiler et al. 2000). Inspired by this coosider-llee-oppsstte strategy, we investi- gate whether presenting LLMs with plausible distractors similarly mitigates their systematic overconfidence and enhances calibration. To be specific, we conduct the first large-scale empirical study of LLM cali- bration, comparing performance under standard (free- (treeegenerttio)) and ietrractor-uugmette settings. Our contributions fourfold: (1) introduce unified calibration benchmark evaluating nine state-of-tho-art LLMs spanning model sizes (8B 70B to greater than IT), architectures (dense vs. MoE), and fine-tuning methods (SFT vs. RLHF) across three factual QA datasets (SimpleQA, FaVTQ, TriviaQA): (2) we pro- pose a structured distractor augnented evaluation paradigm, where models select answers from one correct and multiple plausible incorrect options, enabling simultancous assessment of accuracy improvements and Expected Calibration Error (ECE); (3) we perform fine-grained analyses across question types (e.g., person, date) identify conditions of severe miscalibration; and (4) we systematically disentangle how scale, tuning regime, and architecture independentlyy influence model calibration and responsiveness to distractors.
2
Related Work
Prompt-based approaches that verbalize confidence (Tian et al. 2023 Mielke et al., 2022 Lin et al.) or aggregate multiple outputs (Xiong ct al.) have demonstrated effectiveness, particularly for black-box models.
Intrinsic calibration Methods directly elicit uncertainty from LLMs.
Finc-tuning strategics Approaches such as calibration-aware RLHF (Leng ct al. 2024)) and Mixup-style data augmentation (Park & Caragea| 2022) aim to optimize calibration during training. While these methods reduce FCF,, they remain sensitive to distribution shifts (Liu et al.p.
2

---

Published in Transactions on Machine Learning Research (12/2025)
Chen et al. 2023 show that calibration emerges early in sell-supervised pre-training, and Zhu et al. (2023) demonstrate that instruction tuning and RLHF can preserve or even enhance these gains. Jiang et al. 2021) find that post-hoe methods like temperature scaling often fail to align confidence with accuracy in factual QA. However, the effects of explicit distractor-basod prompling on LLM calibration remain unexplored.
Calibration during pre-training and alignment
Post-hoc calibration Techniques adjust predictions after training, with classical approaches like tempera- ture scaling (Guo et al. 2017) still underexplored for contemporary LLMs. Recent studies indicate persistent overconfidence
larger scales, highlighting potential degradation in calibration performance with increased model size (Zhou et al. 2024a b).
Despite extensive research, existing literature lacks systematic exploration of structured distractor effects on calibration. Motivated by psychological findings that considering alternative answers can reduce human overconfidence, our work introduces a structured distractor dustractor-uugmenter evaluation framework. Unlike prior methods, we empirically investigate how explicit distractor scenarios common in practical applications such as retrieval-angmenneed generation and multiple-choice contexts tereetts--mppct LLM calibration. Additionally, we conduct detailed, fine-grained analyses across different question types (e.g., person, date, place), uncovering context-dependent calibration challenges previously overlooked. Our findings thus address critical gaps in calibration research, offering practical insights to enhance the reliability and trustworthiness of LLM deployments.
3 Experimental Setup
3.1 Evaluation Datasets:
SimpleQA We use the SimpleQA datasel (Wei et al. 2024), which provides reliable benchmark for evaluating LLM factual accuracy and calibration. Comprising short, fact-seeking queries with clearly defined correct answers, SimpleQA enables precise measurement of model confidence and alignment with factual correctness. Its high-quality annotations, verified by multiple independent. AI trainers, ensure accuracy and unambiguity, making it well-suited for calibration assessment. The dataset contains 4326 question-answer pairs.
Fa VIQ Park et al. 2022 We select data points from test subset of the R-set. The dataset initially contains 5877 data points, out of which we focus exclusively the 2922 data points labeled as "supports," " indicating that the provided answer is correct. FaVIQ is particularly appropriate for our experiments duc to its con- struction methodology derived from real-world information-sceking questions. This design inherently reduces strong lexical biases found in other crowdsourced datasets, promoting nuanced semantic understanding.
TriviaQA We use the TriviaQA dataset (Joshi et al. 2017], for evaluating open-domain question answering and factual knowledge retrieval. For our experiments, we select first 1000 question-answer pairs from the validation split of the rc. www.eeeno_oooteeeeee restricting selection to the no-content class, focus settings where models must rely purely prior knowledge without the assistance of retricved supporting context, isolating intrinsic model calibration behavior.
subset, ensuring diverse yet controlled evaluation set. By
3.2 Evaluation Methods
Let our evaluation set be
S ((g, a:))
where da is the i-th question and a; its ground-truth answer. We compare two prompting regimes:
3

---

Published in Transactions on Machine Learning Research (12/2025)
frec-generation bascline (N) We prepend each 9s with fixed prompt template T and let model generate completion answer and the confidence:
( (()) LLM( (lq),
where the model's final answer is (M and the associated confidence is c,
Distractor-augeente setting (D) For each (gi, aj) we sample three distractors {dr,, d. dr.3 and form the choice list.
C. shuffle U {di.))=1)
We then feed the model:
(D) ..cP) = =LLM(TD 11 II.:),
yi
and extract the model's final answer (!)) and record the associated confidence c; cf Prompt templates AD are provided in Appendix A Our distractors were generated using GPT-4o-mini with a carefully designed prompt to ensure they were factually incorrect yet contextually plausible. Specifically, for each question-answer pair in the datascts, we used GPT-4o-mini to generate three distractors that (i) matched the expected answer type (e.g., dates for date-based questions), (ii) remained distinet from the correct answer, and (iii) maintained comparable specificity and context. To further validate plausibility, we manually inspected over 500 randomly sampled examples from the SimpleQA dataset, confirming that the generated distractors were consistently plausible and contextually relevant.
Why Elicited Confidence? We measure confidence via elicited self-reports (0 (0-1 because they capture task-level belief ("How sure are you that your answer is correct?") rather than token-level fluency artifacts. Prior work shows that verbalized probabilities can better reflect correct than raw token likelihoods, which are sensitive to phrasing and tokenization (Lin et al... In RLHF--unne systems, elicited confidence has also been observed to track calibration more reliably than log-probabilities which can degrade post-alignment (Tian et al., 2023 This aligns with "linguistic calibration" evidence that making models state their confi- dence reduces overconfidence and improves user-facing transparency (Mielke et al., 2022). Elicitation also the only uniformly available signal across the nine black-box and open-weight models we evaluate, enabling apples-to-apples comparison. We do not claim it is the sole or optimal measure: logit-based margins margins/entropy /entropy and self-consistency (majority-vote variance) are complementary signals. Our findings should therefore be interpreted in the context of clicited confidence; we add this clarification to promote comparability and to reflect the literature's guidance on when verbalized uncertainty is informative.
3.3 Selected LLMs
We select nine representative models, spanning three major families OpenAl's GPT-4 series, Meta's LLaMA lineage, and two leading open-weight models (Gemma-2 and Qwen-qwq). We select these models from OpenA and GloqCloud" 2 API services. More details about the selected models and their taxonomy are in Table
GPT-4 Family The GPT-4 family consists three variants: GPT-4o, GPT-4-turbo, and GPT-4o-mini (a
smaller, assumed to be 8B-parameter model). All three support an extended 128K-token context window and are instruction-tuned via SFT followed by RLHF to optimize conversational quality. They differ primarily in total parameter count-leading to different trade-offs in inference latency and compute cost.
LLaMA Lineage Meta's LLaMA-3 (ubeyy et al. 2024 series spans three dense checkpoints: an 8B base with 8K window, an 8B "Instant" assistant- model with 128K window, and a 70B base (8K window). Each uses Grouped--uery Attention (GQA) (Ainslic et al.) for efficient long-context processing: all these variant undergoes SFT and RLHF. In contrast, LLaMA-4-Scone 17t adopts a 16-expert MoE transformer which supports up to 10M tokens, and fine-tuned with both SFT and RLHF for reasoning.
https://openal. com
bttps://groq.com
https://ai, meta. wwwp://ww...oooo//aaaaaaaaeaaaaa hww.mmmttiiiiii-iiiilieeeeeeee intelligence/
4

---

Published in Transactions on Machine Learning Research (12/2025)
Table 1: Taxonomy of selected LLMs showing differences in training and fine tuning approaches, where FT (fine-tuning) and IT (instruct-tuning).
Model (Params Context)
Architecture
Dataset Type
Training Strategy
FT
IT
GPT-4o
Dense
Multi-modal (web text, code, images, andio transcripts)
Pre-training +
(undisclosed 128K)
RLHF
GPT-4o-mini
Dense
Multi-modal (weh SFT = RLHF text, code, images)
(undisclosed / 128K)
GPT-4-turbo (undisclosed 128K)
Dense
Multi-modal (weh text, code, images)
Pre-training + SFT
+ RLHF
LLaMA-3. 1-.----BInntann (8B 128K)
Dense (GQA)
tokens Public (web, code, multilingual)
Pre-training + SFT
+ RLHFF
LLaMA-3-8BIInstan (8B 8K)
Dense (GQA)
15T tokens Public (web, code, multilingual) 15T tokens Public (web, code, multilingual)
Pre-training + SET RLHF
LLaMA-3-708-Intzwee (70B / 8K)
Dense (GQA)
Pre-training + SET RLHF
LLaMA-4-Scout-178 (17B / 10M)
MoE (16 experts)
40T tokens Mixed (text + vision,
MoE Pre-training + SFT = RLHF
multilingual)
Gemma2-98-25
Dense (GQA,
tokens Public (web, academic)
Distillation + SFT
(9B / 8K)
interleaved
+ RLHF
socal-global)
Quen qunenn-uu--22
Dense (GQA)
tokens Public (multilingual; web text, code, scientific lit.)
Pre-training + SFT
(32B/ 131K)
Open-Weight Alternatives Gemma2-9b-it TTam al.) 2024) leverages knowledge-distillttion pre- training on 8T tokens with interleaved local global and group-query attention followed by instruction fine-tuning (SFT Direct Preference Optimization (DPO) (Rafailov et al. 2023 within 8K-token context. By contrast, Quen-qwq-32b (Hui et al.
2024 Yang et al. 2024 is 32B-33--pap- paraamree dense model (64 layers,
Rotary Positional Embedding (RoPE)
(Su el al. 2024
SwiGLU (Dauphin et al. 2017)) pre-trained on 18T
multilingual tokens without RLHF.
The GPT-4 family and LLaMA-3 series are dense models trained on massive multimodal or text-only corpora, each undergoing both SFT and RLHF before instruct-tuning. In contrast, LLaMA-4-Scout employs a 16- expert MoE design over mixed vision-text data, while Gemma2-9b-it and Qwen-qwq-32b explore distillation- based and pure SFT regimes, respectively. Despite these varied strategies, all nine models receive dedicated fine-tuning and instruction-tuiing optimize performance and calibration in downstream QA and conver- sational tasks.
3.4 Evaluation Criteria
Following prior work, we use GPT-4o-mini as an LLM-based judge to classify responses as CORRECT, INCORRECT, or ATT_ATTDD (Pacer et al. 2023 Wei et al.) 2024 Chhikara et al. 2025). response is CORRECT if fully captures the gold target's 's key information without contradiction, allowing minor varia- tions in wording, order, hedging. It is INCORRECT it contains factual errors, contradictions, or mislcading speculation, even if hedged. NOT ATTEMPTED applies when response lacks essential information without in-
5

---

Published in Transactions on Machine Learning Research (12/2025)
Table 2: Performance metrics of LLMs in the Normal (A) and Distractor (D) settings on the SimpleQA, Fa VIQ, and TriviaQA datasets, including accuracy (sarress; NOT NOT_ATTEMPTED (m)), ECE, and the number of helped (Dhelped) and harmed (Dhormed) instances with their percentages.
Dataset / LLMs
Neorrett
Nua
NEce Dcorrect
Dua
DECE
Dheiped
D Dharmed
SimpleQA
0PT-4o-mini
8.46%
6.80%
0.750
47.43%
0.02%
0.320
1644 (93.78%)
109 (6.22%)
OPT-4-turbo
20.37%
6.17%
0,612
65.40%
0.00%
0.165
1877
(95.86%
81 (4.14%)
CPT-4o ?
35.14%
7.88%
0.450
73.42%
0.02%
0.037
1569
(91.97%)
137 (8.03%)
0
5.58%
18, 78%
0.799
64.6%
0.12%
0.367
1355 (95.20%
67 (4.71%)
LLaMA-3-BB-9192
4.79%
21 20%
0.810
44.01%
0.55%
0.361
1382 (95.57%)
62 (4.43%)
LLaMA-3-70--8192 00
12.73%
16.46%
0.760
65.81%
0.25%
0.239
1587 (95.09%)
82 (4.91%)
LLaMW-4-scout-17b 00
6.70%
8.76%
0.631
50.30%
0.02%
0.285
1763 (95.40%)
85 (4.60%)
Gemma2-98-it
5.48%
33.29%
0.799
45.58%
1.78%
0.367
1143 (94.38%)
68 (5.62%)
Quen-quq-32b v7
7.59%
3.99%
0.680
51.68%
0.00%
0.253
1784 (96.48%)
65 (3.52%)
VaV
GPT-4o-mini
47.19%
4.08%
0.426
69.73%
0.24%
0.161
722 (85.85%)
119 (14.15%)
GPT-4-turbo
54.76%
5.79%
0.357
80.07%
0.31%
0.062
682 (98.81%)
45 (6.19%)
CPT-4o ?
56.20%
4.83%
0.315
81.37%
0.21%
0.036
688 (93.73%)
46 (6.27%)
LLaMA-33118b-instaa 00
36.27%
6.16%
0.532
60. 60%
0.31%
0.267
776 (80.83%)
184 (19.17%)
LLAMA-3-9b-8192 00
30.85%
4.81%
0.587
58. 53%%
0.68%
0.282
892 (86.02%)
145 (13.98%)
LLaMA-3-700-8192 00
44.95%
4.79%
0.463
72.85%
0.76%
0.139
715 (91.2%)
68 (8.68%
LLaMA-44-cout-170b
39.05%
3.74%
0.499
67.85%
0.28%
0.218
861 (89.41%)
102 (10.59%)
Gemma2-9b-it
35.80%
10.42%
0.581
58.95%
1.28%
0.300
607 (82.25%)
131 (17.75%)
Qwen-quq-32b vt
38.34%
3.62%
0.530
68.5-4%
0.25%
0.200
850 (92.69%)
67 (7.31%)
TriviaQA
GPT-4o-mini
81.13%
0.12%
0.104
87.81%
0.00%
0.065
75 (77.32%)
22 (22.68%)
GoT-4-turbo
90.38%
0.36%
0.025
95.43%
0.00%
0.048
43 (84.31%)
(15.69%)
GPT-4o
89.66%
0.24%
0.071
95.55%
0.00%
0.083
48 (85.71%)
(14.29%)
LLaMA-B.1180-innttaat
75.46%
0.31%
0.153
81.35%
0.00%
0.101
119 (67.61%)
57 (32.29%)
LL.MMA3--b-81992 00
67.41%
0.51%
0.221
78.93%
0.00%
0.113
166 (74.11%)
58 (25.89%)
LLaMM-3-700-88192 00
82.86%
0.30%
0.070
91.43%
0.00%
0.026
103 (83.74%)
20 (16.26%)
LLaMA-4-scoutt117b
77.92%
0.40%
0.141
86.87%
0.00%
0.079
127 (74.71%)
43 (25.29%)
Gemma2-9b-it G
10.10%
1.41%
0.230
81.55%
0.00%
0.107
150 (75.76%)
48 (24.24%)
Qwen-quq-32b sp
75.03%
0.62%
0.158
88.08%
0.00%
0.042
133 (86.93%)
20 (13.07%)
troducing errors, including vague or evasive answers. We experiment with using the same LLM for both prediction and judgment, finding that smaller LLM judges often misclassify responses or hesitate to assign NOT ATTEMPEDD when no valid answer is generated. Manual inspection confirms these issues, and further details are provided in the Appendix B
3.5 Evaluation Metrics:
To evaluate performance, we measure correctly answered questions for both variations (N and D). For calibration assessment, use ECE to quantify the misalignment between model's predicted confidence and actual accuracy. A well-calibrated model produces confidence estimates that closely match its true correctness, with an ECE of zero indicating perfect calibration. Following (Nacini et al. 2015), we compute ECE using empirical binning (bin size 0.1) to ensure a robust measurement of miscalibration. Additionally, we define two complementary metrics: Prelped, denoting instances where the model failed under the A setting but succoeded when distractors were added (N; = 0 and D, = 1); and D Dharmeds capturing the reverse- cases where the model initially answered correctly under N but erred when distractors were introduced (N, = and D; 0)-
6

---

Published in Transactions on Machine Learning Research (12/2025)
4pi-4o
8-c-4-tubo
201-40
1.
1.0
1.0
0.6
0.4
0.4
0.4
20
441
.01
100
20
50
80
100
0
20
601
80
100
Haama-5. -8h-insstam
pemmaz--9h-i
1.0
1.0
0.8
0.88
0.8
0.4
0.4
0.4
0.2
0.2
0.2
20
40
-00
100
20
50
8.0
100
20
40
60
90
1.00
Baama--70b-3 192
1.0
0.88
0.35
0..8
0.2
0.2
0.2
Figure 2: Reliability diagrams (RDs) showing calibration performance in M (o) and D (o) settings on the
SimpleQA dataset. (y-axis: actual accuracy, x-axis: predicted confidence)
4 Experimental Results and Analysis
4.1 Quantifying Baseline Calibration of LLMs
We first quantify each model's out-of-the-box accuracy (Neorrect), NOT NOT_ATTEMPTED (Nna), and NECE on three benchmarks: SimpleQA (a deliberately hard, concise factoid task), FaVIQ (moderate difficulty), and TriviaQA (an easier, QA dataset). Table 2 reports the full metrics.
On the hardest SimpleQA benchmark where direct "off-the-shelf" generation is inherently complex even GPT-4o attains only 35% accuracy (ECE 0.45, NOT _OT_ATTEMPTED 2 8%). Larger GPT-4 variants have been pretrained on vastly more tokens (and may even have encountered similar questions), yet their calibration loss remains on the same order as smaller models. This despite scale and pretraining clame-indicaterr persistent overconfidence across sizes challenging questions.
By contrast, the easy TriviaQA setting reveals the limits of overconfidence: GPT-4o's accuracy rises to 90% with ECE ~ 0.07 and ncar-zero NOT ATT_ATTM,,EDD while smaller or open-source models tighten their reliability curves more markedly. In other words, on simpler, context-rich queries, larger models not only answer correctly but also exhibit proportionally less overconfidence compared to hard benchmarks. FaVTQ again falls between these extremes, with both accuracy and ECE interpolating smoothly.
Comparative analysis across families shows that GPT-4 variants consistently occupy the "high accuracy, low ECE, low NOT ATTEMPTED" regime all datasets. Small LLaMA-3 models, in contrast, post single-digit

---

Published in Transactions on Machine Learning Research (12/2025)
accuracy on SimpleQA, ECEs approaching 0.8, and frequent deferrals; scaling them to 70B or adopting to other open-source alternatives (Gemma2-9b-it, Quen-quq-32b) yields only incremental gains.
4.2 Effects of Structured Distractors on Accuracy and Confidence
To quantify effect of adding correct answer alongside three incorrect options, we measure relative accuracy gain Acce = (Drorrect - counts and percentage of Phelped vs. Daamedd and NOT_ _OT_ATTEMPTED (Dna). ) Figure overlays reliability dia- grams (RDs) for all the nine models on SimpleQA dataset. RDs for FaVIQ and TriviaQA are in the Appendix
and ECF. compression AFCF = NMEC - DECE. Table 2 reports
Across all models, structured distractors boost accuracy and (for most cases) reduce ECE. On the challenging SimpleQA benchmark, distractors often halve ECE (up to AECE 0.4) and more than double accuracy for smaller variants, confirming that explicit options help recalibrate confidence when generation alone is unreli- able. However, on TriviaQA the casiest (ltascttCPT-445 and GPT-4-turbo (the largest models) exhibit a slight increase in ECE under D, despite relative accuracy gains of 3 5%. We observe that rises from 0.071 to 0.083, and E (GPT-4-turbo) rises from 0.025 to 0.048. This counterintuitive effect likely stems from confidence inflation: on already-easy examples, the multiple-choice context encourages the model to assign excessive probability mass to the correct answer, amplifying residual misalignment between predicted confidence and true correctness.
Smaller models (< 10B) show the largest AAec on FaVIQ and SimpleQA but also the highest Dnarmed rates on FaVIQ and TriviaQA, suggesting that limited pretraining makes them more susceptible to distractor- induced errors. Notably, Dna drops for all models (to zero on TriviaQA), underlining that no LLM abstains once explicit options are provided for casier questions.
4.3 Influence of Fine- Tine-Tuing Regime and Model Architecture
Our analysis systematically investigates how different fine-tuning strategies and model architectures impact accuracy and calibration performance, specifically in the context of structured distractors.
First, we observe that effectiveness of RLHF on calibration performance varies significantly across different model implementations and sizes. While RLHF models such as GPT-4o-mini (assumed 8B parameters) exhibit superior calibration performance (ECE 0.750 reduced to 0.320 SimpleQA), smaller-scale RLHF models like LLaMA-3-8b-8192 and -LLMM--3.1-8b-Inttae only model. Specifically, in D setting, Qwen-qwq-32B demonstrates better accuracy (51.68% vs. 44.64% for LLaMA-3.1 1 and 44.01% for LLaMA-3 on SimpleQA) and calibration (ECE 0.253 vs. 0.367 for LLaMA-3. and 0.361 for LLaMA3-8b). highlighting that RLHF alone does not guarantee superior calibration. This indicates that other factors, such as the volume and diversity of training data, the quality of fine-tuning data, and overall training strategies, play crucial roles.
underperform relative to Qwen-qwq-32B, an SFT-
Examining within the LLaMA family, LLaMA variant across all benchmarks despite identical parameter counts. This improved performance, especially evident in all three open-domain QA tasks in N setting (e.g. accuracy 75.46% vs. 67.41% in TriviaQA dataset), stems from enhancements in parametric knowledge, refined instruction tuning, and superior cali- bration. LLaMA-3.1-1 2023), enabling better retention of long-tail factual knowledge and improved instruction-following capabili- ties, thereby enhancing robustness and reliability.
-IIstruc notably outperforms the earlier LLaMA3-8B
benefited from training on an extended and more recent dataset (up to December
Distilled models, notably Gemma2-9b-it, exhibit higher 'NOT ATTEMPTED' rates on all the three datasets (e.g. 33.29% on SimpleQA) compared to similarly sized models, indicating challenges in effectively utilizing their compressed knowledge base without external support. Qwen-qwq despite lacking RLHF fine- tuning, consistently produces answers with lower 'NOT ATTAATTEEPTEE rates and demonstrates robustness against iitrractor-inducee errors, as indicated by its lower percentage of harmed instances.
Furthermore, although significantly larger, the MoE-based LLAMA-4-Scut-111 does not outperform GPT-4o-mini in accuracy or calibration, underscoring that training volume and quality significantly im-
8

---

Published in Transactions on Machine Learning Research (12/2025)
100
+09
70
Ilama3-8b- 812
400
75
gpt 4it-4-turbo
700
-700
llama 4-scout 176
llama-3 3--o--3..-B----- 1-8--nntttt
Accurale (%0
600
60
Ilama33 70b-8192
Accuracy Change
ECE Change 0%
300
25
ECE Change
55
221
+49
200
+48++46
100
74
+8
+17.4
6111-10
35
SimpleGA
Falio Dataset
TrivaOA
SimpleQA
FaVio Dataset
TrivarG
Figure 3: Accuracy and calibration shifts with distractors. We show relative accuracy gains (bars) and ECE changes (points) when distractor options are added. While all models improve in accuracy, calibration effects vary large models benefit most, while smaller or models often remain miscalibrated.
pact performance more than sheer parameter count. However, at a substantially larger scale, LLaMA-3-700 successfully surpasses uPT-4o-mini, highlighting the interplay of extensive parameterization, robust training, and
strategies.
In conclusion, our comprehensive analysis reveals that while RLHF can substantially improve model cali- bration, it is not universally effective without careful consideration of other critical training factors. Models leveraging large-scale training datasets, updated fine-tuning approaches, and comprehensive instruction tun- ing achieve optimal accuracy and calibration. Effective deployment in reliablliy--snniiive contexts thus demands & stralegic blend of parameterization, extensive pre-training, robust fine-tuning methodologies, and carefully designed calibration interventions.
4.4 Effect of Model Size within LLM Families
We disentangle parameter count from other factors by comparing the smallest, mid-sized, and largest check- points released by each provider. In the normal setting (N), accuracy increases monotonically with seale for both families; however, calibration improves much faster for the OpenAT series. GPT-4o already altains an ECE of 0.450 on SimpleQA. This suggests that RLHF alignment, used uniformly across GPT-4 models, amplifics the natural sizc-driven gains self-asscssment that emerge from scaling alonc.
Introducing ristrr options (D) radically alters the picture. The largest relative accuracy jumps occur the smallest models as shown in Figure GPT-4o-mint rockets from 8.5% to 47.4% accuracy on SimpleQA (+461%), and LLaMA-3-BB lcaps +819% over the same split. In contrast, their flagship counterparts-Grrt 4o and LLaMA-706 gain more modest +109% and +338%, respectively. Yet these headline boosts do not translate into equally dramatic calibration improvements. After distractors, GPT-4o compresses its ECE by 92% (to 0.037), whereas GPT-4o-mini still lingers above 0.32. A mirror pattern holds for LLaMA: the 70B model cuts ECE by 69%, finishing at 0.24, while the 8B base remains mis-calibrated (ECE 0.36) despite its vast accuracy lift. When we extend this analysis FaVTQ and TriviaQA, the same ranking by model size holds but with attenuated returns. On VTQ, distractors yield solid-but more modcrate- aacuratt-accuuray boosts across all scales. On TriviaQA, where the baseline performance already high, relative gains shrink into the low-teens, indicating only marginal benefit from explicit distractors.
Two key takcaways emerge from our analysis. First, small models acquire factual knowledge more quickly than they learn to assess their own confidence: adding explicit answer choices boosts accuracy but yields poorly calibrated probability estimates. Second, increasing model scale primarily improves a model's ability to quantify its certainty rather than uncover new knowledge. Consoquently, when downstream tasks rely directly on confidence scores-sute as risk-aware planning or answer adjudication- models offer more trustworthy probabilities. In contrast, in settings where latency compute cost. is the main constraint
9

---

Published in Transactions on Machine Learning Research (12/2025)
Date
Number
gpt-40 (mini)
gpt-4o
gemmaz
gemnaa2
(mini)
(9b-it)
(9b-t))
gpf-4
gp 1-4
(tubbo
(turbo)
46.4%
47.11%
41.33%
Bamaç
59.73%
llama
40.12%
56.56%
(8b-8192)
(8b-8192)
20.345
96%
41%61%
4.87%
43.29%
6.88%
3.48%
Spt-4o
5.13%
gpE-4o
4.96%
70.17%
5.4%%
62.14%
3.29%
42.6%
13,54%
41.78%
10. 10.11%%
1lama-3.1
3lama-3.1
(8b-instiant)
47.25%
(8h-instantt)
44.95%
55,36%
51.2%
4sco---soutt I7b-Iue--nstruct)
50.38%
46.3%
(17b-I6e-nnstruct)
1lmoa4-cout
Ilama3-70b 8192
Haama3770 -00n---0
q 3ww.-3
qwqnnqqq-322b
(a) Date Person
Number Place
gpt-4o
gpt-4o
gemma?
(mipi)
gemma2
(minl)
(9h-it)
(9b-it)
gpr-4
gpt-4
55.81%
(tubbo)
(tubbo)
llamad
50.34%
78.0% 78.0%.
laama
41.69%
38. 38.17%
61.12%
(8b-8192))
(8b-8192)
10.5476
25.1%
49.38%
1.17%
28.72%
gotioo
38.61%
8.44
k.2%
43.5 33.56%
gpt-4o
1.17%6
82.523
8,43%
74.47%
49.28%
41.69%
10.776
11.1a-3.11
1aam-3.11
(8b-instiant)
(8b-instiant)
40.98%
52.35%
47,31%
59.94%
50.34%
4-souu (17b (eh--6e-instruct)
46.14%
(1 (benstee))))
Ilaou
Hama3-70b-8199
Hama3-70h--819
gwen-qwq-32b
qwen-qwq-32b
(c)
Person
(d) Place
Figure 4: Performance (correct) of LLMs across different question types in both A (o) and D settings.
and some post-hoc temperature scaling is acceptable, smaller models can still deliver adequate confidence estimates so long as they're 're provided with distractors retrieval-augmentel context.
4.5 Performance Across Question Types
The SimpleQA dataset already categorizes 4326 questions into four non-overlapping types Date (1418), Number (663), Person (1041), and Place (427)-and we evaluate cach model's accuracy and ECE in both the free-generation (W) and distracttor-uggeeteee (D) settings. To better understand calibration weak- nesses, we analyze model performance and confidence alignment across these question types (Figure 4 Person-based queries are most challenging, likely due to name ambiguitics and inherent variability in names, overlapping roles, and contextual dependencies that require deeper reasoning beyond surface-level pattern matching. L.LSs frequently confuse historical figures with similar names, but providing structured answer
10

---

Published in Transactions on Machine Learning Research (12/2025)
Table 3: ECE comparison across question types.
Date
Number
Person
Place
N
D
N
N
D
N
D
GPT-4o-mini
0.77
0.32
0.73
0.35
0.76
0.25
0.73
0.42
GPT-4-turbo
0.62
0.23
0.65
0.26
0.62
0.03
0.52
0.19
GPT-4o
0.41
0.05
0.53
0.13
0.51
0.07
0.35
0.04
LLaMA-3. 1-8b
0.84
0.37
0.82
0.38
0.80
0.31
0.73
0.40
LLaXA-3-8b
0.85
0.35
0.82
0.36
0.81
0.33
0.73
0.45
1LaMA-3-70b
0.80
0.22
0.80
0.28
0.76
0.19
0.65
0.33
LLaMA-4-170
0.68
0.30
0.65
0.29
0.60
0.24
0.53
0.35
Gemma2-9b-1
0,80
0.33
0.78
0.42
0.84
0.33
0.76
0.41
Qwen-quq-32
0.68
0.21
0.66
0.25
0.68
0.26
0.66
0.31
mean
0.72
0.26
0.72
0.30
0.71
0.22
0.63
0.32
choices significantly improves accuracy, suggesting that explicit disambiguation helps mitigate uncertainty in this category. In contrast, place- based queries exhibit relatively strong performance across both settings, indicating that geographic knowledge is well-represented in pretraining. However, calibration improvements vary (Table 3): the person category sees highest relative ECE drop (69%). while place category shows the lowest (49%). This suggests that structured choices help correct overconfidence in ambiguous queries but offer limited calibration gains when models already retrieve knowledge with high confidence. These results demonstrate that miscalibration depends on both task framing and knowledge representation, not just model scale or architecture. While structured reasoning improves confidence alignment in person-based queries, factual retrieval tasks like place-based questions may require alternative calibration stralegies to prevent persistent overconfidence.
5 Conclusion
Our investigation provides a rigorous empirical foundation for understanding and addressing calibration is- sues in LLMs. We reveal widespread overconfidence across various model families and sizes, significantly improved by employing structured distractors caaraccorssparttuulaallyy effective for smaller models. However, our find- ings also highlight counterintuitive outcomes, including degraded calibration in large models on simpler queries. Moreover, systematic miscalibration across specific query categories underscores the complexity of the calibration challenge beyond simple accuracy. Consequently, achieving trustworthy AI requires mul- tifaceted calibration strategy, integrating robust RLHF, optimized prompt design, and post-hoc calibration adjustments. The evaluation framework and guidelines proposed herein serve critical tools for future research, driving forward the development. of LLMs that are not only accurate but reliably calibrated for safe real-world application.
6 Limitations
Generator/Judge Dependence. Our distractor-augmeentee setting fixes a single generator and single judge (GPT-4o-mini) for consistency across models. While this reduces rubric drift and style confounds, it also risks bias from an "Al-generates/ AAljjudgss loop. We mitigate subjectivity via type- and format- matched distractor prompts, overlap and plausibility checks, and human spot-checks by three reviewers with disagreement resolution. Nevertheless, results should be interpreted as conditional on this (generator, judge) pair. We release prompts to support replication with alternative generators and judges. Exploring cross- model distractor sources and independent judges (including human-only adjudication) is important future work; our present study prioritizes comparability under fixed setup and does not claim generator /judge invariance.
11

---

Published in Transactions on Machine Learning Research (12/2025)
References
Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebron, and Sumit Sanghai. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. In The 2023 Conference on Empirical Methods in Natural Language Processing.
Yangyi Chen, Lifan Yuan, Ganqu Cui, Zhiyuan Liu, and Heng Ji. A close look inlo the calibration of pre- trained language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1343-1367, 2023.
Prateek Chhikara, Dhiraj Chaurasia, Yifan Jiang, Omkar Masur, and Filip llievski. Fire: Food image to recipe generation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Compuler Vision, pp. 8184 8194, 2024.
Prateek Chhikara, Dev Khant, Saket Aryan, Taranjeet Singh, and Deshraj Yadav. Mem0: Building production-ready ai agents with scalable long-tern memory. arXiv preprint arXiv:2504.19411 2025.
Yann N Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolu- tional networks. In International conference on machine learning, pp. 933 941. PMLR, 2017.
Shehzaad Dhuliawala, Vilém Zouhar, Mennatallah El-Assady, and Mrinmaya Sachan. A diachronic perspec- tive user trust in ai under uncertainty. In Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, pp. 5567-5580, 2023.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The Ilama herd of models. arXiv preprint orXiv:2407.21783, 2024.
Jiahui Geng, Fengyu Cai, Yuxia Wang, Heinz Koeppl, Preslav Nakov, and Iryna Gurevych. A survey of con- fidence estimation and calibration in large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume Long Papers), pp. 6577 6595, 2024.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International conference machine learning, pp.
PMLR, 2017.
Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Tei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, et al. Qwen2. 5-coder technical report. ar preprint arXiv:2409. 10x.:2409.12186, 2024.
Zhengbao Jiang, Jun Araki, Haibo Ding, and Graham Neubig. How know? on the calibration of language models for question answering, Transactions of the Association for Compulational Linguistics, 9:962-977 2021.
know when language models
Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: large scale distantly su- pervisod challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1601 1611, 2017.
Pranjal Kumar. Large language models (Ims)): survey, technical frameworks, and future challenges. Artificial Intelligence Review, 266)) 2024.
Jixuan Leng, Chengsong Huang, Banghua Zhu, and Jiaxin Huang. Taming overconfidence Imss: Reward calibration in rlhf. arxiv preprint arXiv:2410.0979, 2024.
Chengzu Lı, Han Zhou, Goran Glavaš, Anna Korhonen, and Tvan Vulié. Can large language models achieve calibration with in-context learning? In ICLR 2024 Workshop on Reliable and Responsible Foundation Models, 2024.
Stephanic Lin, Jacob Hilton, and Owain Evans. Teaching models to express their uncertainty in words. Transactions on Machine Learning Research.
12

---

Published in Transactions on Machine Learning Research (12/2025)
Hongfu Liu, Hengguan Huang, Hao Wang, Xiangming Gu, and Ye Wang. On calibration of Ilm- guard models for reliable content moderation. In The Thirteenth International Conference on Learning Representations.
Charles G Lord, Mark R Lepper, and Elizabeth Preston. Considering the opposite: corrective strategy for social judgment. Journal of personality and social psychology, 47(6))1231, 1984.
Sabrina J Mielke, Arthur Szlam, Emily Dinan, and Y-Lan Boureau. Reducing conversational agents' over- confidence through linguistic calibration. Transactions of the Association for Computational Linguistics, 10:857-872, 2022.
Thomas Mussweiler, Fritz Strack, and Tim Pfeiffer. Overcoming the inevitable anchoring effect: Considering the opposite compensates for selective accessibility. Personality and Social Psychology Bulletin, 26(9): 1142 1150, 2000.
Mahdi pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. Obtaining well calibrated probabilities using bayesian binning. In Proceedings of the AAAI conference artificial intelligence, volume 29, 2015.
Charles Packer, Sarah Wooders, Kevin Lin, Vivian Fang, Shishir Patil, Ion Stoica, and Joseph Gonzalez. Memgpt: Towards Ilms as operating systems. ar preprint or 0rXic:231000850 2023.
Jungsoo Park, Sewon Min, Jaewoo Kang, Luke Zettlemoyer, and Hannaneh Hajishirzi. Faviq: Fact verifica- tion from information-secking questions. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics (ACL), 2022.
Sco Yeon Park and Cornelia Caragea. On the calibration of pre-trained language models using mixup guided by area under the margin and saliency. In Procerdings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 5364-5374, 2022.
Rafacl Rafailov, Archit Sharma, Eric Mitchell, Christopher Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Newral Information Processing Systems, 36:53728-5321 53741, 2023.
Marita Skjuve, Petter Bae Brandtzacg, and Asbjørn Følstad. Why do people use chatgpt? exploring user motivations for generative conversational ai. First Monday, 2024.
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024.
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models practical size. ar preprint ar arXiv:2408.00 001:2008.01168 18, 2024.
Katherine Tian, Eric Mitchell, Allan Zhou, Archit. Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher D Manning. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models finc-tuned with human foedback. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 5433 5442, 2023.
Jason Wei, Nguyen Karina, Hyung Won Chung, Yunxin Joy Jiao, Spencer Papay, Amelia Glaese, John Schulman, and William Fedus. Measuring short-form factuality in large language models. XrXiv preprinl arXiv:2/11.04366, 2024.
Miao Xiong, Zhiyuan Hu, Xinyang Lu, YIFEI LI, Jie Fu, Junxian He, and Bryan Hooi. Can 1ims express their uncertainty? an empirical evaluation of confidence clicitation in 1mss. In The Twelfth International Conference on Learning Representations.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. ar preprint or Xixi: 12.111211111 15115, 2024.
13

---

Published in Transactions on Machine Learning Research (12/2025)
Jiarui Zhang. Guided profile generation improves personalization with large language models. In Findings of the Association for Computational Linguisties: EMNLP 2024, pp. 4005-4016, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/v1 10 findings-emnlp.223 URL http:////acaaantoooorrrrr///////
Jiarui Zhang, Mahyar Khayatkhoei, Prateek Chhikara, and Filip Ilievski. Milms know where to look: Training-Tree perception of small visual details with multimodal llms. In The Thirleenth International Conference on Learning Representations.
Jiarui Zhang, Mahyar Khayatkhoei, Prateek Chhikara, and Filip Ilievski. Visual cropping improves shot question answering of multimodal large language models. In RO-FoMo: Robustness of Few-shot and Zero-shot Learning in Large Foundation Models, 2023.
Kaitlyn Zhou, Jena Hwang, Xiang Ren, and Maarten Sap. Relying on the unreliable: The impact of language models' reluctance to express uncertainty. In Proceedings of the 62nd Annual Meeling of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3623 3643, 2024a.
Lexin Zhou, Wout Schellaert, Fernando Martinez- Mlrtinez-Plumed, Yael Moros Daval, Cèsar Ferri, and José Hernández-Oralle Larger and more instructable language models become less reliable. Nature, 634(8032): 61 68, 2024b.
Chiwei Zhu, Benfeng Xu, Quan Wang, Yongdong Zhang, and Zhendong Mao. On the calibration of large language models and alignment. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 9778 9795, Singapore, December 2023. Association for Computational Linguistics. doi: 10. 18653/v1/2022 finniiin//n. emnlp.654. URL aetta://aclaatiooogyooor//////// finiiiffinninnneeee Findings-emmlp.655 emnlp.654/
14

---

Published in Transactions on Machine Learning Research (12/2025)
A
Custom Prompts
We generate the answers for N setting using the following prompt. The prompt outputs the answer and confidence for the answer in a json format.
LLM RESPONSE SEEPPONS__PRMPIII = "" You are an intelligent assistant who is given a question. Your role is provide accurate, helpful, and well-reasoned responses based on your knowledge and capabilities.
Along with the question, you need provide confidence score for your answer. The confidence score should be number between 0 and 100, where:
-0-25 indicates low confidence
26-75 indicates moderate confidence
76-100 indicates high confidence
Guidelines for providing answers:
1. Be direct and concise in your answer while ensuring completeness. Avoid unncoessary words or tangents.
2. If you are uncertain, provide lower confidence score.
3. Base your confidence score
The reliability and recency of available information
Your knowledge of the specific domain
Here are some examples:
Example
Question: What is the capital of France?
Answer: Paris
Confidence score: 91 (High confidence as this is well-established fact)
Example 2:
Question: Which country has the best healthcare system?
Answer: It depends on the criteria used. Some rankings favor Switzerland, while others favor Sweden or Singapore.
Confidence score: 25 (There is definitive answer, and the confidence is low due to the lack of clear
consensus.)
Example 3:
Question: Which state is between Washington and California?
Answer: Oregon
Confidence score: 87 (Maximum confidence as this a clear geographic fact)
Example
Question: What was Albert Einstein's favorite food?
Answer: There is no definitive record of his favorite food, but he reportedly liked pasta.
Confidence score: 25 (There are anecdotal mentions, but no verified records.)
Example 5:
Question: Is Irvine a city California?
Answer: Yes
Confidence score: 81 (High confidence as this is verifiable fact)
Example 6:
Question: What is the most popular programming language for AI development?
Answer: Python
Confidence score: 66 (Moderate-high confidence based on current trends, but this can change over time)
Here now example. Simply reply with your answer and confidence score.
Question: (question)
15

---

Published in Transactions on Machine Learning Research (12/2025)
Provide your response in the following JSON format: { (aanswer:: 'Your answer here", Confidence_scores number between 0-100) "
We generate the answers for D setting using the following prompt. The prompt outputs the answer and confidence for the answer in a json format.
LLM RESPONSE PROMPT DISTRACTORS
You are an intelligent assistant who given question and list of options. Your role provide accurate, helpful, and well-reasoned answer based your knowledge and capabilities and the options provided.
Along with the answer, you need to provide confidence score for your answer. The confidence score should be a number between 0 and 100, where: 0-25 indicates low confidence
26-75 indicates moderate confidence
76-100 indicates high confidence
Guidelines for providing answers:
1. Return the answer from the list of options provided only, It is guaranteed that the answer will be one of the options provided.
2. If you are uncertain, provide a lower confidence score.
3. Base your confidence score on:
The reliability and recency of available information Your knowledge of the specific domain
Here examples:
Example 1:
Question: What is the capital of France?
Options: Paris London Rome Madrid
Answer: Paris
Confidence score: 91 (High confidence as this is well-established fact)
Example 2:
Question: Which country has the best healthcare system?
Options: Switzerland Sweden - Singapore United States
Answer: It depends on the criteria used. Some rankings favor Switzerland, while others favor Sweden or Singapore.
Confidence score: 25 (There is no definitive answer, and the confidence low due to the lack of a clear consensus.)
Example 3:
Question: Which state is between Washington and California?
Options: Oregon Washington California Idaho
Answer: Oregon
Confidence score: 87 (Maximum confidence as this is a clear geographic fact)
Example
Question: What Albert Einstein's favorite food?
Options: Pizza Pasta Sushi Tacos
Answer: There is no definitive record of his favorite food, but he reportedly liked pasta.
Confidence score: 25 (There are anecdotal mentions, but no verified records.)
Example 5:
Question: Is Irvine a city in California?
Options: Yes No
Answer: Yes
Confidence score: 81 (High confidence as this is verifiable fact)
16

---

Published in Transactions on Machine Learning Research (12/2025)
Example 6:
Question: What is the most popular programming language for AT development?
Options: Python- - Java C++ JavaScript
Answer: Python
Confidence score: 66 (Moderate-high confidence based on current Lrends, but this can change over time)
Here new example. Simply reply with your answer and confidence score.
Question: {question) Options: {options)
Provide your response in the following JSON format: { ('answer": 'Your answer here", 'confidence_scone _ucocce____rrrr number between 0-100 } "
To generate the 3 distractors for each question-answer pair, we use the following prompt with few shot examples.
DISTRACTORS GENERATION PROMPT "You an expert synthetic data generator. Your
task is to generate three plausible but incorrect answers to given question.
Guidelines for generating wrong answers:
1. Fach answer should be factually incorrect but plausible within the context
2. Match the answer type (e.g. if asking for a date, provide wrong dates)
3. The wrong answers should be clearly distinct from the correct answer and from each other
4. Maintain similar level of specificity as the original answer
5. The answers should be realistic and not obviously wrong
Example 1:
Question: What is the capital of France?
Answer: Paris
Wrong Answers: Lyon Marseille Bordeaux
Reason: All are major French cities, but incorrect as capital
Example 2:
Question: Who was the first president of the United States?
Answer: George Washington
Wrong Answers: John Adams Thomas Jefferson Benjamin Franklin
Reason: All are founding fathers but not the first president
Example
Question: In what year did World War 1l end?
Answer: 1945
Wrong Answers: 1943 1944 1946
Reason: All are plausible years during or near WWII but not when it ended
Example 4:
Question: Who wrote Romeo and Juliet?
Answer: William Shakespeare
Wrong Answers: Christopher Marlowe Ben Jonson John Webster
Reason: All prominent Flizabethan playwrights
Example 5:
Question: What is the largest planet in our solar system?
Answer: Jupiter
Wrong Answers: Saturn Neptune Uramus
Reason: All are gas giant planets, but smaller than Jupiter
Please generate three wrong answers that follow these guidelines for the given question.
The answers should be:
Factually incorrect but plausible
Match the same answer type (e.g, date, person, number)
17

---

Published in Transactions on Machine Learning Research (12/2025)
Clearly distinct from the correct answer and each other
Similar in specificity /detail level
Realistic and not obviously wrong
Return only three wrong answers as a list in JSON format with the following requirements:
Each wrong answer should be a string
The output should be single JSON object with key "wrong answers'
The value should be array of exactly 3 wrong answers
No explanations additional text should be included
The answers should maintain consistent formatting with the correct answer
{ Example format: (wwrong answers': ['optl", 'opt2', 'opt3"]
Question: {question)
Correct Answer: (answer)
Generate three wrong answers:
Table Performance metrics of LLMs on SimpleQA dataset the Normal (N) and Distractor (D) settings, including accuracy (wesms), non-attempt (w.), ECE, and the number of helped (Dhelped) and harmed (D harmedd) instances. Here the LLM judge model is same as the prediction model.
LLMs
Neorreat
Nnone
NECE
Doorreet
Doone
DECE
Dhelped
D Dharmed
GPT-4o-mini
8.46%
6.80%
0.750
47.43%
0.02%
0.320
1644 (93.78%)
109
GPT-4-turbo
20.99%
7.14%
0.616
65.33%
0.02%
0.165
1821 (95.44%)
87
GPT-4o ?
36.75%
8.16%
0.437
73.48%
0%
0.037
1507 (91.22%)
145
LLMMA3.1 1.AMA3.1-Bbbinstam 00
8.24%
19.58%
0.780
44.94%
0.21%
0.367
1294 (91.45%)
LLaMA 3-8B-8192
9.27%
24.99%
0.790
45.56%
2.45%
1251 (90.46%)
Gemma2-9B-it G
9.52%
34.49%
0.771
46.58%
1.87%
0.359
1060 (88.70%)
135
B Same LLM judge as Prediction LLM
We employ the same LLM as both the judge and the base model responsible for predicting answers to the questions. The performance metrics are presented in Table Upon manually inspecting instances from smaller LLMs, we observe that the LLM judge occasionally misclassifies responses and refrains from assigning NOT_ ATTEMPTED to certain data points. This can be seen by comparing Nnone in Table and 4 for the SimpleQA datasct. To address this issue and ensure consistency, we use GPT-4o-mini as the LLM judge across all models. We also create reliability diagrams of pipeline where LLM-judge was different and the graphs are shown in Figure 5
To validate the reliability GPT-4o-mini as an LLM judge, we conducted a small-scale human evalua- tion. We sampled 100 responses and had three human annotators independently classify them as CORRECT, INCORRECT, NOT ATTEMPTED. The inter-annotator agroement, measured using Cohen's kappa, was 0.82 for GPT-4o-mini, indicating substantial agreement. This comparison allowed us to measure the extent of biases introduced by automated evaluation and confirm that LLM judges generally aligned with human judgments, though minor inconsistencies were observed in ambiguous cases.
C Reliability Diagrams of Selected Models on FaVIQ and TriviaQA dataset
18

---

Published in Transactions on Machine Learning Research (12/2025)
g01-4o
1.0
0.8
0.8
0.8
0.6
0.6
0.2
0.2
0.2
0.0
0.0
0.0
1.. ---biinstan
Harma 9b-8 192
gommma2 99bit
0.8
0.6
0.6
0.6
1124
0.2
0.2
STA
0.0
0.0
Figure 5: Reliability diagrams (RDs) on SimpleQA dataset showing calibration performance in N and D settings. The numbers on top of bars represent the number correctly predicted instances (y-axis: actual accuracy, x-axis: predicted confidence). Here the LLM judge model is same as the prediction model.
19

---

Published in Transactions on Machine Learning Research (12/2025)
2/6-4-imbo
1.0
1.0
0.8
0.8
0.8
04
0.4
0.4
0.2
0.2
0.2
20
Hamm--3 1.000--.-RR-i-eee -Rb-insttant
[eems4-88b88100 192
pemmu2-9h-ll
0.6
0.6
0.6
0.4
0.0
0.0
0.0
80
300
20
80
100
qwen-qqwq-322
3-70h-
(7b-16e-instruct
1.0
1.0
0.8
0.8
0.8
0.4
0.2
0.2
0.8
29
Figure Reliability diagrams (RDs) showing calibration performance in A (o) and D (+) settings on the TriviaQA dataset. (y-axis: actual accuracy, x-axis: predicted confidence).
20

---

Published in Transactions on Machine Learning Research (12/2025)
1.0
1.0
0.8
0.8
0.8
0.4
0.4
0.4
0.2
0.2
0.2
Hamm--3 1.--a--.--------- -Rb-isstant
Hemm43-8b-4190
pommu2-90bil
0.6
0.6
0.6
0.4
0.0
0.0
0.0
300
20
60
100
20
qwen-qww--12hh
3-70h-5
17b- 10-166--estructt
1.0
1.0
0.5
0.8
0.8
0.4
0.2
0.2
08
20
Figure Reliability diagrams (RDs) showing calibration performance in A (o) and D (+) settings on the Fa VaQ dataset. (y-axis: actual accuracy, x-axis: predicted confidence)
21


Reviews:
[{"review_id": "4nkVv4PVwq", "reviewer": "Reviewer_raGB", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper conducts a large-scale empirical study on the calibration of large language models (LLMs). The authors analyze different model families, sizes, and architectures across three factual question answering (QA) datasets of varying difficulty, and ultimately offer recommendations for deploying more reliable LLMs.", "claims_and_evidence_yn": "No", "claims_explanation": "1. The paper does not compare against any other calibration methods. Without such a comparison, we can’t tell whether the proposed distractor-based prompt engineering is truly a superior calibration technique, or merely a more complex and costly way to achieve what a simpler method could do.\n\n2. The study obtain confidence by explicitly asking the model to provide a “confidence_score” between 0 and 100. Which is better: this verbalized confidence approach, or a method based on internal token-level log probabilities of the model’s outputs? If model calibration is poor, is it because the model is overconfident about its knowledge, or because it struggles with the secondary task of mapping its internal uncertainty onto the numeric value requested in the prompt?", "audience_yn": "Yes", "audience_explanation": "Yes, LLM calibration is a relatively important issue and will attract some people’s interest.", "requested_changes": "Please address the concerns raised in my explanation regarding the criterion“claims made in the submission supported by accurate, convincing and clear evidence”.", "broader_impact_concerns": "No.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "EPEcqkaXMU", "reviewer": "Reviewer_hv9U", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper conducts a study to figure out when and why LLMs act overconfident. The authors ask the LLM to predict answer and confidence score and measure the confidence score calibration error using ECE metrics. Their main contributions are\n1. Testing LLMs with Structured Distractors: The key idea is to compare how an LLM answers a question in two ways: (1) just asking it directly (free-generation) versus (2) giving it a multiple-choice question with one correct answer and three plausible-but-wrong answers (called \"distractors\"). Then study the affect of distractors on accuracy and confidence score calibration.\n2. Detailed Breakdown of Failures and Model Capability Analysis: They ran this experiment on nine different popular LLMs (including various GPT, LLAMA, and other models) across three different question-answering datasets to see if the results were consistent. Instead of just a single \"pass/fail\" score, they dug into the details. They analyzed what makes models more or less overconfident, looking at factors like:  \n(a) Model Size: Do bigger models (like GPT-4o) behave differently than smaller ones (like LLAMA-8B)?  \n(b) Training Method: Does the \"RLHF\" tuning that makes models good at chatting also make them more or less overconfident?  \n(c) Question Type: Are models more overconfident about \"people\" than they are about \"dates\" or \"places\"?", "claims_and_evidence_yn": "Yes", "claims_explanation": "[Borderline Yes]  \n* This paper claims that LLMs are generally overconfident and adding structured distractors as options mitigate this problem.  \n* The authors have studied the LLMs overconfidence problem by asking LLM directly about its confidence in the generated answer. Standard datasets (SimpleQA, FaVIQ, TriviaQA) were used to measure the quality of answers and confidence score estimations. The prompts for answer generation, structured distractor generation and LLM Juding are provided. The results from judging and structured distractor generation prompting are said to be manually verified by 3 reviews which I believe to be true.  \n* The authors have not only provided the metrics but also provided rationale and plausible reasoning for the results and unexpected results.\n\nThe reason why I have borderline yes vote, is due to the following concerns\n* Confidence score generation via LLM prompting isn't the only way of generating confidence score through LLMs. Model's internal logits can also be utilized to generate confidence score. Similarly majority voting approach can be used as well. This paper doesn't provide any details of other confidence score generation algorithms and affect of distractors on the confidence score calibration.\n* The LLM was asked to provide confidence score in 3 wide bins. In my opinion, criteria for more granular confidence score estimation should have been given.\n* The entire distractor-augmented setting relies on the quality of the three incorrect answers. These distractors were generated by GPT-4o-mini. The authors state they were carefully designed and manually inspected, but the \"plausibility\" of these distractors is a critical, subjective variable. If the distractors are too simplistic or \"obviously wrong,\" the task becomes trivial and the calibration findings are skewed. The experiment's validity is thus dependent on the quality of an AI-generated artifact.\n* There is a risk of systemic bias, where the judge (GPT-4o-mini) may favor the answer style or factual knowledge of its own model family (the GPT series), potentially inflating their performance scores relative to other models like LLAMA or Qwen.", "audience_yn": "Yes", "audience_explanation": "[Borderline Yes]  \n* The central theme of the paper is the \"misalignment between predicted confidence and true correctness\" (miscalibration). This is one of the most significant barriers to deploying LLMs in critical decision-making applications. The audience, which includes researchers focused on AI safety, robustness, and reliability, is deeply invested in understanding and solving this exact problem. Therefore, it would be interesting topic to them.\n* The paper provides more than just confirmatory results. The discovery that large, advanced models (like GPT-4-turbo and GPT-4o) can \"paradoxically suffer increased miscalibration on easier queries\"  is a novel, surprising, and important finding. This kind of non-obvious finding and analysis would be interesting to the audience.\n* The analytical rigor in this paper is very good. The authors haven't only provided top line scores but also analyzed the score at different angles - how scale, tuning regime, and architecture independently influence model calibration.\n\nThe reason why I have borderline yes vote, is due to the following concerns\n* The study is performed exclusively on three factual Question-Answering datasets. The findings may not apply to other common LLM tasks, such as creative writing, summarization, mathematical reasoning, or code generation.\n* The distractors format is fixed: one correct answer plus three distractors. This doesn't fully capture real-world scenarios (like Retrieval-Augmented Generation) where a model might receive five retrieved chunks, none of which are fully correct, or all of which are partially correct. The findings may not hold in a \"noisy\" environment with a different number of distractors or no guaranteed correct answer.", "requested_changes": "**Issue**: The paper measures confidence by prompting the LLM for a 0-100 score. This is a valid method, but it is not the only one. Other methods, such as using the model's internal logits or employing a majority voting approach, are not explored or justified.  \n**Proposed Adjustment**: The authors must add a section (in Methodology or Limitations) that explicitly justifies their choice of using elicited confidence. This discussion should acknowledge the existence of alternative methods (like logit-based confidence) and critically evaluate how this choice might impact the findings.\n\n**Issue**: The experimental design relies heavily on a single model (GPT-4o-mini) for two key functions:  \nDistractor Generation: The \"plausibility\" of the AI-generated distractors is a critical, subjective variable that underpins the entire distractor-augmented experiment.  \nAnswer Adjudication: The same model is used as the judge to determine correctness, which risks systemic bias (e.g., favoring the style or factual knowledge of its own model family).  \n**Proposed Adjustment**: While the authors do perform some validation, they should explicitly discuss the risk of bias from this \"AI-judging-AI-generated-content\" loop in their Limitations section. Acknowledging that the results are contingent on this specific generator/judge model's behavior is necessary for full transparency.  \nOne option to mitigate this problem would be to generate distractors and evaluate the scores using other LLMs as well. Then provide provide a wholistic evaluation analysis.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ROnH6tQFle", "reviewer": "Reviewer_Aqgr", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a comprehensive empirical study on the calibration of LLMs. The authors investigate the pervasive issue of overconfidence by evaluating nine different LLMs across three factual question-answering (QA) datasets. The core contribution is the systematic comparison between a standard free-generation setting and a novel \"distractor-augmented\" setting, where models are prompted with the correct answer alongside several plausible, synthetically generated incorrect options.\n\nStrengths:\n- Comprehensive Scope: The evaluation is extensive, covering a wide range of modern LLMs that vary in size, architecture (Dense vs. MoE), and fine-tuning methods (SFT vs. RLHF).\n\n- Novel Methodology: The introduction of the distractor-augmented setting is a key strength. This approach effectively simulates real-world scenarios like multiple-choice questions or retrieval-augmented generation.\n\n- Detailed and Insightful Analysis: The paper goes beyond simple accuracy metrics to provide a fine-grained analysis of calibration, including Expected Calibration Error (ECE), and dissects performance by question type (Person, Place, Date, etc.).\n\nWeaknesses:\n- Possibility of Methodological Bias: Both the distractors and the final judgments of correctness were generated by a single LLM (GPT-4o-mini). This introduces a risk of systemic bias, as the distractors may be inherently easier for models from the same family, and the judge may favor certain types of answers.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The claims made in the paper are well-supported by extensive empirical evidence. The authors conduct a large-scale, systematic evaluation across nine different models and three datasets, which provides significant weight to their conclusions. The core claims are all substantiated by the quantitative results presented in the tables and figures.\n\nFor instance, Table 2 clearly shows the dramatic improvements in accuracy and reductions in ECE when moving from the Normal (N) to the Distractor (D) setting. Figure 3 effectively visualizes the trade-offs, illustrating that while smaller models see massive accuracy gains, their calibration improvement is less pronounced compared to larger models. Figure 4 provides further convincing evidence for the claim that certain categories are inherently more challenging and benefit more from the structured-choice format.", "audience_yn": "Yes", "audience_explanation": "The topic of LLM calibration is of critical importance as these models are increasingly deployed in high-stakes applications. This work provides comprehensive and systematic studies on the topic. It moves beyond simply measuring calibration to actively proposing and testing a method (distractor-augmentation) to mitigate miscalibration. The nuanced findings—such as the counterintuitive degradation of calibration in large models on easy tasks and the identification of specific question types prone to miscalibration—are novel and will be of significant interest to researchers or engineers working on LLM safety, interpretability, and robustness.", "requested_changes": "- Address Potential Judge/Distractor Bias (critical): The reliance on a single model (GPT-4o-mini) for both generating distractors and judging correctness is a significant methodological concern. I strongly recommend the authors expand their analysis of this potential bias. The authors can use a more diverse set of models to generate distractors and/or use a holdout LLM family as the judge.\n\n- Explore Different Distractor Types (strengthening): The current distractors are designed to be \"plausible but incorrect.\" It would strengthen the paper to include an experiment or at least a detailed discussion on how different types of distractors (e.g. common misconceptions or adversarial examples) might affect calibration.", "broader_impact_concerns": "The work does not raise significant ethical concerns. Its primary goal is to improve the reliability and trustworthiness of LLMs, which is a positive contribution to AI safety.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)