Restoring Calibration for Aligned Large Language Models: A Calibration-Aware Fine-Tuning Approach
Authors: Jiancong Xiao
Bojian Hou
Zhanliang Wang
Ruochen Jin
Qi Long
Weijie J Su
Li Shen
Venue: ICML 2025 poster (2025)
Keywords: Large Language Models;  Calibration; Fine-Tuning
Primary area: deep_learning->large_language_models
TL;DR: An approach for restoring calibration in LLMs.

Abstract: One of the key technologies for the success of Large Language Models (LLMs) is preference alignment. However, a notable side effect of preference alignment is poor calibration: while the pre-trained models are typically well-calibrated, LLMs tend to become poorly calibrated after alignment with human preferences. In this paper, we investigate why preference alignment affects calibration and how to address this issue. For the first question, we observe that the preference collapse issue in alignment undesirably generalizes to the calibration scenario, causing LLMs to exhibit overconfidence and poor calibration. To address this, we demonstrate the importance of fine-tuning
with domain-specific knowledge to alleviate the overconfidence issue. To further analyze whether this affects the model's performance, we categorize models into two regimes: calibratable and non-calibratable, defined by bounds of Expected Calibration Error (ECE). In the calibratable regime, we propose a calibration-aware fine-tuning approach to achieve proper calibration without compromising LLMs' performance. However, as models are further fine-tuned for better performance, they enter the non-calibratable regime. For this case, we develop an EM-algorithm-based ECE regularization for the fine-tuning loss to maintain low calibration error. Extensive experiments validate the effectiveness of the proposed methods.

Restoring Calibration for Aligned Large Language Models: A Calibration-Aware Fine-Tuning Approach
Jiancong Xiao "
Bojian Hou
Zhanliang Wang"
Ruochen Jin
Qi Long'
Weijie Su'
Li Shen'
Abstract
of natural language processing tasks (Bubeck al., 2023; Chowdhery et al., 2023; Touvron et al., 2023; Team et al., 2023). These models, built upon the Transformer architec- ture (Vaswani et al., 2017), have demonstrated remarkable abilities to process and generate human-like text, making them increasingly integral to various applications. A crucial development in making LLMs more reliable and aligned with human values is preference alignment techniques, par- ticularly Reinforcement Learning from Human Feedback (RLHF) (Ouyang et al., 2022) and Direct Preference Opti- mization (DPO) (Rafailov et al., 2023). However, an im- portant side effect of preference alignment is its impact on model calibration the relationship between model's predicted probabilities and its actual accuracy. While pre- trained LLMs typically demonstrate good calibration prop- erties, preference-aligner models become poorly calibrated, as initially observed in GPT-4 with RLHF (OpenAI, 2023), Our investigation reveals that this is a universal issue across different models aligned with various alignment methods. An example shown in Figure (left), where the calibra- tion performance of model aligned by DPO illustrates poor calibration performance.
One of the key technologies for the success of Large Language Models (LLMs) is preference alignment. However, a notable side effect of pref- erence alignment is poor calibration: while the pre-trained models typically well-calibrated, LLMs tend to become poorly calibrated after alignment with human preferences. In this pa- per, we investigate why preference alignment af- fects calibration and how to address this issue. For the first question, we observe that the pref- erence collapse issue alignment undesirably generalizes to the calibration scenario, causing LLMs to exhibit overconfidence and poor cal- ibration. To address this, we demonstrate the importance of fine-tuning with domain-speciffi knowledge to alleviate the overcontidence is- sue. To further analyze whether this affects the model's performance, we categorize models into two regimes: calibratable and non-calibratable. defined by bounds of Expected Calibration Error (ECE). In the calibratable regime, we propose calibration-aware fine-tuning approach to achieve proper calibration without compromising LLMs' performance. However, as models are further fine- tuned for better performance, they enter the non- calibratable regime. For this case, we develop an ao--algorithm-basee ECE regularization for the fine-tuning loss to maintain low calibration error. Extensive experiments validate the effectiveness of the proposed methods.
Understanding and addressing this calibration issue is cru- cial. First, well-calibrated prediction is essential for reliable decision- maaiing in real-world applications, particularly in high-stakes domains such as legal or healthcare analysis (Savage al., 2025). Second, overconfident models may mislead users about their capabilities and limitations, poten- tially leading to inappropriate reliance model outputs.
In this paper, we conduct a systematic investigation into two fundamental questions: (1) Why does preference alignment affect calibration? and (2) How can we effectively restore calibration while maintaining the benefits of alignment? Our analysis reveals that the preference collapse phenomenon (Xiao et al., 2024a; Chakraborty al., 2024), known issue in alignment methods where models excessively favor cer- tain responses, undesirably generalizes to multiple-choice questions, a crucial scenario where calibration performance can be conveniently evaluated. This generalization leads to overconfidence and poor calibration, as models tend to collapse their prediction to one option while ignoring oth- ers, regardless of the correctness of their chosen option.
1. Introduction
Large Language Models (LLMs) (OpenAI, 2023; Anthropic, 2024) have emerged as powerful tools for wide range
Equua contribution UUniversity of Pennsylvania, PA, USA. Correspondence to: Qi Long <qlong@ upenn.edus. Weijie Su <suw @ www.:auuuunnnnnnnneeemm Li Shen <lishen @ @upen..du>.....
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Restoring Calibration for Aligned Large Language Models: A Calibration-Aware Fine- Sine-Tuning Approach
Classwise Calibration Curve
Casswise libration Curve
Underconfidence
Underconfidence
Over confidenne
Overconfideene
0.8
0.8
0.16
Prob. of Cormett
0.6-
0.6
0.12
0.4-
Prob.
0.4
0.00
0.06
0.04
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.02
SF1
Prob.
Prob. A.
Stages
Figure/ 1. Calibration performance comparison between DPO and approach on Llama3.1-8B-Tul DPO-aligned version of Llama- 3.1 (Touvron et al., 2023)). Left: Model calibration plots after DPO alignment, showing significant overconfidence. Middle: Calibration plots after applying our fine-tuning approach, demonstrating improved calihration. Right: The evolution of confidence ECE and classwise ECE across different stages (pre-trained, SFT, DPO, and our method) shows how our approach effectively restores calibration errors.
Building on these insights, we demonstrate that fine-tuning with domain- domain-specific knowledge can extensively alleviate overconfidence on incorrect answers and marginally am- plify overconfidence on correct answers, thus improving calibration performance.
2. Related Work
Calibration of Traditional Deep Learning Models. Prior work has explored various approaches to model calibra- tion. Some researchers have investigated parameterized temperature scaling methods (Guo et al., 2017; Joy et al., 2023; Yu et al., 2022), primarily focusing on vision models. Beyond temperature scaling, researchers have proposed al- ternative approaches including Dirichlet calibration (Kull et al., 2019), training with labelssmoothing (Szegedy et al., 2016), mix-up augmentations (Zhang et al., 2017), and fo- cal loss-based methods (Mukhoti et al., 2020), though these approaches are challenging to implement with LLMs due to their computational demands. Notably, Kumar et al. (2019) conducted rigorous verification of uncertainty calibration, revealing that temperature scaling may be less effective than initially reported.
For further analysis, we develop theoretical framework that characterizes the conditions under which calibration can be restored. We introduce the concepts of calibratable and non-calibratable regimes, defined by upper and lower bounds of Expected Calibration Error (ECH) and a criti- cal accuracy threshold. This framework provides insights into when and how models can achieve good calibration while maintaining performance. We consider fine-tuning the model after RLHF or DPO, rather than modifying these methods directly, as in many practical cases only the fi- nal DPO/RLHF model is available, while the pretrained or SFT model not. We find that preference-aligned mod- els typically lie in the calibratable regime, and propose a calibration-aware fine-tuning (CFT) approach that restores calibration without compromising performance. However, when overly fine-tune these models to achieve better performance, they shift into the non-calibratable regime. In this case, we identify fundamental trade-off between ECE and performance, and develop an IM-aalgrithhm-bsee ECE regularization for the CFT loss to effectively navigate this trf
Calibration for LLMs. Several studies have specifically examined L.M calibration (Jiang et al., 2021; Xiao et al., 2022; Chen et al., 2022), identifying miscalibration issues and evaluating various interventions. Some researchers have explored auxiliary models (Zhang et al., 2021; Ka- davath et al., 2022: Shen et al., 2024) improve calibration, while others have investigated prompt-based approaches for RLHF-tund LLMs (Xiong et al., 2023). Other researchers teach the LLMs to express their uncertainty and calibra- tion (Lin et al., 2022; Tian et al., 2023; Xu et al., 2024b). A distinct line of research focuses on debiasing in-context learning (Abbas et al., 2024; Han et al., 2023; Jiang et al., 2023b), though this differs from statistical calibration. Re- searchers have studied improving the calibration of LLMs during the SFT phase (Han al., 2024) well during the PPO/DPO phase (Tao et al., 2024; Leng et al., 2024). Notably, previous studies consistently identify temperature scaling as the most effective calibration method. Given its strong performance, we use temperature scaling as our primary baseline for comparison.
Through extensive experiments, we demonstrate that our proposed methods significantly improve calibration perfor- mance while preserving even enhancing model's lan- guage and knowledge capabilities. Figure 1 (middle) illus- trates the calibration plots of our approach and Figure (right) shows the evolution of ECE across different training stages. Our approach reduces the ECE of preference-alignel models from 14.22%-20 20..-20..0% ferent model architectures and alignment approaches, which validates the effectiveness of the proposed methods.
2.39%-6.51%% 6.51% across dif-
2

---

Restoring Calibration for Aligned Large Language Models: A Calibration-Aware Fine-Tuning Approach
3. Preliminaries
Table I. The format used in the sample the McdMCQA dataset.
The calibration error of an LLM is typically evaluated on a multiple-choice question and its corresponding correct an- swer, which can be formalized as an input-label pair (z, y), where represents the problem instance consisting of question stem and four candidate answers, one of which is correct and y is the label of the true answer. While we consider four options in our analysis for concreteness, the
The system message is for the Llama family models.
System:
Al assistant that answers multiple choice questions. You must only respond with a single letter corresponding to your choice with- out any explanation or additional text.
You
Input:
Which of the following is very difficult to induce antibody? Options: A. Polysaccharide. B. Pro- tein. C. Antigen. D. Effector. Answer with only single letter:
framework naturally extends to any number of alternatives. The label y E (A, B, C, D) denotes the position index of the correct answer among the choices.
Label:
A
Calibration of Classification Models. Consider a prob- abilistic classification model p X' + A that outputs class probabilities for k classes 1, ...., ,. For any given in- stance x E X' it would output some probability vector p = (x)).....f((()))) (x)) belonging to A = {(q)...., Tx) E [0, 11 شー 42 } which is the (A: - 1) dimeenssian probability simplex over & classes.
that LLMs respond with single letter from {A, B, C, D hint-'Answer with only single letter'-is appended to the prompt. For models in the Llama family, we use an additional system prompt to enforce this response format.
An example of this setup is illustrated in Table 1.
The confidence of LLMs in option j is defined as:
T = =} P5(0) = EyE(AB.C.D To j'(z)
In this paper, we mainly consider the classwise calibration (Zadrozny& Elkan, 2002), which requires calibration of all one-vs-rest probability estimators derived from the original multiclass model, and the confidence calibration (Guo al., 2017), which requires that for instances where the model assigns confidence to its most probable class prediction, the empirical accuracy should match this confidence c.
(1)
Preference Optimization. The RLHF framework for learning L.LMS typically consists of the following three steps: (1) supervised fine-tuning (SFT), (2) Reward mod- eling, and (3) RLHF fine-tuning. The DPO method is to directly optimize the policy without explicitly training the reward function. More details about preference alignment are provided in Appendix A.
Definition 3.1 (Classwise Calibration). A probabilistic clas- sifier p: : X + AG is classwise-calirrtee. if for any class j and any predicted probability jj for this class:
PVy = ومار (I) = q3) = 95-
4. A Generative Calibration Framework
Classwise- Eass (cw-ECE) is defined as:
4.1. A Generative Model for Multiple-Choice Problems
k Σ 1=1
Consider multiple-choice dataset defined as S {(11, ((11,9),,.... ... (Zn, vn)), which can be viewed as an ex- amination created by a test designer. The designer's main constraint is to ensure an approximately balanced distribu- tion, with each option (A, B, C, and D) containing about 25% of the correct answers. This test designer can be con- ceptualized as probabilistic generative model that assigns correct answers to positions according their implicit pref- erences while maintaining this distribution. For given dataset, there exist many different probabilistic generative models. Here are two illustrative examples:
cw-ECE = Ep(x) k
Fly = -1b,
Definition 3.2 (Confidence Calibration). A probabilistic
classifier p p: > A₁ is confidence-calibratte,, if for any C [1/k, 1]:
Ply = argmax p(1)) max p(x) = c) =
Confidence-Fod ECE (conf-ECE) is defined as:
Ply arg max ((x) max p(x)) max p(x)). Ep(x) Σ
Example 1: Pure Random Models. The model assigns equal probability to each option regardless of the question: P(y = j| 25% for all j and z. When generating a dataset using this model, we maintain P(y j|S) 25%.
Calibration of LLMs. To answer multiple-choice ques- tions, LLMS typically process the question along with all available answer choices as a single input. The model gener- ates a probability distribution over the possible responses, al- lowing selection of the most likely correct answer. To ensure
Example 2: Deterministic Models. For each question
We the four-option format (A, B, D) for readability; however, the analysis extends to general k-choice setting.
3

---

Restoring Calibration for Aligned Large Language Models: A Calibration-Aware Fine- Fine-Tuning Approach
the model assigns probability of exactly one option and 0 to all others: P(y = =)(²) = 1 for some and P(y- k/zz) = 0 for all k * j. The questions are distributed such that approximately 25% of samples assign probability 1 to each option j. Using this model, we also maintain P(y = j| 25%.
Definition 4.3 (Target Calibration Error (TCE).). Let p' be the target probabilistic generative model defined in Defini- tion 4.2. The TCE is defined as
1 TCE = E Σ k
(1)))) ==1
(3)
The concept of probabilistic generative models provides natural connection to calibrated models. Since these mod- els generate the positions of correct answers according to their probability distributions, the observed accuracy always equals the model's 's confidence (i.e., its predicted probabili- ties). This inherent property ensures that probabilistic gener- ative models are always call-calibrated. We formalize this relationship in the following proposition.
To provide better intuition for TCE, we establish its upper and lower bounds. For Theorem 4.4 and 4.5, we assume that the policy can represent an arbitrary probability dis- tribution function over the universe of responses. The re- gion between these bounds is illustrated in Figure (yellow shaded area).
Theorem 4.4 (Upper Bound of TCE). Let nT" be the target probabilistic generative model. For any accuracy a E [0, 1], there exists a model π with ACC(#) such that
Proposition 4.1. Let p X + As be probabilistic gen- erative model that assigns correct answers of question x to options A, B, C, and D according to distribution p(z). Then achieves zero classwise and confidence ECE.
TCE(T) S |ACC(T*) ACC(TT)
(4)
Theorem 4.5 (Lower Bound of TCE). Let nt" be the target probabilistic generative model. For any accuracy E [0, 1] and any model x with ACC(Tz) = o, there exists a constant C C (0, 11, s.I.
Here, p(z) is unknown. Based on Proposition 4.1, training well-calibrated LLM can be reformulated as training prob- abilistic generative model. However, not all probabilistic generative models are desirable. Consider Example 1, the pure random model: while achieves zero ECE, it only at- a poorly performing LLM.
TCE(T) 2 C|ACC(**) -ACC(T)
(5)
tains 25% accuracy-rendente This illustrates that well-calibration alone is insufficient; we need models that achieve both strong calibration and good accuracy.
Having introduced the concept of TCE, we can now estab- lish the theoretical bounds for ECE,
Theorem 4.6 (Upper bound for ECE). For any model with predicted probabilities p. the cw-ECE is bounded above by its TCE:
Definition 4.2 (Target probabilistic generative model). Let Ty be an L.LM parameterized by 6, and let ACC(Ta) and ECE((.) denote the accuracy and ECE of me on dataset S. respectively. Let 'T" be the optimal solution of
Cw-ECE(P) S TCE(
max (Tg) 9
Regarding the lower bound of ECE, we have trivial bound ECE 2 0. However, the lower bound has different implica- tions depending on whether a model's accuracy falls above or below the critical accuracy threshold. With this under- standing, we can now formally characterize the calibratable and non-calibratable regimes.
(2)
s.t. ECE(T.)= =
We define the target probabilistic generative model p" as the confidence distribution of x* (given by Equation (I)).
Here, π² is not fixed but rather depends on the current state of LLM development, particularly the capabilities en- abled by architectural innovations such as the Transformer. As LLM architectures and training methods advance, the achievable accuracy may also improve.
Calibratable Regime. When model's accuracy is below the critical accuracy threshold, the following bounds hold for ECE: 0 < ECE < TCE < 2(ACC(T*) ACC(+)). We define this region as the calibratable regime, where models can achieve perfect calibration without sacrificing accuracy.
4.2. Calibratable and Non-Calibratable Regime
Non-calibratable Regime. When a model's accuracy ex- ceeds the critical accuracy threshold, the following bounds hold for ECE: 0 < ECE < TCE < We define this region as the non-calibratable regime, where achieving perfect calibration is impossible according to the target probabilistic generative model definition.
In this subsection, we develop a theoretical framework an- alyze when an LLM can achieve perfect calibration by intro- ducing the calibratable and non-calibratable regimes. These regimes are characterized by a critical accuracy threshold by n that separates them, along with distinct upper and lower bounds for ECE in each regime. To establish the upper and lower bound for ECE, we start from introducing the target calibration error.
))-
The distinction between calibratable and non-calibratable regimes and their corresponding bounds are illustrated in

---

Restoring Calibration for Aligned Large Language Models: A Calibration-Aware Fine-Tuning Approach
Algorithm (Regularized) Calibration-Aware FT Require: Number of epochs L, Number of bins Mi;
1.0
1.00
TCE
Calibratable Regime
0.8-
0.5
Non-calizratable Regime
Initialize model To by the alligned LLMs;
Vilue Calbration
Value Calhtmtton
6.6
0.0
for 1 = 0 to I. do
E-Step: // Use max confidence to stratify samples
0.4-
0.4
for = 1 m do
0.2
0.2
for m 1: M do
0.0
0.0 0.50
if max conf (1r2) E ( M. then
0.25
0.50 Accurary
0.75
1.00
0.75
L.0
24 = mi 11 24 is defined as the latent variable
Accuracy
end
Figure 2. Illustration of TCE, calibratable and non conlcalibr Regimes, Assume that the target probabilistic generative model p' has 60% accuracy. Left: The TCE range (yellow) is bounded hetween C160% Accuracy| and |60% Accuracy| Right: The calibratable regime (red) spans from 0 to TCE when accuracy S 60%. The nollcalibr regime (blue) spans from a non-zero lower bound to TCE when accuracy > 60%.
end end
M-Step: // Calibrate model towards accuracy
for m : M do
Sm { y:), m, m...
4m = [m 2 (n.))ss.... 1 (argmax conf,, (x) v);
1
end
Update p(x;) by Equation (6), i = I...., M++1 + min + (CEce(p(a)), m(x)))):
Figure 2. What remains undiscussed are the models that in the white area above the line of 2 |60% Accuracy|- If model lies in this regime, it has a high ECE, as well as TCE, that is above 2 |60% Accuracy|. The model not well-trained and can be casily fine-tuned to move to one of the calibratable and non-calibratable regimes. In our experiments, we expand the notion of the calibratable regime from accuracy alone to its general performance.
end
loss:
LSTT, - log (ys(a;))
Through this fine-tuning, models learn Nomain-specifii knowledge. For instance, when trained the example in Table 1, the model learns that polysaccharides are very diffi- cult to induce antibody production. To further strengthen the model's domain knowledge, we also consider the following loss that focuses more on comprehending the question x::
5. Algorithms
The objective in Equation (2) can be reformulated mini- mizing a combination of accuracy loss and ECE loss using a Lagrange multiplier. In this section, we discuss how to approximate both losses specifically tailored for LLMs.
LITT, - log (v) + log ग (x"|x"
x2)
5.1. Approximating the Accuracy Loss for LLMs
where x' are the tokens of question x. Here, LSrT, or LSFT. play a role as ACC(.) in Equation (2) in the LLMs scenario.
In this session, we study why LLMs become poorly cali- brated after preference alignment. We first discuss the prefer- ence collapse phenomenon (Xiao et al., 2024a; Chakraborty al., 2024) in alignment methods like RLHF and DPO. When L.Mss are aligned with sample (2r, Dw 3t), they tend collapse their relative preference on Yue while ignoring yı- This typically results in preference ratio exceeding human preference proportions: T (Vbwl=)) l/) (T](l) + ( (x)) > PV > yn 3r)r). This preference collapse issue generalizes to unseen preference pairs (y1, U2). where LLMs collapse their preference to one option.
5.2. Approximating the ECE loss for LLMs
Now, we move our attention to the constraint ECE = 0 in Equation (2). Let the confidence of a model π be defined as conf, (x) = (x), (z) Po (z) where P, (z) is defined Equation (1). From probabilistic generative model perspective, the ECE loss can be written as:
LECF conf
where p(x) is some unknown probabilistic generative model and D is the divergence between two distributions. In our experiments,
In multiple-choice problems, this collapse manifests as LLMs strongly preferring option among A,B,C,D, re- gardless of its correctness. This leads to overconfidence in incorrect answers across many samples, resulting in poor calibration, as demonstrated in Section 6. To address this issue, we propose fine-tuning the model on domain speciffcc datasets, We formulate this as an SFT
use the Mean Square Error (MSE) loss. We use an Expectation-Maximzzaioo (EM) algorithm to estimate p(x).
For all E A data points with p(x) form a subset sharing the same label generation distribution. Since exam- ining all values of q E A would be impractical, we make
5

---

Restoring Calibration for Aligned Large Language Models: A Calibration-Aware Fine- Fine-Tuning Approach
the following simplification. First, for all q E A we define
In this regime, we find that with (A 0) or without (A 0) an explicit ECE regularization produces simi- lar results and we can simply report the unregularized one in our following experiments.
q = arg max q and consider the subset {ar] max p(x) 9)- Second, we discretize q E [0, 1] into M equal-width bins:
[o, 1/M) [M 1)/M, 1], consistent with traditional ECE estimation methods. Within the mith bin, the generative model assigns answers according to an unknown probabil- ity simplex where the largest probability equals qm- Let I arg maxp(2)) Then, we set P)=)(x)= = 9m- The prob- abilities (z) may vary across within the same bin. We apply mapping to align these probabilities with the model's 's confidence scores while maintaining the simplex constraints. We define p(z) as:
Regularized CFT (RCFT): We use LSrT, to fine-tune the models. This increases the model accuracy signif- icantly and moves the model to the non-calibratable regime, Therefore, we use an explicit ECE regulariza- tion with À = in Algorithm 1.
Evaluation Metries. In this paper, we use four metrics to evaluate the effectiveness of the proposed method. They are (1) class-wise Expected Calibration Error (cw-ECE), (2) Conference Expected Calibration Error (conf-ECE) (Guo et al., 2017), (3) Accuracy and (4) Win Rate where the first two are to evaluate the calibration performance (the lower the better). We use Accuracy and Win Rate to evaluate the ability of LLMS to understand languages and answer the questions correctly (the higher the better). The continuous cw-ECE and conf-ECE are defined in Def. 3.1 and 3.2. We use their discrete versions by stratifying all the samples into ten bins according to the prediction probability. The discrete versions can be found in Section C.I.
fm a tanh (yconf, (x)1,) + 3. i I,
j=l;
P5(x)
(6)
{
where the value of a and B discussed in Section A.4. The proposed EM-algorithm is provided in Algorithm I.
6. Experiments
6.1. Experimental Setup
Models. We evaluate our methods using four open-source
large language models: Llama-3.1-Tulu-888 (Lambert et al., 2024), Vicuna-7B-vl.5 (Chiang et al., 2023), Olmo 2- 7B (OLMo al., 2024), Mistral-7B (Jiang et al., 2023a), with detailed information provided in Appendix C.I.
The Win Rate metric measures how often model assigns higher sequence probability to the "preferred" response than to the "non- moon-preferred" one in pairwise comparison. Specifically, for each pair in the preference dataset, if the model ranks the chosen (preferred) response higher than the rejected one, that counts as "win," and the Win Rate is the fraction of wins among all comparisons. A more detailed description of Win Rate can be found in Appendix C.I.
We deliberately focus on these four models because each has undergone alignment using either RLHF (Rafailov et al., 2024b) or DPO (Rafailov et al., 2024b) and exhibits no- tably poor calibration. Specifically, Vicuna-7B is aligned via RLHF, while Mistral-7B, Olmo 2-7B, and Llama-3.1 1- Tulu-8B are aligned via DPO. Note that the original official Llama-3.1 1-BB does not provide a clear version among the three-stage pipeline (pre-training, SFT, alignment) and in our tests actually shows decent calibration performance, so we opt to work with the Tulu-8B variant for a more chal- lenging scenario. In this way, we ensure that our calibration methods are thoroughly evaluated against a range of align- ment strategies and baseline calibration outcomes.
We anticipate that the model adjusted using our approach will achieve a competitive Win Rate in comparison to the model prior to calibration. This will demonstrate that the good alignment performance brought by DPO/RLHF will not be diminished when we lift the calibration performance.
Datasets. We evaluate the ECE and Accuracy using four diverse datasets: MMLU, MedMCQA, OpenBookQA, Are- Challenge, and evaluate the Win Rate using AlpacaF- val, Arena-Hard, and UltraFeedback serraeeebbbk-BBnrizeedddd Binarized-Prefernncee The detailed descriptions of these datasets are provided in Appendix C.I.
Baselines. We validate the effectiveness of our methods by comparing performance across five methods:
RLHF or DPO Preference Alignment. Temperature Scaling (TS) (Guo et al., 2017): Intro- duces a temperature parameter T to scale the logits, minimizing ECE on a validation set to achieve better calibration.
Evaluation Schemes. We evaluate not only the in-domain performance (i.e., the training and testing data share the same domain), but also the out-domain (zero-shot) ability on an out-of-domain dataset. More specifically, we have two schemes:
Label Smoothing (Müller et al., 2019): Replaces hard one-hot labels with smooth label that assigns 1 e to the correct class and distributes & over the remaining classes
Scheme 1 (In-Domain): We combine the three datasets MMLU, MedMCQA, OpenBookQA into a single dataset and split it into training (3,000 samples) and testing (2,000 samples) datasets. The model is trained (calibrated) on the training dataset and evalu-
Calibration-Awart Fine-Tuning (CFT): In the calibrat- able regime, we use LSET, to fine-tune the models.
6

---

Restoring Calibration for Aligned Large Language Models: A Calibration-Aware Fine-Tuning Approach
Table Performance comparison among DP Temperature Scaling, Label Smoothing, CFT, and RCFT
four models
(Llama3,1-8--1u Vicuna-7B, Olmo2-7B, and Mistral-7B) in in-domain and out-domain scenarios. Best results in each metric block are bold. Blue highlights indicate superior in-domain conf ECE our CFT while red highlights denote best in-domain accuracy of our RCFT. "1"1"" means the smaller/larger the better. "-" means the results of Temp. Scale. are the same the original DPO/LHPP version.
conf-ECE. +
cw-ECE 4
Accuracy 1
Model
Method
In-Domain
Out-Domain
In-Domain
Out-Domain
In-Domain
Out-Domain
DPO
0.1953
0.1212
0.0953
0.0650
0.6228
0.7810
Llama3.1- 8B-Tulu
Temp. Scale.
0.1126
0.0679
0.0336
0.0514
Label Smooth.
0.1898
0.1009
0,0692
0.0639
0.6372
0.7116
Cur(S)rr)
0.0239
0.0688
0.0582
0.0375
0.6410
0,8000
RCFT(Ours)
0.0897
0.0810
0.0771
0.0526
0.8341
0.7991
RLHF
0.1422
0.0852
0.0979
0.0560
0.4344
0.5233
Vicuna-7B
Temp. Scale.
0.0598
0.0224
0,0488
0.0484
Label Smooth.
0.1221
0.0823
0.0517
0.0544
0.4517
0.5767
CFT(O)rs)
0.0379
0.0331
0.0583
0.0491
0.4481
0,6172
RCFT(Ourr)
0.0474
0.0672
0.0459
0.0530
0.6015
0.6035
DPO
0.1555
0.1325
0,0873
0.1331
0.6210
0.6635
Olmo2-7B
Temp. Scale.
0.0665
0.1160
0.0355
0.1196
Label Smooth.
0.1010
0.0499
0.0791
0.1298
0.6808
0.6431
CFT(Ours)
0.0544
0.0225
0.0804
0.0637
0.6606
0.7085
RCFF(Ourr)
0.0989
0.0781
0.0806
0.0707
0.8510
0.7099
DPO
0.2010
0.1318
0,0909
0.1103
0.6331
0.7567
Mistral-7B
Temp. Scale.
0.0802
0.0991
0.0399
0.0909
Label Smooth.
0.1874
0.1121
0.0900
0.0990
0.6479
0.6997
CFT(Ours)
0.0651
0.0424
0.0712
0.0614
0.6514
0.7863
RCFF((urr))
0.0979
0.0731
0.0877
0.0739
0.8297
0.7768
ated on the testing dataset. This setup aims to assess the calibration performance an in-domain scenario. Scheme (Out-Domain); The model is calibrated using the same training dataset as the In-Domain ap- proach, which consists of the three datasets: MMLU, MedMCQA, and OpenBookQA, totaling 3,000 sam- ples, and evaluated on the Arc-Challenge dataset (2,000 samples). This scenario tests the model's ability to transfer the calibrated capabilities from one domain to another, highlighting its zero-shot (generalization) potential.
Tulu, Vicuna-7B, Olmo2-7B, and Mistral-7B) in terms of calibration performance (conf-ECE, cw-ECE) and Accuracy under both in-domain and out-domain evaluations. Key insights include:
CFT significantly improves calibration while preserv- ing enhancing language capabilities. For Vicuna-7B, CFT reduces in-domain conf-ECE by 73% compared to DPO (0.0379 vs. 0.1422) while increasing out-domain accuracy to 0.6172 (vs. DPO's 0.5233). Similarly, for Mistral-7B, CFT achieves a 68% reduction in in-domain conf-ECE (0.0651 vs. DPO's 0.201) and maintains strong out-domain accuracy (0.7863 vs. DPO's 0.7567). CFT also outperforms Temperature Scaling and Label Smooth- ing in challenging scenarios, such reducing Olmo2-7B's out-domain conf-ECE to 0.0225 (vs. Label Smoothing's 0.0499 and Temperature Scaling's 0. 160). This verifies that DPO or RLHF- models typically lie the cali- bratable regime, where calibration can be restored without sacrificing language capabilities.
Given our computational resources (four A100 40G GPUs), employ the Quantized Low Rank (QLoRA) tech- nique (Dettmers et al., 2024) to fine-tune all models with rank 128, LoRA scaling parameter bfloat 6 precision on 2 A A100-40G GPUs. The train- ing conducted for epochs with batch size of 2 and a learning rate of 10-6
64, and
6.2. Numerical Results
RCFT prioritizes accuracy gains while maintaining
Table compares five alignment
Code is publicly available at https://github.com
Temperature Scaling, Label Smoothing, CFT (ours), and RCFT (aurs)-across across four language models (-IB

---

Restoring Calibration for Aligned Large Language Models: A Calibration-Aware Fine-Tuning Approach
Table Win rate comparisons among HP (DPO used in Table 3), CFT and RCFT across four models (IJama3. 1-8B-Tulu, Vicuna-7B, Olmo2-7B and Mistral-7B) on three datasets (AlpacaEval, Arena-Hard and Ultrafeedback). The best performance for each dataset is in bold. The competitive performance indicates that our methods can preserve the alignment performance.
AlpacaEval (vs DPO) CFT vs DPO
Alpa
Arena-Hard CFT
Ultrafeedback CFT
Model
RCFT vs DPO
DPO
CFT
RCFT
DPO
RCFT
DPO
RCFT
Llama-3.1-8B-
1-8BBB
51.68 vs 48.32
46.83 vs 53.16
21.4
22.6
19.6
44.6
45.0
43.6
0.7295
0.7460
0.7118
Vicuna-7B
46.46 vs 53.54
50.43 vs 49.57
2.60
2.60
3.60
1.00
1.00
1.00
0.2271
0.2279
0.2257
Olmo2-7B
62.48 vs 37.52
46.12 vs 53.88
24.2
22.9
23.1
19.4
19.2
20.2
0.7493
0.7588 0.7517
Mistral-7B
46.96 vs 53.04 49.81 vs 50.19
26.0
26.8
25.2
18.9
18.3
18.0
0.7066
0.7124 0.7221
Confidence Calibration Curve
Confidence Calibration Curve
Confidence Calibration Curve
Confidence Calibration Curve
1.0
10-
1.0
1.0
ürdercontidence
Undercontts oncee
Undercontieence
Undercontienne
Dvercontidence
Overcortidence
Overcontiserce
Overcentisence
0.8
0.8
0.8
Prob. of Correct
Prob. of Correct
Prob. of Correct
0.6-
0.6
0.6
0.5
Prob. of
0.21
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.5
0.8
1.0
Prob. of Answer
Prob. of Answer
Prob. of Answer
Prob. of Answer
(a) DPO (9.3)
(b) TS (0.1 126)
(c) CFT (0.0239)
(d) RCFT (0.0897)
Figure 3. Calibration Plots of (a) DPO, (b) Temperature Scaling (TS), (c) our CFT, (d) our RCFT Llama-3. 1----...1------- 1-8B-Tulu. Each panel plots the model's predicted probabilities (i.e., confidence) on the r-axis against the observed accuracy (fraction correct) on the y-axis, binned into ten groups. The diagonal line in each panel represents perfect calibration. The depth of the color indicates the sample density in that column. DPO has the worst calibration performance. Other three methods improve the calibration performance where our CFT has the lowest con-ECE (shown in the parenthesis). This figure omits the first bins because the model selects answer with the largest predicted probability which is always larger than 0.25 in the four options prediction task (so no samples exist below that threshold).
competitive calibration. For Llama3.1 1-8B-Tulu, RCFT achieves the highest in-domain accuracy (0.8341) compared to DPO (0.6228) but exhibits slightly higher calibration er- rors (in-domain Conf-ECE: 0.0897 vs. CFT's 0.0239). This trend holds for Olmo2-7B, where RCFT boosts in-domain accuracy 0.851 (vs. DPO's 0.621 and Label Smooth- ing's 's 0.6808) with modest conf-ECE (0.0989). Here, the models move to the non-calibratable regime. While RCFT trades some calibration performance for accuracy, it remains comparable to Temperature Scaling, making it suitable for accuracy-critica applications.
verse benchmarks. Table 3 demonstrates this through three complementary evaluation protocols:
1. AlpacaEval Head-to-Head (CFT/RCFT vs DPO): Re- sponses from our methods and the original DPO model were directly compared by GPT-4 (judge):
Protocol: Same prompts - CFT/RCFT vs DPO re- sponses + GPT-4 preference
Results:
Statistical parity (e.g.. Vicuna-7B: 50.43%
RCFT vs 49.57% DPO)
2. Standard AlpacaEval (vs GPT-4o):: Methods were evaluated against the standard GPT-4o baseline:
Baseline methods show consistent limitations. Temper- ature Scaling reduces calibration errors (e.g.. Vicuna-7B conf-ECE: 0.0598 vs. 0.1422) but cannot improve accu- racy (marked by "-"). Label Smoothing provides moderate calibration benefits but underperforms our methods example, on Mistral-7B, its in-domain conf-ECE (8.1874) is significantly higher than CFT's (0.0651), and its accuracy gains (+1.5% over DPO) are dwarfed by RCFT's +31 %. DPO/RLHF consistently shows the poorest calibration (e.g.. Mistral-7B conf-ECE: 0.201), highlighting the need for spe- cialized techniques.
Protocol: Responses vs GPT-4o - GPT-4 preference
Results: CFT achieved best performance for Llama3.1 (22.6%) and Mistral-7B (26.8%)
3. Arena-Hard (vs GPT-4): Responses were compared
against the standard GPT-4 baseline?:
Protocol: Responses vs GPT-4 - GPT-40 preference
Results: RCFT achieved best for Olmo2-7B (20.2%);
'The value of the instruct version of Llama-3.1 3.1--B is 20.6%. is not clear why the Tulu version has significant improvement.
CFT and RCFT preserve alignment quality across di-
8

---

Restoring Calibration for Aligned Large Language Models: A Calibration-Aware Fine-Tuning Approach
CFT showed gains (Llama3.1: I: 45.0% vs DPO's 44.6%)
dence 0.25 in the four-choice task, the first two bins effectively empty and thus omitted. Overall, DPO is sub- stantially miscalibrated, while TS, CFT, and RCFT bring the model's confidence closer to true accuracy, with CFT having the best calibration (lowest conf-ECH). Other mod- els share similar phenomenon. We defer other results to Appendix C.3.
4. Ultrafeedback Binary Selection: Models performed
preference classification:
Protocol:
Given question, response A, response B ->
Select better response
Results: CFT improved win rates for all models (e.g.,
Olmo2-7B. 0.7588 vs DPO's 0.7493)
To conclude this section, we present an ablation study ex- amining the effects of X. Using only LSFT, achieves high accuracy ( (90%) but shows poor calibration. Conversely, using only LECE drives the model toward random guess- ing, achieving near-zero ECE but with only 25% accuracy. These results, detailed in Section C.2, demonstrate the im- portance of balancing both objectives to achieve both good calibration and performance.
In summary, CFT excels in calibration--senitive tasks while RCFT dominates accuracy-critical applications. Both methods outperform existing techniques (including Label Smoothing), preserve alignment quality across all benchmarks, and demonstrate robust generalization across models and evaluation scenarios.
6.3. Calibration Plots
7. Conclusion
Besides the numerical results, we also illustrate the cali- bration plots to provide a more intuitive display. Figure 3 shows the confidence calibration plots for the four methods: DPO, Temperature Scaling (TS), CFT, and RCFT on Llama- 3.l..B-Tulu. The horizontal axis of each panel indicates the model's predicted probability (confidence), while the vertical axis is the observed accuracy for predictions within that confidence bin.
In this work, we have investigated the critical issue of cal- ibration degradation in LLMs following preference align- ment procedures. To address this, we introduced a theo- retical framework distinguishing between calibratable and non-calibratable regimes, and developed practical solutions through calibration- fine-tuning approaches. Our ex- perimental results across multiple models demonstrate that our methods can significantly improve calibration while maintaining or enhancing model performance. Future work could explore extending these methods to other types of language tasks and investigating the relationship between calibration and other aspects of model reliability.
The diagonal line in each plot indicates perfect calibration, meaning model's pre- dicted confidence aligns exactly with its true probabil- ity of correctness. Bars above the diagonal are under- confident; bars below the diagonal are overconfident. (a) DPO (0.1953): Direct Preference Optimization shows the worst calibration. Its con-ECE value (0. (053) is the highest among the four, and the de- viation of its yellow bars from the diagonal line is rela- tively large, indicating substantial mismatch between predicted probabilities and actual accuracies.
Perfect calibration:
Limitation and Future Work. Our findings suggest sev- eral promising directions for research on LLM calibration. First, determining which regime model falls into ultimately reduces to understanding whether its accuracy exceeds a certain threshold. This, in turn, depends on two key fac- tors: 1. What is the accuracy threshold for given neural network architecture? 2. Given such an architecture, which training algorithms lead the model to fall into each regime? Answering these questions requires deeper theoretical anal- ysis of the properties of transformers, which is currently an open and challenging direction. Second, from a theoretical perspective, determining the constant in Theorem 4.5 is nontrivial, and we leave it as an open problem for future work. Third, our current study focuses on multiple-choiee settings; extending the method to free-form generation is an important direction for future investigation. Fourth, ex- ploring the effectiveness of our approach in scenarios where the LLM is initially poorly calibrated remains an impor- tant direction. Finally, exploring the calibration properties of quantized models, black-box API models, and whether low-rank adaptation methods such as LoRA suffer from calibration issues also presents valuable avenues for future research.
(b) TS (0.1126); Temperature Scaling reduces over- confidence compared to DPO, bringing the bars closer to the diagonal. Its con- con-ECE drops to 0. 126, reflecting a moderate improvement in calibration. (c) CFT (0.0239): This method demonstrates much stronger calibration. The bars closely track the diago- nal line, and its con-ECE of 0.0239 is the lowest among the four methods. This indicates that CFT is nearly well-calibrated for the task.
(d) RCFT (0.0897): While RCFT is not as well- calibrated as CFT, still surpasses DPO and TS in bringing predicted probabilities in line with actual ac- curacies. Its con-ECE is 0.0897, representing a moder- ate level of calibration improvement. Note that RCFT has a better accuracy as indicated in Table 2.
In all panels, darker yellow bars indicate higher density of samples within that bin. Since the models' largest confi-

---

Restoring Calibration for Aligned Large Language Models: A Calibration-Aware Fine-Tuning Approach
Impact Statement
Chen, Y., Yuan, L.,, Cui, G., Liu, Z., and Ji, H. A close look into the calibration of pre-trained language models. arXiv preprint 2tt.::22211.0011000 2022.
Our work advances the reliability and trustworthiness of large language models by improving their calibration while maintaining alignment with human preferences. The pri- mary positive impact is enabling safer deployment of these models in real-world applications by providing better- calibrated confidence estimates, particularly crucial in high- stakes domains. However, we acknowledge that these tech- niques must be implemented thoughtfully to avoid potential misuse in creating deceptively confident models. We en- courage practitioners to carefully consider these trade-offs when applying our methods.
Chiang, W.-L.., Li, Z.,, Lin, Z.., Sheng, Y., Wu, Z., Zhang, H., Zheng, L., Zhuang, S., Zhuang, Y., Gonzalez, J. E., al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* * chatgpt quality. See https://vicuna. Imsys. org (accessed 14 April 2023), 2(3):6, 2023.
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1-13,, 2023.
Acknowledgments
Christiano, F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D. Deep reinforcement learning from human preferences. Advances in neural information pro- cessing systems, 30, 2017.
We would like to thank all the anonymous reviewers for their comments and suggestions. This work was conducted while Ruochen Jin was a visiting scholar at the University of Pennsylvania. This work was supported in part by NIH grants U01CA274576 and P30AG073105, ARPA-H Award D24AC00253, NSF grant DMS-2310679, a Meta Faculty Research Award, Wharton Al for Business, and PSOM A12D Seeding Project.
Dettmers, T., Pagnoni, A., Holtzman, A., and Zettlemoyer, L. Qlora: Efficient finetuning of quantized 1lms. Advances in Neural Information Processing Systems, 36, 2024.
Fujisawa, M. and Futami, Pac-bayes analysis for recalibra- tion in classification. arXiv preprint 22Xx::24006062222. 2024.
References
Abbas, M., Zhou, Y., Ram, P., Baracaldo, N., Samulowitz, H., Salonidis, T., and Chen, T. Enhancing in-context learning via linear probe calibration. In Proceedings of The 27th International Conference on Artificial Intelli- gence and Statistics, pp. 307-315, 2024.
Futami, F. and Fujisawa, M. Information-theoreti gener- alization analysis for expected calibration error. In The Thirty-eightt Annual Conference on Neural Information Processing Systems, 2024.
Gorbatovski, A., Shaposhnikov, B., Malakhov, A., Sur- nachev, N., Aksenov, Y., Maksimov, I., Balagansky, N., and Gavrilov, D. Learn your reference model for real good alignment. arXiv preprint arXiv:240.009656, 2024.
Anthropic, A. The claude 3 model family: Opus, sonnet, haiku. Claude-3 Model Card, 2024.
Azar, M. G. Guo, Z. D., Piot, B., Munos, R., Rowland, M., Valko, M., and Calandriello, D. A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, pp. 4447-4455. PMLR, 2024.
Gruber, S. and Buettner, F. Better uncertainty calibration via proper scores for classification and beyond. Advances in Neural Information Processing Systems, 35:8618 15:8118-632,, 2022.
Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lec, Y. T., Li, Y., Lundberg, S., al. Sparks of artificial general intel- ligence: Early experiments with gpt-4, arXiv preprint arXiv:2303.12712. 2023.
Guo, C., Pleiss, G., Sun, Y., and Weinberger, K. Q. On calibration of modern neural networks. In International conference on machine learning, pp. 1321-1336 PMLR, 2017.
Chakraborty, S., Qiu, J., Yuan, H., Koppel, A., Huang, F,, Manocha, D., Bedi, A. S., and Wang, M. Maxmin- rlhf: Towards equitable alignment of large language models with diverse human preferences. arXiv preprint arXiv:2402.08925 2024.
Gupta, C. and Ramdas, A. Distribution-free calibration guarantees for histogram binning without sample split- ting. In International conference on machine learning, pp. 3942-3952, PMLR, 2021.
Gupta, C., Podkopaev, A., and Ramdas. A. Distribution- binary classification: prediction sets, confidence inter- vals and calibration. Advances in Neural Information Processing Systems, 33:3711-3722 2020.
Chang, J. D., Shan, W., Oertell, O., Brantley, K., Misra, D., Lee, J. D., and Sun, W. Dataset reset policy optimization for rlhf, arXiv preprint arXiv:2404.00 08495, 2024.
10

---

Restoring Calibration for Aligned Large Language Models: A Calibration-Aware Fine-Tuning Approach
Han, H., Li, T., Chen, S., Shi, J., Du, C., Xiao, Y., Liang, J., and Lin, X. Enhancing confidence expression in large language models through learning from past experience. arXiv preprint 2rXiv:2404.1000 10315, 2024.
dirichlet calibration. Advances in neural information processing systems, 32, 2019.
Kumar, A., Sarawagi, S., and Jain, U. Trainable calibration measures for neural networks from kernel mean embed- dings. In International Conference Machine Learning, pp. 2805-20880 2015-2814. 14. PMLR, 2018.
Han, Z., Hao, Y., Dong, L., Sun, Y., and Wei, F. Prototypi- cal calibration for few-shot learning of language models. In The Eleventh International Conference on Learning Representations, 2023.
Kumar, A., Liang, P. S., and Ma. T. Verified uncertainty calibration. Advances in Neural Information Processing Systems, 32, 2019.
Jiang, A. Q.. Sablayrolles, A., Mensch, A., Bamford, C.,, Chaplot, D. S., Casas, D. d. I., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al. Mistral 7b. arXiv preprint arXiv: 10.00685,, 2023a.
Lambert, N., Morrison, J., Pyatkin, V., Huang, S., Ivison, H., Brahman, F., Miranda, L. J. V., Liu, A., Dziri, N., Lyu, S., et al. Tę" ulu 3: Pushing frontiers in open language model post-training. arXiv preprint arXiv:2411.15124 2024.
Jiang, 7., Araki, J., Ding, H., and Neubig, G. How can we know when language models know? on the calibration of language models for question answering. Transactions of the Association for Computational Linguistics, 9:962- 977, 2021.
Leng, J., Huang, C., Zhu, B., and Huang. J. Taming overcon- fidence in llms: Reward calibration in rlhf. arXiv preprint 2410022410.09724, 2024.
Jiang, Z., Zhang, Y.. Liu, C., Zhao. J.. and Liu, K. Gen- erative calibration for in-context learning. The 2023 Conference on Empirical Methods in Natural Language Processing, 2023b.
Li, Z., Xu. T., and Yu, Y. Policy optimization in rlhf: The impact of out-of-preference data. arXiv preprint arXiv: 2023a.
Jin, R., Hou, B., Xiao, J., Su, W. J., and Shen, L. Fine- tuning attention modules only: Enhancing weight dis- entanglement in task arithmetic. In The Thirteenth In- ternational Conference on Learning Representations, www.:eeetrrrrrrr
Li, Z., Xu, T., Zhang, Y., Yu, Y., Sun, R., and Luo, Z.- Q. Remax: A simple, effective, and efficient method for aligning large language models. arXiv preprint arXiv:2310.1083 10505, 2023b.
2025. URL id- -d-jjOOkKJVVI.
Li, Z., Chen, C., Xu, T., Qin, 7.,, Xiao, J., Luo, Z.-Q.. and Sun, R. Preserving diversity in supervised fine- tuning of large language models. In The Thirteenth International Conference on Learning Representations, 2025. URL https: hwtp///weeeeeeeeeeeeeeeeeee net/ forum? id=NCEe7B7bS.
Joy, T., Pinto, F., Lim, S.-N., Torr, P. H., and Dokania, P. K. Sample-dependent adaptive temperature scaling for improved calibration. In Proceedings of the AAAI Con- ference Artificial Intelligence, volume 37, pp. 14919- 14926, 2023.
Kadavath, S., Conerly. T., Askell, A., Henighan, T., Drain, D., Perez, E., Schiefer, N., Hatfield-Dodss Z., DasSarma, N., Tran-Johnson, E., et al. Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221, 2022.
Lin, S., Hilton, J., and Evans, O. Teaching models to express their uncertainty in words. Transactions on Machine Learning Research, 2022. ISSN 2835-8856.
Liu, K.. Long, Q.. Shi, Z., Su, W. J.. and Xiao, Statistical impossibility and possibility of aligning lims with human preferences: From condorcet paradox to nash equilibrium. arXiv preprint arXiv:2503. 10990, 2025.
Karandikar, A., Cain, N., Tran, D., Lakshminarayanan, B., Shlens, J., Mozer, M. C., and Roelofs, B. Soft calibra- tion objectives for neural networks. Advances in Neural Information Processing Systems, 34:29768-29777 2021.
Liu, T., Zhao, Y., Joshi, R., Khalman, M., Saleh, M., Liu, P. J., and Liu, J. Statistical rejection sam- pling improves preference optimization. arXiv preprint arXiv:2309.066 066006657, 2023.
Krishnan, R. and Tickoo, O. Improving model calibration with accuracy versus uncertainty optimization. Advances in Neural Information Processing Systems, 33:18237- 18248, 2020.
Mukhoti, J., Kulharia, V., Sanyal, A., Golodetz, S., Torr, P., and Dokania, P. Calibrating deep neural networks using focal loss. Advances in Neural Information Processing Systems, 33:15285 0a2115588-59999 2020.
Kull, M., Perello Nieto, M., Kängsepp. M., Silva Filho, T., Song, H., and Flach, Beyond temperature scaling: Obtaining well-calibrated multi-class probabilities with

---

Restoring Calibration for Aligned Large Language Models: A Calibration-Aware Fine-Tuning Approach
Müller, R., Kornblith, S., and Hinton, G. E. When does label smoothing help? Advances in neural information processing systems, 32, 2019.
Shi, Z., Liu, K., Long, Q. Su, W. J., and Xiao, J. Fun- damental limits of game-theoretic IIm alignment: Smith consistency and preference matching. arXiv preprint arXiv:2505.2062) 2025.
Munos, R., Valko, M., Calandriello, D., Azar, M. G., Row- land, M., Guo, Z. D., Tang, Y., Geist, M., Mesnard, T., Michi, A., et al. Nash learning from human feedback. arXiv preprint (t..Xi:2322008880 2023.
Sun, Z., Song, D., and Hero, A. Minimum-riss recalibration of classifiers. Advances Neural Information Processing Systems, 36:69505-69531 2023,
OLMo, T., Walsh, P., Soldaini, L., Groeneveld, D., Lo, K., Arora, S., Bhagia, A., Gu, Y., Huang, S., Jordan, M., et al. 2 olmo furious. arXiv preprint arXiv:2501.00653 2024.
Szegedy, C., Vanhoucke, V., loffe, S., Shlens, J., and Wojna, Z. Rethinking the inception architecture for computer vi- sion. In Proceedings of the IEEE conference on computer vision and pattern recogmition, pp. 2818-2826, 2016.
OpenAl. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
Tajwar, F., Singh, A., Sharma, A., Rafailov, R., Schneider, J., Xie, T., Ermon, S., Finn, C., and Kumar, A. Preference fine-tuming of Ilms should leverage suboptimal, on-policy data. arXiv preprint arXiv: 24X.::2444.14367 2024.
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright,
C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L.,, Simens, M., Askell, A., Welinder, P., Christiano, P., Leike, J., and Lowe, R. Training language models to follow instructions with human feedback, 2022.
Tang, Y., Guo, D. Z., Zheng, Z., Calandriello, D., Cao, Y., Tarassov, E., Munos, R., Pires, B, Á., Valko, M., Cheng, Y., et al. Understanding the performance gap between online and offline alignment algorithms. arXiv preprint mrxXiv:205.0884 00448888.., 2024.
Popordanoska, T., Sayer, R., and Blaschko, M. A consistent and differentiable ip canonical calibration error estimator. Advances in Neural Information Processing Systems, 35: 7933-7946, 2022.
Tao, S., Yao, L., Ding, H., Xie, Y., Cao, Q. Sun, F., Gao, J., Shen, H., and Ding, B. When to trust Ilms: Aligning confidence with response quality. In Findings of the Association for Computational Linguistics ACL 2024, pp. 5984-5900 2024.
Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., and Finn, C. Direct preference optimization: Your language model is secretly reward model, 2023.
Team, G., Anil, R., Borgeaud, S., Wu, Y., Alayrac, J.-B., Yu, J.. Soricut, R., Schalkwyk. J., Dai, A. M.. Hauth, A., et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312. 1805, 2023.
Rafailov, R., Hejna, J., Park, R., and Finn, C. From r to q': Your language model is secretly a q-function. arXiv preprint arXiv:2404.12355 12358, 2024a.
Rafailov, R., Sharma, A., Mitchell, E., Manning, C., Er- mon, S., and Finn, C. Direct preference optimization: Your language model is secretly reward model. Ad- vances in Neural Information Processing Systems, 36, 2024b.
Tian, K., Mitchell, E., Zhou, A., Sharma, A., Rafailov, R., Yao, H. Finn, C., and Manning, C. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feed- back. In Proceedings of the 2023 Conference on Empiri- cal Methods in Natural Language Processing, pp. 5433- 5442, December 2023.
Savage, T., Wang, J., Gallo, R., Boukil, A., Patel, V., Safavi- Naini, S. A. A., Soroush, A., and Chen, J. H. Large language model uncertainty proxies: discrimination and calibration for medical diagnosis and treatment. Journal of the American Medical Informatics Association, 32(1): 139-149, 2025.
Touvron, H., Martin, L.,, Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine- tuned chat models. arXiv preprint arXiv:2307.09288. 2023.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. 2017. URL https:
ArXiv preprint, abs// //aaxiiv..o http://.ggaabb////a/aaaa 06347.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L.. Gomez, A. N., Kaiser, L., and Polosukhin, I. Atten- tion is all you need. In Advances in Neural Information Processing Systems 30, pp.
Shen, M., Das, S., Greenewald, K., Sattigeri, P., Wornell, G., and Ghosh, S. Thermometer: Towards universal calibration for large language models. arXiv preprint arXiv:2403.00 08809, 19, 2024.
2017.
Wang, M., Ma, C., Chen, Q., Meng, L., Han, Y., Xiao, J., Zhang, Z.,, Huo, J., Su, W. J., and Yang, Y. Magnetic
12

---

Restoring Calibration for Aligned Large Language Models: A Calibration-Aware Fine-Tuning Approach
preference optimization: Achieving last-iterate conver- gence for language model alignment. In The Thirteenth International Conference Learning Representations, 2025. URL https: http///ppppeeeeeeeeeeeeeeeeeeeet wwwpppeeewwweeeeeeeeee...eeeeeee 1d=PDnEDS244P.
Yu, Y., Bates, S., Ma, Y., and Jordan, M. Robust calibration with multi-domain temperature scaling. Advances in Neu- ral Information Processing Systems, 35:27510-27523, 2022.
Zadrozny, B. and Elkan, C. Transforming classifier scores into accurate multiclass probability estimates. In Proceed- ings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 694 699, 2002.
Wu, C. J. On the convergence properties of the em algorithm. The Annals of statistics, pp. 95-103, 1983.
Xiao, J., Sun, R., and Luo, Z.-Q. Pac-bayesian spectrally- normalized bounds for adversarially robust generalization. Advances in Neural Information Processing Systems, 36: 36305-36323, 2023.
Zhang, H., Cisse, M., Dauphin, Y. N., and Lopez-Paz, D. mixup: Beyond empirical risk minimization. arXiv preprint arXiv: 1710.0921 2017.
Xiao, J., Li, Z., Xie, X., Getzen, E., Fang, C., Long, Q., and Su, W. J. On the algorithmic bias of aligning large lan- guage models with rlhf: Preference collapse and matching regularization. arXiv preprint 240.:.:2000.6.45555 2024a.
Zhang, S., Gong, C., and Choi, E. Knowing more about questions can help: Improving calibration in question answering. arXiv preprint arXiv:2106.01 19940099.114 2021.
Xiao, J., Sun, R., Long, Q.. Su, W., et al. Bridging the gap: Rademacher complexity in robust and standard gen- cralization. In The Thirty Seventh Annual Conference on Learning Theory, pp. 5074-50777 PMLR, 2024b.
Zhao, Z., Wallace, E., Feng, S., Klein, D., and Singh, Calibrate before use: Improving few-shot performance of language models. In International conference on machine learning, pp. 12697- -1706.. PMLR, 2021.
Xiao, J., Zhang. J., Luo, Z.-Q., and Ozdaglar, A. E. Uni- formly stable algorithms for adversarial training and be- yond. In International Conference on Machine Learning, pp. 54319-54340. PMLR, 2024c.
Xiao, Y., Liang, P. P., Bhatt, U., Neiswanger, W., Salakhut- dinov, R., and Morency, L.-P. Uncertainty quantification with pre-trained language models: A large-scatt scale empiri- cal analysis. arXiv preprint wtxiv:2220.04170
2022.
Xiong, M., Hu, Z., Lu, X., Li, Y., Fu, J., He, J., and Hooi, B. Can Ilms express their uncertainty? an empirical evaluation of confidence elicitation in llms. arXiv preprint arXiv:2306. 13063, 2023.
Xu, S., Fu, W., Gao, J., Ye, W., Liu, W., Mei, Z.,, Wang, G., Yu, C., and Wu, Y. Is dpo superior to ppo for Ilm alignment? comprehensive study. arXiv preprint 10X1v:2404.10719, 2024a.
Xu, T., Wu, S., Diao, S., Liu, X., Wang, X., Chen, Y., and Gao, J. Sayself: Teaching Ilms to express confidence with self-reflective rationales. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 5985-5998, 2024b.
Yang, J. Q., Salamatian, S., Sun, Z., Suresh, A. T., and Beirami, A. Asymptotics of language model alignment. arXiv preprint arXiv:2404.07730 2024.
Ye, C., Xiong, W., Zhang, Y., Jiang, N., and Zhang, T. A the- oretical analysis of nash learning from human feedback under general kl regulazard preference. arXiv preprint arXiv:2402.073/ 14, 2024.
13

---

Restoring Calibration for Aligned Large Language Models: A Calibration-Aware Fine-Tuning Approach
A. Additional Discussion of Calibration
A.1. Preference Optimization
RLHF. RLHF is an approach to align LLMs with human preferences, ensuring that the models are useful and safe for their human users (Ouyang al., 2022). The framework for learning LLMs typically consists of the following three steps: (1) supervised fine-tuning (SFT), (2) Reward modeling, and (3) RLHF fine-tuning. The loss function of RLHF fine-tuning
max maxByyn (1e y) (F (F(())))
where 8 > 0 is parameter controlling the deviation from the base reference policy Tee, namely the initial SFT model TSFT. DxL. is the Bellback--eibblr (KL) divergence between Ty (y(a)) and [
DPO, The original DPO method (Rafailov et al., 2023) is to directly optimize the policy without explicitly training the reward function in supervised manner:
Blog (V(()(z)) [yr()/lzz)
To (u (z:) Teer [1 xz)
E (( log
33 log
A.2. Other Related Work
Several preference fine-tuning methods have been proposed for RLHE. Li et al. (2023b) showed that proximal policy optimization (PPO) (Schulman et al., 2017) does not fully exploit the potential of RLHF in aligning LLMs with human preferences. Tang et al. (2024) examined the performance gap between online and offline algorithms for alignment tasks, while Ye et al. (2024) introduced an online iterative RLHF algorithm that incorporates a general preference model. Li et al. (2025) studied the diversity of models trained during the SFT phase. Beyond fine-tuning, model editing (Jin al., 2025) has emerged as another approach to adapt model behavior to various tasks.
Several notable variants of DPO have been developed (Liu et al., 2023; Azar et al., 2024; Chang et al., 2024; Gorbatovski et al., 2024; Rafailov et al., 2024a; Yang et al., 2024). However, recent studies (Li et al., 2023a; Xu et al., 2024a; Tajwar et al., 2024) suggest that DPO is less effective than reward-based RLHF methods for aligning LLMs. Both Li et al. (2023a) and Xu et al. (2024a) attributed this shortcoming to representation misspecification in DPO, which limits its ability to achieve robust alignment compared to reinforcement learning approaches such as PPO. Additionally, the on-policy nature of reward-based fine-tuning helps mitigate distribution shifts between the training data and online responses, thereby enhancing
LLM performance (Tajwar et al., 2024).
Nash learning from human feedback (NLHF) has also been proposed as framework to align LLMs with general preference models (Munos et al., 2023; Liu et al., 2025; Shi et al., 2025). Moreover, Wang et al. (2025) analyzed the convergence behavior of NLHF, showing that it approaches the Nash equilibrium in the last iterate.
Prior work (Guo et al., 2017; Müller et al., 2019) has shown that temperature scaling is generally more effective than label smoothing and other techniques for improving calibration. Zhao al. (2021) proposed a contextual calibration procedure to improve few-shot performance of language models. Predictive accuracy and calibration trade-off have been studied in general machine learning classification problems (Kumar et al., 2018; Krishnan & Tickoo, 2020; Karandikar al., 2021; Popordanoska et al., 2022). Additionally, several studies have provided theoretical analyses of the bias in ECE estimators based on miform-mass binning (UMB) (Gupta et al., 2020; Gupta & Ramdas, 2021). Other related works include Gruber & Buettner (2022) and Sun et al. (2023), which explore the framework of proper calibration errors and minimum-rise recalibration of classifiers.
While temperature scaling (TS) is strong baseline, our method's superiority stems from addressing fundamental limitations of post-hoc calibration LLMs:
Direct optimization of calibration: Our approach explicitly minimizes the diserepancy between accuracy and confidence-thh definition of ECF-rathher than relying a single scaling parameter. This direct optimization improves calibration while maintaining or enhancing performance. This aligns with previous work on training classifiers with calibration objectives, such as the AvUC loss in Krishman & Tickoo (2020) and the PAC-Bayes- objective in Fujisawa & Futami (2024), which demonstrated the superiority of optimization based strategies over host-hoe methods.
14

---

Restoring Calibration for Aligned Large Language Models: A Calibration-Aware Fine-Tuning Approach
Improved generalization under distribution shift: Our approach generalizes better unseen data and distribution shifts compared to TS, which risks overfitting specific validation sets. Table 2 clearly demonstrates this advantage in out-domain scenarios-for example, with Olmo2-7B, our CFT method reduces out-domain cw-ECE to 0.0637 (vs. TS's 0. 196) while improving accuracy to 7070 (vs. DPO's 0.6635). Distribution shifts are common in language tasks, and TS is insufficient to handle such variations.
A.3. Multiclass Calibration
Definition A.1 (Multiclass Calibration). A probabilistic classifier P X - As is multiclass-calibratdd or simply calibrated, for any prediction vector = (q),.... q&) E As. the proportions of classes among all possible instances æ: getting the same prediction p(x) = q are equal to the prediction vector q:
ply jlp(z) q) 9h. j 1.2....k.
(7)
A widely used metric for evaluating confidence calibration is the Expected Calibration Error (ECE) Naeini et al. (2015), which is defined as the expected absolute difference between the model's confidence and its accuracy. Miticlass-ECE (mc-ECE) defined as:
k = j((x)) (z)x
me-ECE = -Ep(())
Relationships Between Calibration Types.
The following relationships hold:
Multiclass calibration implies classwise calibration and confidence calibration.
Classwise calibration implies confidence calibration.
A.4. Rank Preserving Mapping
We denote the model confidence vector as Conf, (z) = [c1, c22 c3, cı). where c1 2
2 c3 2 c₁ without loss of generality, Let the calibrated confidence be p(x) = [c], c2 cg. c4]- We set the top-1 calibrated confidence match the bin accuracy:
dj
4m-
To preserve the rank ordering of all four options, we aim to compress the remaining three confidence values c2, сa G4 into the interval (0, gm) while ensuring their sum equals scaling:
In- We apply nonlinear transformation followed by linear
&f tamh(74,) + + for = 2, 3. 4.
To control the saturation level of the non-linear function tanh(-), we select y such that the largest among C2 C3, maps close to 1:
Determining
(anh (7 max(c2, ca c4)) = 0.99.
To ensure numerical stability, we avoid tanh-1 (1) and use the approximation at.99)(O.99)) In(3). This gives:
ln(3)
max C3, ca} = In(3),
max C3, c4
4m
Solving for and | To satisfy the constraint that Ei c Oma define:
T = tanh(yc,).
Then:
c
(a tann((ycc) + aT +33 4m
(8)
15

---

Restoring Calibration for Aligned Large Language Models: A Calibration-Aware Tine-Tuning Approach
We solve for a and in two ways:
(a) Simplified assumption: 3 = a Substitute into Equation (8):
aT' + 3a
1 qm
(
aTT + 3)
Qm
(10)
1 4m
a
3 = a.
(11)
T + 3
(b) General solution (no assumption):
From Equation (8), isolate 3::
aT +3 1 qm
(12)
8 qm af
(13)
Thus, given any a, 3 can be directly computed.
<m Ensuring Rank Preservation To preserve the ordering of < ym for i 2,3, 4, enforce:
c = a tanh(rc)))) B qm-
Since (an)(70s) < 1, a sufficient condition
+ qm.
Substitute the simplified solution 3 = a:
22x 4m
Then:
a < 9m
2a Qm
+
(14)
Um
1 4m
4n
and since a
(15)
T 3
Multiply both sides of Equation (15) by (T + 3):
2(1 - (m) < 4m
5m (T + 3),
(16)
2 < 9m (T + 5)
(17)
2
mm T +
(18)
This inequality gives general bound 9m for the mapping to preserve the ordering. Noting that T = Ei= tanh(yc;) < 3. we conservatively estimate 7 < 3:
2
2 8
Qm
0.25.
7+5
Conclusion When pm > 0.25, the parameters a and 3 can be chosen such that the transformed confidences df satisfy of < 9m and preserve the original rank ordering. When 9m S 0.25, the non-linear mapping using tanh(y.) compresses all cf (for 2,3.4) 4) near 1, effectively reflecting high uncertainty with low distinguishability across non-top predictions.
16

---

Restoring Calibration for Aligned Large Language Models: A Calibration-Aware Fine-Tuning Approach
A.5. Convergence of EM Algorithms
The convergence of EM algorithms is well studied; see, for example, Wu (1983), To ensure the convergence of the EM algorithm in our setting, three additional assumptions are required.
First, for the ECE loss,
LECE ((p(())))
assume that D is the cross-entropy loss. In this case,
LECR -Ep(s) [log conf (x)]
Under this assumption, the loss function becomes negative log-likelhhoo,, which aligns with standard convergence theory for EM algorithms. However, practice, alternative distance measures may also be used.
Second, the binary choice setting also required. In this setting, the probability of the false answer uniquely determined once the probability of the true answer is given. However, in the multiple-choice setting, the generative model assigns probability only the correct answer, without imposing any constraints the probabilities of the remaining choices. As result, these probabilities do not influence the loss function, and the EM algorithm is not uniquely defined in this case. Therefore, the convergence guarantee only applies in the binary setting. In practice, we can use the mapping discussed above to assign probabilities to the remaining choices.
Finally, the optimization performed over the choice probabilities rather than the neural network parameters, as the convergence of training deep neural networks remains an open problem.
B. Proof of Technical Results
B.1. Proof of Proposition 4.1
Proof. Based on the definition of the probabilistic generative model p, for all and v, we have
P(y = jlp()) =())).
Then,
mc-FCE(p) = Ey IP(y = jlp(x)) -P)(2)) = 0.
1=1
B.2. Proof of Theorem 4.4
Proof. We first recap the definition of TCE:
TCE Ex
We partition all the samples into two subsets.
ST {xl arg max n T'()) (x) = y)
Sj = { arg max π (x) + y)
Denote ACC(=*) = a". We first assume that a² < a. We then partition Sp in to two parts: Sp and Spg, with
Pfx E SRI) = a"a"
and
Pfa E SE2)
17

---

Restoring Calibration for Aligned Large Language Models: A Calibration-Aware Fine-Tuning Approach Then, We define (xx) follows. When S1 or Sp2-
<(x))= = = (x).
In this case, 1 ブブ= 103(z) p)(()) = O.
When x E Sp.1
1 if yi
x,(x)
0
if +
In this case, 2/2 و (x)(2)) = + Ej = 2 - 22, 2.
If a. We then partition ST in to two parts: ST, and ST,, with
P(x Sr1} =a a²,
and
Pfx E ST.2] = a.
Then, We define (xx) as follows. When Sp or Sy.2
π(x) = π (x))
In this case, 1/2 Ef= (x)(x) (z(r = O.
When x E Sr.,
0 if - y;
xy(x)=
1 jf
In this case, 3/2 Σ 1 ()(x) p5(x) - Ejjn D3 = 2 - -2pp(z) S 2.
Therefore, for both of the two cases, we have
TCE(T) s 2/a a'1 =2|ACC(() -CC()T)))
B.3. Proof of Theorem 4.5
Proof. Denote ACC(T") = a" and ACC(Tt) = a. By the definition of accuracy,
F( arg max (x) A arg max ((x)) 2 la la-a"")) a").
For these
(z(x)) p)({)|
max (1(())- - > o.
1 j=1
j=
Let
min C 1c (x) arg (rr) max n (xx)) j=l....
max
13(()) - (z()
We can see that C' > 0 and
TCE(T) 2 Cla a"},
for all æ.
18

---

Restoring Calibration for Aligned Large Language Models: A Calibration-Aware Fine-Tuning Approach
B.4. Proof of Theorem 4.6
Proof. We first recap the definition of cw-ECE:
cw-FCE
RBp()) k /=1
Σ
P(y= (o)) (,)(
For any y E [0;
Ply = j1o,(x) = 9) (p)
By a triangle inequality, we have
P(y jlβg(x)) )3((x) s (o)))))))())) 1T(2)
Therefore, we obtain that
cw-ECR < TCE.
C. Additional Experimental Details
C.1. Comprehensive Descriptions for Models, Baselines, Datasets, and Metrics
Models In our study, we employ four widely-used open-source large language models to investigate the calibration issue and validate the effectiveness of our proposed method. They include
---uuu---TT -DD- LLaMA-3.1 -.1-A-3.--Tuuuee --lu--uu-u---- (Lambert et al., 2024) is a sfate-of-the-ute cessruction-followinn model developed by Allen Institute for AI. It is part of the Tulu 3 family, which is designed for diverse tasks beyond chat, such as MATH, GSM8K, and IFEval. The model is trained using supervised fine-tuning (SFT) and Direct Preference Optimization (DPO), achieving competitive performance on benchmarks like MMLU and TruthfulQA. is fully open-source, with data, code, and training recipes available for reproducibility.
Vicuna-7B-v1.55 Vicuna-7B- 7B-51.5 (Chiang et al., 2023) is chat assistant developed by LMSYS, fine-tuned from Llama 2 using approximately 125,000 user-shared conversations collected from ShareGPT.com. This auto-regressive language model employs the transformer architecture and is designed for research purposes in natural language processing, machine learning, and artificial intelligence. The model has been evaluated using standard benchmarks, human preferences, and LLM-as-a-jude methodologies.
Olmo 2-7B-DPO Olmo 2-7B-DP (OL et al., 2024) a fully open-source language model from Allen Institute for AI, designed for research and educational use. is trained on the Dolma dataset and fine-tuned using DPO for improved performance on tasks like text generation and instruction following. The model is part of the OLMo series, which emphasizes transparency by releasing weights, data, and training details. It achieves strong results benchmarks such
GSM8K and MATH.
Mistral 7B-DPO": Mistral 7B-DPO (Jiang al., 2023a) is high-performance language model fine-tuned using Direct Preference Optimization. It is designed for tasks like text generation, instruction following, and reasoning. The model is part of the Mistral family, known for its efficiency and strong performance on benchmarks such as HumanEval and GSMSK. is widely used in research and applications requiring robust natural language understanding.
Given the limitation of our computational resources (only four A100 (40G) GPUs). we utilize the Quantized Low Rank (QLoRA) technique (Dettmers al., 2024) optimize workflow. It is worth noting that all selected models aligned with human values according to either Reinforcement Learning with Human Feedback (RLHF) (Christiano et al., 2017) Direct Preference Optimization (DPO) (Rafailov et al., 2024b), providing robust framework for alignment and adaptability in our experimental evaluations.
19

---

Restoring Calibration for Aligned Large Language Models: A Calibration-Aware Fine-Tuning Approach
Baselines We validate the effectiveness of the proposed method by comparing performance among four kinds of models.
The model after alignment with human preference: This kind of model is typically aligned with human preference via RLHF or DPO.
The model calibrated by Temperature Scaling (TS) (Guo et al., 2017): TS adjusts the confidence scores of a pre- pre-trained model by introducing a single parameter called the "temperature" (denoted as T). This parameter scales the logits (outputs before the softmax function) to produce softer probability distributions. The temperature T is searched on a validation set to directly minimize the ECE and thus the model calibated by TS is usually well-calibrated.
The model fine-tuned by CFT (ours): This kind of model is calibrated by our method CFT where we only use the completion or response to calculate the loss. In this way the model learns patterns rather than knowledge. This approach does not improve generalization (hence, accuracy remains largely unchanged), but it mitigates the model's overconfidence by focusing on patterns that influence confidence. As a result, the ECE is significantly reduced.
The model fine-tuned by Regularized CFT (ours): This kind of model is calibrated by our method Regularized CFT (RCFT (Ours)) where we use both the prompt (questions and options) and the completion (response) to calculate the loss. this way, the model learns knowledge from the prompt, improving its generalization ability. This leads to higher accuracy on the test set but also keeps the model in an overconfident state. To address this, additional calibration loss is required to adjust the model's confidence.
Dataset To evaluate the efficacy of our proposed calibration method, we employ five datasets to conduct comprehensive experiments:
MMLU (Massive Multitask Language Understanding)): MMLU is a benchmark dataset designed to evaluate the knowledge and reasoning capabilities of language models across 57 subjects, ranging from STEM to humanities and social sciences. It includes questions at various difficulty levels, from elementary to advanced professional, and is particularly useful for assessing zero-shot and few-shot learning performance. The dataset is structured to test both world knowledge and problem-solving abilities, making it a comprehensive tool for identifying model blind spots. MedMCQA": MedMCQA designed for medical multiple-choice question (MCQs), which includes a comprehensive collection of advanced-level questions covering various medical fields such as Anesthesia, Anatomy, Biochemistry, and more. The dataset comprises approximately 194k MCQs, sourced from AIIMS and NEET PG entrance exams. We randomly select 5000 samples, with 3500 used as the training set for fine-tuning and 1500 as the testing set for inference
OpenBookQA 10; OpenBookQA is a dataset modeled after open-book exams, designed to assess the understanding and application of core scientific facts. It consists of 5,957 elementary-leve science questions, each linked to a small "book" of 1,326 core science facts. The dataset requires models to apply broad common knowledge beyond the provided facts, making challenging for retrieval-based and word co-occurrence algorithms. It includes 4.957 training, 500 development, and 500 test questions.
ARC- BRC-Challengee' ": The ARC-Challenge dataset is collection of 2,590 multiple-choice science questions designed to test advanced knowledge and reasoning skills. These questions are derived from science exams for grades 3 through 9 and are specifically curated to be challenging for both humans and Al systems, with each question having four answer choices. The questions require a deep understanding of scientific concepts, logical reasoning, and the ability to infer relationships between ideas. The ARC-Challenge is widely used as benchmark to evaluate the performance of AI models in complex question-answerinn answering tasks, pushing the boundaries of natural language understanding and reasoning capabilities.
"/ Evaluation of instruction models (e.g., ChatGPT) typically requires human interaction, which is time-consuming, expensive, and difficult to replicate. AlpacaEval is an LLM- based automatic evaluation framework that is fast, cost-effective, reproducible, and validated against 20K human annotations. It is particularly useful for model development. Although it improves upon prior automatic evaluation pipelines, A Alpa pa 1 still exhibits fundamental limitations, such as a preference for longer outputs. The framework provides the following components:
Leaderboard: A leaderboard reporting reporttin the performance of common models the Alpaca evaluation set.
'https: http://ggiingacee.o//aaaaaaaaeee
Pnttps://githin howp//////eeeeeoog///mmmmcccc
http:///ooon//////boooooggg /openbookqa
httts////gggtttt.oo////sssaaaaa/ a_ar
20

---

Restoring Calibration for Aligned Large Language Models: A Calibration-Aware Fine-Tuning Approach
Caution: Automatic evaluators (e.g.. GPT-4) may be biased toward models that produce longer outputs or were fine-tuned the same base model as the evaluator (e.g., GPT-4).
Automatic Evaluator: An evaluator with high agreement with human judgments (validated on 20K annotations). It measures performance by computing the fraction of times a strong LLM (e.g., GPT-4) prefers the outputs from the evaluated model over those from a reference model. The evaluator supports output caching and randomization by default.
Toolkit for Building Evaluators: simple interface for constructing advanced automatic evaluators with features such as caching, batching, multi-annotator support, and statistical analysis (c.g., quality, price, speed, statistical power, bias, and variance).
Human Evaluation Data: A dataset of 20K human preferences comparing outputs from given model and reference model on the A Alpa capcaFa am evaluation set. This includes 2.5K cross-annotations, where four human annotators rated the same 650 examples.
AlpacaEval Dataset: simplified version of the AlpacaFarm evaluation set, where "instructions" and "inputs" are merged into single field and reference outputs are extended length. See the documentation for further details.
Arena-Hard Arena -are--arrd-aaa Auto is an automatic evaluation tool for instruction-tuned LLMs. Among pop- ular open-ended LLM benchmarks, Arena-Hard-Auto demonstrates the highest correlation and separability with LMArena (Chatbot Arena); see the associated paper for details. If you are interested in estimating how well your model might perform on LMAreNa prior to deployment, we recommend using the latest evaluation set, Arema-Hard-v2. Peereee...
UltraFeedback -iinarized----dd---- Prrriieeeeerffereeeess. This dataset, hosted on Hugging Face by Argilla, is a resource designed to support research in preference modeling and reinforcement learning from human feedback (RLHF). This dataset provides collection of binary-labeled preferences derived from UltraFeedback data, where each entry represents human judgment indicating preference between two competing outputs. It is tailored for fine-tuning models to align with human-ll like decision- maaking and to enable evaluation of performance in feedback based ranking tasks. By offering structured preference annotations, it empowers researchers to develop and evaluate models that incorporate nuanced human feedback, facilitating advancements in Al alignment and personalization.
These datasets are instrumental in assessing the calibration performance as well as the model's original language ability.
Metric
Here we introduce three metrics used in our paper: The discrete conf-ECE and cw-ECE, and Win Rate. The discrete conf-BCE and cw-ECE are defined as follows.
Confidence Expected Calibration Error (conf-ECE) is defined
M Bm N
conf-ECE =
P(Y = 7|x E Bm)) - Elmooy = 7xx)|x E Bm]).
(19)
m==
where B,, is the j-th bin in terms of max confidence; [Bm] denotes the cardinality of the bin; P(Y = T|x E Bm) and EfTo(y /|x)|x E Bm] denote the average prediction of winning class probability and the actual probability that the winning class is the ground truth class.
Class- Wise Expected Calibration Error (cw-ECE) is defined as
M Bm N
1
CW-ECE
PPY = i|x E Bm.j)) Ey = =ixx) E Bmill
(20)
K
where Bm, is the m-th bin of the j-th class; Bmn.jl denotes the cardinality of the bin; P(Y j|x Bmg) and E[ = j|x)[x Bl,s)) denote the average prediction of bin j probability and the actual proportion of bin in the bin Bm.j-
We then introduce the detail of the Win Rate metric. Win Rate metric evaluates the model's "language ability" in two aspects.
21

---

Restoring Calibration for Aligned Large Language Models: A Calibration-Aware Fine-Tuning Approach
Confidence Callbration Curve
1.0
1.0
Lnderconfdarce
Ovenconfidence
FZ2
Overcontidence
0.8
0.8
Prob. Correct
0.6
0.8
of
0.4
0.00
0.2
0.4
0.6
0.8
1.0
0.00
0.2
1.0 0.4
0.8
Prob. Answer
Prob. Answer
(a) SFT2 only.
(b) ECE only.
Figure 4. (a) Using only LSFT, achieves high accuracy (~90%) but shows poor calibration. (b) Using only LECE drives the model toward random guessing, achieving near-zero ECE but with only 25% accuracy.
The first one is the ability to distinguish between preferred and non- preferred responses within given dataset. To this end, we utilize widely-used dataset UltraFeedback which provides pairs of responses: chosen_rense rresponse (preferred) and reject_response (non-pree con-preferred) for each instruction. We then calculate the sequence probabilities of these provided responses under the target model's distritution. The sequence probability of a response R = (W1, W2,--., Wr), where w; denotes the i-th token, is computed as:
response
T II
P(R) P(W1, W2...... wr)
P(ws W1, W2...... .... W;-1).
(21)
Wt W:-1) represents the probability of token w; given its preceding context, as determined by the model. For each instruction in the dataset, we calculate the sequence probabilities Petssen and Preject for the chosen_ response and reject__, respectively. A "win" is recorded if P'hosen Prejecti otherwise, it "loss."
Here, P(w; | Wi; W2, ..,
The second aspect is the quality of the response. We use the questions from two popular datasets AlpacaEval16 and Arena-Hard and let our model and a baseline model (usually GPT-4/GPT-40 or the DPO/RLHF model that is well aligned with human preference) make two separate responses. We then let third party large language model (such as GPT-4 or GPT-4o) judge which one is better. "win" of our model recorded if the judge thinks its response better than the baseline; otherwise,
"loss."
The Win Rate is then defined as the proportion of wins over the total number of comparisons:
Number of Wins
Win Rate Total Number of Comparisons
We anticipate that the model adjusted using approach will achieve competitive Win Rate in comparison the model prior to calibration. This will demonstrate that the good alignment performance brought by DPO/RLHF will not be diminished when we lift the calibration performance.
C.2. Additional Ablation Study
Figure 4 shows the ablation study in RCFT. In RCFT's objective, have two components. One is the SFT loss LSrt, and the other is the calibration loss LECF. We want to investigate the effect of each of them solely. Figure 4(a) shows the calibration plot of using LSFT, to optimize only. We can see that this loss function can significantly improve the accuracy but the calibration performance is destroyed. On the other hand, Figure 4(b) exhibits a very good calibration performance (the column is right on the perfect calibration diagonal line). However, its accuracy is very low (around 0.25), indicating the model is randomly guessing. RCFT combines these two loss functions together, resulting in good trade-off between accuracy and calibration.
http:/:///hgggggg//caeee hhcres////anginiree....o/attttss Feedhaaaeaaaataaeaacc "https: h/tp:s:///iii thub. com/tattsu--abb/appaaa
feedback-ebbeeee binari-i zedi red-preferences
"https:// thub. http:tt....m/mmmeeaaaeeeeeeeeeee
22

---

Restoring Calibration for Aligned Large Language Models: A Calibration-Aware Fine-Tuning Approach
Table 4 presents the effect of varying the calibration regularization weight À in our fine-tuning objective. When À the model corresponds standard SFT, achieving the highest accuracy but relatively poor calibration. As À increases, both ECE and class-weighted ECE generally improve, indicating better alignment between model confidence and correctness. However, this comes with noticeable drop in accuracy, particularly for higher values of X, such as 1.8 or in the "ECE only" setting. The results highlight a trade-off between predictive accuracy and calibration, and suggest that moderate values of (e.g., = 1) can strike a balance between the two.
Table 4. Ablation on calibration weight À in the loss function. We report the accuracy and calibration metrics (ECE and cw-BCE) for models fine-tuned with varying in the objective. A higher À places more emphasis on calibration.
Lambda Metric
0.4
1.8 ECE cw-ECE Acc ECE
ECE only cw-ECB Acc
ECE
cw-ECE Ace
ECE
cw-ECE Ace
cw-ECB Acc
ECE
Llama-3.1 10-8B/0..o
1- 000083 0.0808 0.8964 0.1535 0,1014 0.8409 00.097 0,0771 0.8341 00.0188 0.0106 0,4366|0.0002
0.0081 0.2475
Vicuna-7B 7B 0.12119
0.0774 0.8322 0.1620 0.0991 0.7315 0.0474 0.0459 0.6015 1.00150.102 0.1052 0.0799 0.38770.0133
0.0270 0.2290
Olmo2-7B
0.1003
0.0992 0.8846 0.1771 0.1008
0.8427 0.0989 0.0806 0.8510 0.0038
0.0113 0.4901 0.0030 0.0043 0.2765
Mistral-7B (0.0976 0.0785 0.9091 0.1316 0.0733
0.8085
0.0979
0.0877
0.8297 0.0366
0.0617
0.4217 0.0021
0.0108
0.2670
C.3. Additional Experimental Results
Figure and Figure show the complete calibration plots on Llama-3.1-8B-TT We can see that CFT and RCFT exhibit calibration improvement by providing good alignment with the perfect calibration diagonal line. In Figure 7, the columns not be able to show a good alignment with the diagonal line, showing various behaviors in different options. After merging them into one figure (shown in Figure 5), the biases of the different options cancel each other out, resulting in a good classwise calibration. Figure 6 and Figure 8 are the calibration plots on Olmo-7B, which show similar phenomenon as Figure 5 and Figure 7, validating the effectiveness of the proposed methods.
C.4. Discussion of Bin Size
Recent studies have shown that the estimation of ECE suffers from significant estimation bias (e.g., Futami & Fujisawa (2024) for binary classification; Fujisawa & Futami (2024) for multiclass classification). According to these works, binning- based ECE exhibits a slow convergence rate of O(n -1/3) and incurs substantial bias. According to the referenced work, the optimal bin size for the multiclass setting scales as O(n -133.. However, to apply this practically, one needs the exact constant rather than just the asymptotic rate. Upon further examination, we found that the optimal bin size contains a Lipschitz constant of the model, in rate of
1/3)
((L3//n-1/3)
In practice, the Lipschitz constant of deep neural networks is known to be very large; see, for example, our previous work (Xiao et al., 2023; 2024b;c). Estimating the Lipschitz constant of Transformer architectures in particular remains challenging.
In our experiments, the sample size is 3,000, which implies an optimal bin size on the order of (14.4). As suggested by the anonymous reviewer, the value n - 1/3 plays an important role estimating the ECE, and the actual ECE value can vary significantly depending on whether the number of bins is 10 or 14.
On this point, we hold different view: we believe that the Lipschitz constant dominates the optimal convergence rate, and that the theoretical order is primarily of academic interest rather than of practical significance.
Nonetheless, we can still make use of the rate as in the theoretical papers. We adopt a bin size of 14 and evaluate all four methods across different architectures, ECE variants, and both in-domain and out-of-domain settings. The results are presented in Table 5.
We observe that CFT and RCFT consistently improve the ECE of DPO models. The comparison between approach and TS remains consistent with our original findings: in 5 out of 8 conf-BCE comparisons, CFT outperforms TS.
Use of Generative AI
The authors used generative LLMs only for proofreading, checking grammar, and correcting typos to improve the readability of the paper.
23

---

Restoring Calibration for Aligned Large Language Models: A Calibration-Aware Fine- Tuning Approach
Callbration Curve
Curve
1.0
1.0
1.0
1,0
irdercantidence Overcorfidence
Underconteerce
Undercontioeece
Undercortidecee
F77
Overcorfidence
Overconfizlrce
Overcenfieence
0.8-
0.8
6.8-
0.8
Proh. of Correct
Prob of Correct
Prob of Correct
0.6
6.6
0.6
0.6
0.4
0.0
0.2
0.4
0.6
0.5
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.5
0.8
1.0
0.0
0.2
0.4
0.5
0.8
1.0
Prob. of B.
Prob. Af C,
Prob. of C.
Prob. of C,
(a) DPO (0.0953)
(b) TS (0.0336)
(c) CFT (0.0582)
(d) RCFT (0.0771)
Confidence Calibration Curve
Confidence Calibration Curve
Confidence Calibration Curve
Confidencee Calibration Curve
Undercanniieerce
Underconfisence
Undercarl
Overcoofiidenc
Overcor fidence
Overccnffilerce
Opercenfieence
Correct
Prob of Correct
Prob. of Correct
0.6-
0.6
0.6
Prob
0.4
0.4
0.4
0.4
0.2
0.2
0.2
0.2
0.0
0.0
0.9
Prob. Answer
Prob.
(e) DPO (0.1953)
(1) TS (0.1126)
(g) (0.0239)
(h) RCFT (0.0897)
Figure 5. Calibration Plots (a, e) DPO, (b, f) Temperature Scaling (TS), (c, g) our CFT, (d, h) our RCFT on 1.ama-3.1-8B --B--uulu. (a-d) are the classwise calibration curve and (c-h) the confidence calibration curve. Each panel plots the model's predicted probabilities (i.e., confidence) on the æ-axis against the observed accuracy (fraction correct) on the y-axis, binned into ten groups. The diagonal line in each panel represents perfect calibration. The depth of the color indicates the sample density in that column. DPO has the worst calibration performance. Other three methods improve the calibration performance where our CFT has the lowest con-ECE (shown in the parenthesis). The figures of conf-ECT, (e-h) omit the first bins hecause the model selects probability which is always larger than 0.25 in the four options prediction task (so no samples exist below that threshold).
with the largest predicted
Table Performance comparison among DPO/RLHF, Temperature Scaling, CFT, and RCFT across four models (Llama3.18B--UV Vicuna-7B, Olmo2-7B, and Mistral-7B) in in-domain and out-domain scenarios. Best results in each metric block are bolded. "I" means the smaller the better.
conf-ECEE
ew-ECE +
Model
Method
In-Domain
Out-Domain
In-Domain
Out-Domain
Llama3.1- 8B-Tulu
DPO
0. 0811
0.1188
0.0988
0.0657
Temp Scale.
0.1158
0,0559
0.0349
0.0256
CFT(Ours)
0.0441
0.0520
0.0418
0.0344
RCFT((urr)
0.1011
0.0801
0.0783
0.0525
Vicuna-7B
RLHF
0.1418
0.0888
0.0664
0.0993
Temp Scale.
0.0377
0.0297
0.0220
0.0523
CFT(Ours)
0.0216
0.0308
0.0295
0.0516
RCFF((urs)
0.0508
0.0677
0.0397
0.0552
Olmo2-7B
DPO
0.1370
0.0914
0,0773
0,0630
Temp Scale.
0.0490
0.0272
0.0329
0,0252
CFT(Ours)
0.0587
0.0573
0.0376
0.0356
RCFT(Ourr)
0.0730
0.0663
0.0365
0.0512
Mistral-7B
DPO
0.1979
0.1346
0.1010
0.1187
Temp Scale.
0.0771
0.1093
0.0380
0.0582
CFT(Ours)
0.0602
0.0511
0.0207
0.0601
(Cur()uur))
0.0817
0.0658
0.0457
0.0506
24

---

Restoring Calibration for Aligned Large Language Models: A Calibration-Aware Tine-Tuning Approach
Classwise Calibration Curve
Casswise Calibration Curve
Classwise Calibration Curve
Classwise Calibration Curve
underconfidece
Underconfioerc: Overcer fidence
Underconfiience Overconfielece
Underconf
Dvercconiieence
Operconfiéence
Correct
Correct
Prob. Correct
Prob of Correct
0.6
0.4-
0.4
0.4
0.4
0.2
0.2
0.2
0.2
0.0
0.0
0.2
0.6
0.0
0.0
0.0
0.0
1.0
Frob.
Prob.
(a) DPO (0.0873)
(b) (0.0589)
CFT (0.0804)
(d) RCFT (0.0806)
Confidence Calibration Curve
Confidence Calibration Curve
Confidence Calibration Curve
Confidence Calibration Curve
1.0
10
1.0
1.0
urderconttdence
Undercontis ence
Undercenffisnce
Undercort dence
Dverconfidence
Overcorfidence
Overcentieerce
Overcentisence
0.8
0.8
0.8
0.8
Prob. Correct
Prob. of Correct
Prob. of Correct
0.8-
Prob. of
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Prob. of Answer
Prob. Answer
Prob. of Answer
Prob. of Answer
(e) DPO (5.1555)
(f) TS (0.0544)
(g) CFT (0.0544)
(h) RCFT (0.0989)
Figure 6. Calibration Plots of e) DPO. (b. 1) Temperature Scaling, (c, g) CFT, and (d. h) RCFT Olmo2-7B. (a-d) the classwise calibration curve and (e-h) are the confidence calibration curve. Each panel plots the model's predicted probabilities (i.e., confidence) the r-axis against the observed accuracy (fraction correct) the y-axis, binned into groups The diagonal line in each panel represents perfect calibration. The depth of the color indicates the sample density in that column. DPO has the worst calibration performance. Other three methods improve the calibration performance where CFT has the lowest con-ECE (shown in the parenthesis). The figures of Donf-ECEE (c-h) omit the first two bins because the model selects answer with the largest predicted probability which always larger than 0.25 in the four options prediction task (so samples exist below that threshold).
25

---

Restoring Calibration for Aligned Large Language Models: A Calibration-Aware Fine- Fine-Tuning Approach
Calibration Curve
Calibration Curve
Calibraton Curve
Callbration Curve
1.C
10
1,0
10
ürdercontidence
ordercontoerce
Undercontiieece
Undercort dence
Duerconfidence
Overcoefidence
Overcenlilerre
Overconticince
0.8
0.8
0.8
0.5
Froh. Correct
Prob. of Correct
Prob. of Correct
0.6
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
100
0.0
022
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Prob. of Option A
Prob. of Optiion A
Prob. of Option
Prob. of Option
(a) DPO
(b) TS
(c) CFT
(d) RCFT
Colibration Curve
Callaration
Callbration
Callbration
1.0
Ordercarfidence
Underconfiéerrc
Underconfisence
Undercanf
7
Overcorfidence
Overcerfidence
Overcontioarce
Opercenfieence
0.8
Prob,, Connec
Conrec
Prob of Correct
Prob. of Correct
0.6
0.6
0.66
0.6
0.4
0.4
0.4
0.4
0.2
0.2
0.2
0.2
0.8
1.0
Option
Option
Option
Option
(e) DPO
(f) TS
(g) CFT
(h) RCFT
Calibration Curve
Calibration Curve
Callbration Curve
Calibration Curve
100
dedeeccofidence
Undercontiserce
Underconfidence
Undercant
Overconfidence
Cvercorfideence
Overconfidence
Overconticence
Commet
Correct
Prob. Conrect
Prob. Correct
0.4
0.4-
0.6
0.2
0.2
0.2
0.2
0.0
0.0
022
0.0
0.0
0.0
0.0
Prob. Option
Prob. of Option
Prob. Option
Prob. Option
(i) DPO
(i) TS
CFT
RCFT
Calibration Curve
Callbration Curve
Calibration Curve
Calibration Curve
1.0
10
1.0
1.0
ürdercontidence
DYA Undercontssoree
Undercontisence
Undercorf dence
Overcortidence
Overcortidence
Overcentiderce
Osecenfiieene
0.8
0.8
0.8
0.8
Prob. of Correct
Prob. of Correct
0.6
Prob.
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4 0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.5
0.8
1.0
Prob. Option D
Prob. of Option D
Prob. of Option D
Prob. of Option
(m) DPO
(n) TS
(o) CFT
(p) RCFT
Figure 7. Calibration Plots of (a, m) DPO, (b, f, n) Temperature Scaling (TS), (c, g. k, CFT, (d, I. RCFT on Llama-3.1 -uluu Each panel plots the model's predicted probabilities (i.e., confidence) on the r-axis against the observed accuracy (fraction correct) the y-axis, binned into ten groups. The diagonal line each panel represents perfect calibration. The depth of the color indicates the sample density in that column.
26

---

Restoring Calibration for Aligned Large Language Models: A Calibration-Aware Fine- Fine-Tuning Approach
Calibration Curve
Calibration Curve
Calibraton Curve
Callbration Curve
1.C
10
1,0
10
ürdercontidence
Undercontiierce
Undercontiieece
Undercort dence
Duerconfidence
Overccefidence
Overcenfilerre
Dsercontizence
0.8
0.8
0.8
0.5
Froh. Correct
Prob. of Correct
Prob. of Correct
0.6
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
100
0.0
022
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Prob. of Option A
Prob. of Optiion A
Prob. of Option
Prob. of Option
(a) DPO
(b) TS
(c) CFT
(d) RCFT
Colibration Curve
Callaration
Callbration
Callbration
1.0
1.0
Ordercarfidence
Underconfidercc
Underconfisence
Undercanl
7
Overcorfidence
Overcerfidence
Overcontioance
Opercenfieence
0.8
0.8
Prob,, Connec
Coneect
Prob of Correct
Prob. of Correct
0.6
0.6
0.66
0.6
0.4-
0.4
0.4
0.4
0.2
0.2
0.2
0.2.
0.4
0.5
1.0
Option
Option
Option
(e) DPO
(f) TS
(g) CFT
(h) RCFT
Calibration Curve
Calibration Curve
Callbration Curve
Calibration Curve
1.0
dederconfidene
Undercontiserce
Underconfidence
Undercont
Overconfidence
Cvercorfideence
Overconfidence
Overconticence
Commet
Correct
Prob. of Correct
Prob. Correct
0.4
0.4-
0.6
0.2
0.2
0.2
0.2
0.0
0.0
0.0
0.0
0.0
0.0
1.0
Prob. Option
Prob. of Option
Prob. Option
Prob. Option
(i) DPO
(i) TS
CFT
RCFT
Calibration Curve
Callbration Curve
Calibration Curve
Calibration Curve
1.0
10
1.0
1.0
ürdercontidence
Undercontssorce
Undercontisence
Undercorf dence
Overcorfidence
Overcortidence
Overcentiderce
Osecenfiieene
0.8
0.8
0.8
0.8
Prob. of Correct
Prob. of Correct
0.6
Prob.
Prob.
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6 0.8
1.0
0.2
0.4
0.5
0.8
1.0
Prob. Option D
Prob. of Option D
Prob. of Option D
Prob. of Option
(m) DPO
(n) TS
(o) CFT
(p) RCFT
Figure 8. Calibration Plots of (a, c, m) DPO, (b, f. Temperature Scaling, (c, o) our CFT, and (d, h, RCFT on Olmo2-7B. Each panel plots the model's predicted probabilities (i.e., confidence) on the r-axis against the observed accuracy (fraction correct) on the y-axis, binned into groups. The diagonal line in each panel represents perfect calibration. The depth of the color indicates the sample density in that column.
27


Reviews:
[{"review_id": "GNFpUIG9MS", "reviewer": "Reviewer_QE3c", "summary": "The paper tries to answer a well-known question: why an aligned model is not well-calibrated and how to fix it? Authors start with a probabilistic generative model and define TCE, and then derive an upper and lower bound of TCE. Then, depends on the accuracy of the current model, one can either get calibration without sacrificing accuracy or not. The authors state that, most cases are calibratable. \n\nTo recover the calibration of models, authors propose an EM algorithm to compute calibration loss. Together with SFT loss, the CFT get better calibration loss and accuracy.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer QE3c for the insightful comments and questions. Below we provide our responses to the questions.\n\n>Q1. Can we draw any senses about when the model falls into each case from a theoretical perspect?\n\nA. This is a great question. Determining which regime a model falls into ultimately reduces to understanding whether its accuracy exceeds a certain threshold. This, in turn, depends on two key factors: 1. What is the accuracy threshold for a given neural network architecture? 2. Given such an architecture, which training algorithms lead the model to fall into each regime?\nAnswering these questions requires a deeper theoretical analysis of the properties of transformers, which is currently an open and challenging direction. While we do not yet have a definitive theoretical answer, our experimental results provide some insights. We will add a discussion of this point in the final section and consider it an important direction for future work.\n\n>Q2. The theory contribution is unclear. If my understanding is correct, the theory mainly states that there are two possible scenarios and does not directly relate to other parts of the paper, such as algorithms. The insight drawn from the theory is vague.\n\nA. Our theoretical results are directly connected to the algorithmic design and other components of the paper. Let us clarify. In Theorem 4.6, we prove that ECE ≤ TCE. In other words, obtaining a well-calibrated model reduces to finding one that is close to the **target probabilistic model**, i.e., the solution to the following constrained optimization problem:  \n\n$$\n\\max \\text{ACC}(\\pi) \\quad \\text{s.t.} \\quad \\text{ECE}(\\pi) = 0.\n$$\n\nIn Section 5, we discuss how to approximate ACC and ECE in the context of LLMs—specifically in Sections 5.1 and 5.2, respectively. Having defined approximations for the ACC and ECE losses, we introduce an EM algorithm to learn the underlying probabilistic model. This choice is motivated by our theoretical formulation of calibration as a probabilistic inference problem, for which EM is a natural and widely used solution.\n\n>Q3. There is only one baseline (temp scaling); it would be better to compare  with other baselines such as label smoothing, [1], etc.\n\nA. Thank you for the question. First, we would like to emphasize that temperature scaling is a strong baseline for reducing ECE. Prior work [3,4] has shown that it is generally more effective than label smoothing and other techniques for improving calibration.\nNonetheless, we conducted additional experiments and found that, similar to image classification tasks, label smoothing remains less effective than temperature scaling on LLMs as well. We will add the experiments to our revised paper.\n| Model            | conf-ECE   |            | cw-ECE    |            | Accuracy  |            |\n|-|-|------------|-|-|-|-|\n|                  | In-Domain | Out-Domain | In | Out | In | Out |\n| Llama3.1-8B-Tulu | 0.1898     | 0.1009     | 0.0692    | 0.0639     | 0.6372    | 0.7116     |\n| Vicuna-7B        | 0.1221     | 0.0823     | 0.0517    | 0.0544     | 0.4517    | 0.5767     |\n| Olmo2-7B         | 0.1010     | 0.0499     | 0.0791    | 0.1298     | 0.6808    | 0.6431     |\n| Mistral-7B       | 0.1874     | 0.1121     | 0.0900    | 0.0990     | 0.6479    | 0.6997     |\n\n\n\nRegarding the contextual calibration method introduced in [1], we have carefully reviewed the paper. Although the term \"calibration\" appears in its name, the method is primarily designed to improve task performance across various NLP tasks, rather than to reduce ECE specifically. As such, their objective differs from ours, and the approaches are not directly comparable.\n\n[3] When Does Label Smoothing Help?\nRafael Müller, Simon Kornblith, Geoffrey Hinton\n\n[4] On Calibration of Modern Neural Networks\nChuan Guo, Geoff Pleiss, Yu Sun, Kilian Q. Weinberger\n\n>Q4. A better comparison of the Win rate is between CFT and DPO-generated responses since response qualities may decrease but still outperform the one being compared in the current setting.\n\nA: Thank you for the suggestion. We agree that comparing the win rate between CFT and DPO-generated responses provides more information. We have conducted these additional experiments on AlpacaEval dataset, and the results are now included in our revised paper and provided below for reference.\n|Win Rate | CFT| DPO | RCFT| DPO|\n|-|-|-|-|-|\n|  Llama-3.1-8B-Tulu | 51.68 | 48.32    | 46.83 | 53.16    |\n| Vicuna-7B | 46.46 | 53.54    | 50.43 | 49.57    |\n| Olmo2-7B | 62.48 | 37.52    | 46.12 | 53.88    |\n| Mistral-7B  | 46.96 | 53.04    | 49.81 | 50.19    |\n\nExperiments on Alpaca-Eval and Arena-hard are provided in the end of our response to Reviewer USq4.\n\n>Q5. Some relevant methods are not discussed: [1,2].\n\nA. We have included both [1] and [2] in the revised version of the paper.", "claims_and_evidence": "* Upper and lower bound of TCE. Suppored by Thm 4.4-4.6\n* Calibratable and Uncalibratable cases based on Accuracy. Supported by Thm 4.4-4.6.\n* EM algorithms to recover calibration. Supported by experiment.", "methods_and_evaluation_criteria": "Methods and evaluation makes sense.", "theoretical_claims": "I did not carefully check the proof Thm4.4 - 4.6, but they seem correct.\n\n### Question & Weakness\n* One theory claims that both calibratable and un-calibratable cases exist. Can we draw any senses about when the model falls into each case from a theoretical perspect?\n* The theory contribution is unclear. If my understanding is correct, the theory mainly states that there are two possible scenarios and does not directly relate to other parts of the paper, such as algorithms. The insight drawn from the theory is vague.", "experimental_designs_or_analyses": "The experiment is overall sound.\n\n### Question and Weakness\n* There is only one baseline (temp scaling); it would be better to compare with other baselines such as label smoothing, [1], etc.\n* A better comparison of the Win rate is between CFT and DPO-generated responses since response qualities may decrease but still outperform the one being compared in the current setting. \n\n[1] https://arxiv.org/abs/2102.09690", "relation_to_broader_scientific_literature": "The paper aligns a line of calibration work such as temperature scaling, Say-Self, etc. This work gives a theory analysis of calibration and a new EM calibration algorithm, which differs from prior works.", "essential_references_not_discussed": "Some relevant methods are not  discussed:\n[1] https://arxiv.org/abs/2102.09690\n[2] https://arxiv.org/pdf/2405.20974", "other_strengths_and_weaknesses": "Other strengths:\n* The paper is generally well-written.\n\nOther weakness:\n* See above", "other_comments_or_suggestions": "## update after rebuttal\n\nSince the authors addressed my concerns, I raised my score to 3.", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SDCYxSnd8G", "reviewer": "Reviewer_Ak1F", "summary": "This paper addresses the calibration issue in aligned large language models (LLMs) and proposes a calibration-aware fine-tuning approach to restore proper uncertainty quantification in these models. The motivation stems from the observation that alignment techniques can distort model confidence, leading to miscalibrated probabilities in downstream tasks.\nThe proposed method introduces a fine-tuning procedure that explicitly optimizes for calibration metrics while preserving the alignment properties of the LLM. The approach is designed to be agnostic to the underlying alignment strategy, making it adaptable to different LLM architectures and training paradigms. The authors provide theoretical insights into how alignment affects calibration and demonstrate the effectiveness of their method through extensive empirical evaluations on multiple benchmarks.\n\n## update after rebuttal\n\nFirst, I would like to apologize for not being able to respond directly to the authors during the discussion phase, due to limitations of the review system.\nInstead, I am using this update to clearly state my current position.\n\nThe authors have provided a strong and effective rebuttal that addresses many of my original concerns.\nIn particular, I sincerely appreciate their effort in conducting additional experiments based on theoretically justified bin sizes—especially considering how challenging such LLM experiments must be.\n\nThat said, I still have two remaining concerns:\n- **Presentation clarity:** I believe the paper does not sufficiently distinguish between the ECE used for evaluation and the ECE used in the objective function. Given that multiple definitions of “ECE” appear throughout the paper, I found it somewhat confusing—even as someone who works regularly with calibration metrics. Improving this distinction would greatly help with readability.\n\n- **Distinction between fine-tuning and standard classification tasks:** While I agree with the authors’ explanation regarding how labels are handled differently in the objective, I still find it unclear why methods from classification settings would not be applicable here.After all, the goal—minimizing ECE—is common to both this work and prior approaches in classification calibration. Is the difference simply in how probabilities are computed with respect to the labels? (It is possible that I am still misunderstanding this point.)I believe it is crucial to clearly articulate the distinction between the two approaches.If that proves difficult, an alternative could be to empirically compare the proposed method with existing calibration regularization techniques developed for classification—such as smooth CE or other differentiable calibration metrics—applied in this context. This would be a simple yet effective way to differentiate your method.\n\nI would like to raise these points for discussion among the reviewer panel. For now, I will keep my original score, but depending on the outcome of the discussion, I may consider increasing it.\n\nLastly, I would like to emphasize that I find this to be a very interesting and promising paper.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer Ak1F for the insightful comments and questions. Below we provide our responses to the questions.\n\n>Q1. Why fine-tuning with a calibration objective superior to post-hoc calibration methods e.g. TS?\n\nA. While temperature scaling (TS) is a strong baseline, our method's superiority stems from addressing fundamental limitations of post-hoc calibration in LLMs:\n\n1. Our approach explicitly minimizes the discrepancy between accuracy and confidence—the definition of ECE—rather than relying on a single scaling parameter. This direct optimization improves calibration while maintaining or enhancing performance. This aligns with previous work on training classifiers with calibration objectives, such as the AvUC loss in [Krishnan et al](https://arxiv.org/pdf/2012.07923) and the PAC-Bayes-based objective in [Fujisawa & Futami, 2024](https://arxiv.org/pdf/2406.06227), which demonstrated the superiority of optimization-based strategies over post-hoc methods.\n\n2. Our approach generalizes better to unseen data and distribution shifts compared to TS, which risks overfitting to specific validation sets. Table 2 clearly demonstrates this advantage in out-domain scenarios—for example, with Olmo2-7B, our CFT method reduces out-domain cw-ECE to 0.0637 (vs. TS's 0.1196) while improving accuracy to 0.7085 (vs. DPO's 0.6635). Distribution shifts are common in language tasks, and TS is insufficient to handle such variations.\n\nWe will include these points in our revised manuscript.\n\n>Q2. Relationship between this paper and calibration-aware training in classification models?\n\nA. The referenced works propose various calibration-aware training objectives for classification models, such as MMCE, AvUC, S-AvUC, and SB-ECE. Some of these are designed as accuracy–calibration trade-off objectives.\n\nOur approach shares this general principle, as it can also be viewed as an accuracy–calibration trade-off method. The key difference is that our accuracy and calibration objectives are specifically designed for generative models and LLMs, rather than classification models.\n\nFor the accuracy objective, we use the SFT loss on next-token prediction. This loss not only promotes accuracy but also supports instruction following, knowledge grounding, and coherent text generation—all essential to LLMs.\n\nFor the ECE objective, we formulate calibration as a generative modeling problem and apply an EM algorithm to optimize it accordingly.\n\nIn summary, our method is tailored for LLMs, which represents a significant departure in both design and application.\n\n>Q3. Evaluation Metrics and ECE Bias and optimal bin size.\n\nA. Thank you for the helpful suggestion. The two referenced papers on ECE bias and optimal bin size are indeed insightful.\n\nIn our work, we use the UWB strategy, and we will make this choice explicit in the revised manuscript.\n\nAccording to the referenced work, the optimal bin size for the multiclass setting scales as $O(n^{1/3})$. However, to apply this practically, one needs the exact constant rather than just the asymptotic rate. Upon further examination, we found that the optimal bin size contains a Lipshitz constant of the model, in a rate of $(1 + L)^{2/3} n^{1/3}.$\n\nIn practice, estimating the Lipschitz constant for transformers is challenging, as it is known to be potentially very large and difficult to compute reliably. Nonetheless, we can still make use of the theoretical rate $n^{1/3}$ as in the two mentioned papers. In our experiments, the sample size is 3,000, which implies an optimal bin size on the order of O(14.4). The bin size of 10 used in our paper is reasonably close to this rate, suggesting that our choice is consistent with the theoretical guidance.\n\n>Q4. In which cases the proposed methods perform well and fail?\n\nA. Our approach is specifically designed and performs well for LLMs which are originally well-calibrated but become poorly calibrated after preference alignment. In other scenarios—such as (1) traditional classification tasks, or (2) cases where the LLM is not well-calibrated to begin with—our method may not be as effective.\n\n>Q5. Certain types of LLM architectures, alignment techniques ... where this approach is less effective?\n\nA. In our experiments, we evaluate the method across different architectures, alignment techniques, and datasets to demonstrate its generality. However, due to computational constraints, we did not experiment with large-scale models such as 70B. While the effectiveness may vary at that scale, we believe our method is conceptually scalable and can be applied to larger models with appropriate resources.\n\n>Q6. Computational costs? Compare to other techniques? ECE term differentiable?\n\nA. On two A100 40GB GPUs, our approach takes approximately 1.5 hours to run for 5 epochs. The training time is comparable to label smoothing, but significantly slower than temperature scaling, as the latter does not require fine-tuning model weights. In addition, Our ECE term is differentiable.", "claims_and_evidence": "### Claim 1 (Explaining why alignment affects calibration):\n\nThe paper provides an intuitive argument that preference alignment alters model confidence by enforcing human-desired responses, potentially distorting probability estimates. While it is not entirely clear which part of the  results explicitly show the critical answer for \"why preference alignment affects calibration\", the fine-tuning approach naturally introduces a trade-off between accuracy and calibration error, which is well-motivated. The design of the objective function balances these two aspects in a reasonable manner, making the proposed method a natural approach to addressing miscalibration in aligned LLMs.\n\n\n### Claim 2 (Proposed method mitigates calibration issues):\nWhile the paper focuses on the LLM fine-tuning, the underlying problem is fundamentally equivalent to the long-standing question in the machine learning community: how to balance classification accuracy and calibration in predictive models. \nAlthough the specific task setting is different, the challenge addressed here is closely related to classical studies in calibration-aware learning.\n\nThe proposed fine-tuning approach somewhat reduces calibration error in experiments. However, the justification for why this method is the best solution is weak.\n- Why is fine-tuning with a calibration objective superior to alternative methods such as temperature scaling?\n- Good predictive accuracy and calibration simultaneously have been widely studied in general machine learning classification problems (please see the following references). Beyond differences in objectives and tasks, what is the fundamental distinction between these approaches and the proposed method? Why is the proposed method preferable to extending those studies? Could some of the methods proposed in the following papers be incorporated into the fine-tune objective of the LLM without too much computational complexity?\n    - [Kumar et al., ICML2018](https://proceedings.mlr.press/v80/kumar18a/kumar18a.pdf#:~:text=Trainable%20Calibration%20Measures%20For%20Neural,so%20compro%02mise%20the%20many%20legitimately)\n    - [Krishnan et al, NeurIPS2020](https://arxiv.org/pdf/2012.07923)\n    - [Karandikar et al., NeurIPS2021](https://arxiv.org/pdf/2108.00106)\n    - [Popordanoska et al., NeurIPS2022](https://proceedings.neurips.cc/paper_files/paper/2022/file/33d6e648ee4fb24acec3a4bbcd4f001e-Paper-Conference.pdf#:~:text=We%20propose%20a%20tractable%2C%20differentiable%2C,ECEKDE%20scales%20well)\n\n\n### Claim 3 (Empirical results demonstrate effectiveness without degrading model performance):\n\nIn p.6, the Evaluation Metric section defines ECE as a nonparametric estimator based on binning. Recent studies have shown that this estimator suffers from significant estimation bias (e.g., [Futami & Fujisawa, NeurIPS2024](https://proceedings.neurips.cc/paper_files/paper/2024/file/9961e42624a6c083279303767c73269d-Paper-Conference.pdf), for binary classification; [Fujisawa & Futami, 2024](https://arxiv.org/pdf/2406.06227), for multiclass classification).\n\nAccording to these works, binning-based ECE has a slow convergence rate of $O(1/n^{1/3})$ and introduces a significant bias. The optimal number of bins to minimize this bias in an upper-bound sense is $O(n^{1/3})$, but this paper fixes the number of bins to $10$, which likely results in substantial estimation bias.\n\nGiven this, the numerical results presented in the paper should be interpreted with caution, as the validity of the estimated calibration error is questionable.", "methods_and_evaluation_criteria": "Many of the concerns regarding the methods and evaluation criteria have already been discussed in the above section. To briefly summarize:\n\n- While the paper focuses on LLM fine-tuning, the underlying problem is fundamentally equivalent to the long-standing question in the machine learning community: how to balance classification accuracy and calibration in predictive models. \nAlthough the specific task setting is different, the challenge addressed here is closely related to classical studies in calibration-aware learning. The justification for why this method is the best solution remains weak:\n    - Why is fine-tuning with a calibration objective superior to alternative methods such as temperature scaling?\n    - Could existing methods from classification calibration research (e.g., those by [Kumar et al., ICML2018](https://proceedings.mlr.press/v80/kumar18a/kumar18a.pdf#:~:text=Trainable%20Calibration%20Measures%20For%20Neural,so%20compro%02mise%20the%20many%20legitimately), [Krishnan et al, NeurIPS2020](https://arxiv.org/pdf/2012.07923), [Karandikar et al., NeurIPS2021](https://arxiv.org/pdf/2108.00106), [Popordanoska et al., NeurIPS2022](https://proceedings.neurips.cc/paper_files/paper/2022/file/33d6e648ee4fb24acec3a4bbcd4f001e-Paper-Conference.pdf#:~:text=We%20propose%20a%20tractable%2C%20differentiable%2C,ECEKDE%20scales%20well) be incorporated into the fine-tuning objective without excessive computational complexity?\n- The empirical evaluation has methodological concerns, particularly regarding the estimation bias in ECE due to fixed binning (see [Futami & Fujisawa, NeurIPS2024](https://proceedings.neurips.cc/paper_files/paper/2024/file/9961e42624a6c083279303767c73269d-Paper-Conference.pdf), [Fujisawa & Futami, 2024](https://arxiv.org/pdf/2406.06227)). The choice of 10 bins likely introduces substantial estimation bias, affecting the validity of the reported calibration improvements.", "theoretical_claims": "I have reviewed the proofs and found no obvious errors; the mathematical derivations appear to be correct.\n\nAssuming that the proofs are entirely correct, the claims derived from the theorems are reasonable in themselves. However, it is not clear that these results provide a direct answer to the question of \"why preference alignment affects calibration.\" Instead, the theoretical findings seem to demonstrate the trade-off between accuracy and calibration, a well-known issue that has been widely discussed in the general machine learning community.\n\nFrom a scientific perspective, a more appropriate claim may be that the paper formalizes this accuracy-calibration trade-off within the context of fine-tuning LLMs, rather than providing a direct theoretical justification for the miscalibration induced by preference alignment.", "experimental_designs_or_analyses": "As I mentioned above, the evaluation relies on ECE with binning, which introduces significant estimation bias.\nAs shown in [Futami & Fujisawa, NeurIPS2024](https://proceedings.neurips.cc/paper_files/paper/2024/file/9961e42624a6c083279303767c73269d-Paper-Conference.pdf) and [Fujisawa & Futami, 2024](https://arxiv.org/pdf/2406.06227), binning-based ECE has a slow convergence rate and can significantly over- or under-estimate calibration error.\nThe choice of $10$ bins likely introduces substantial bias, affecting the reliability of reported calibration improvements.\nAlternative or bias-corrected calibration metrics should be considered to ensure more robust evaluation.", "relation_to_broader_scientific_literature": "This work is closely related to prior research in calibration-aware learning in classification models, where calibration metrics are incorporated as regularization terms in the objective function to jointly optimize predictive accuracy and calibration performance. Although this paper focuses on fine-tuning LLMs, the core problem it addresses is fundamentally equivalent to the long-standing challenge in classification models:　balancing accuracy and calibration in predictive learning. \n\nSeveral prior studies have explored this challenge:\n\n- [Kumar et al., ICML2018](https://proceedings.mlr.press/v80/kumar18a/kumar18a.pdf#:~:text=Trainable%20Calibration%20Measures%20For%20Neural,so%20compro%02mise%20the%20many%20legitimately): One of the earliest works to directly integrate Expected Calibration Error (ECE) optimization into model training. They propose the Maximum Mean Calibration Error (MMCE), a new metric based on RKHS kernels, which is added as a regularization term to the cross-entropy loss.\n\n- [Krishnan et al, NeurIPS2020](https://arxiv.org/pdf/2012.07923): Introduce a differentiable loss that explicitly penalizes the discrepancy between model confidence and actual correctness, enabling direct optimization of calibration error.\n\n- [Karandikar et al., NeurIPS2021](https://arxiv.org/pdf/2108.00106): Develop a differentiable calibration loss by continuous relaxation of histogram binning for ECE computation. Their method, when incorporated into training, significantly reduces ECE for individual models.\n\n- [Popordanoska et al., NeurIPS2022](https://proceedings.neurips.cc/paper_files/paper/2022/file/33d6e648ee4fb24acec3a4bbcd4f001e-Paper-Conference.pdf#:~:text=We%20propose%20a%20tractable%2C%20differentiable%2C,ECEKDE%20scales%20well): Propose a Kernel Density Estimation-based ECE estimator, making Canonical Calibration Error (an extension of ECE) differentiable. This allows it to be integrated into the training objective, achieving a favorable trade-off between accuracy and calibration performance.\n\nBeyond calibration-aware training methods, the validity of ECE-based uncertainty estimation itself has been questioned in recent studies. \nAs highlighted by [Futami & Fujisawa, NeurIPS2024](https://proceedings.neurips.cc/paper_files/paper/2024/file/9961e42624a6c083279303767c73269d-Paper-Conference.pdf) and [Fujisawa & Futami, 2024](https://arxiv.org/pdf/2406.06227) and [Fujisawa & Futami, 2024](https://arxiv.org/pdf/2406.06227), binning-based ECE estimators suffer from significant estimation bias and slow convergence rates.\n\nAdditionally, some studies have provided theoretical evaluations of bias in ECE estimators based on uniform-mass binning (UMB)([Gupta et al., ICML2021](https://arxiv.org/pdf/2105.04656), [Gupta et al., NeurIPS2020](https://arxiv.org/pdf/2006.10564)).\nThere are some other related studies ([Gruber et al., NeurIPS2022](https://arxiv.org/pdf/2203.07835), [Sun et al., NeurIPS2023](https://arxiv.org/pdf/2305.10886)).\nIt is unclear whether this paper uses UMB or uniform-width binning, but the choice of binning strategy significantly impacts bias and variance in calibration error estimation. Given these findings, it is important to acknowledge the potential limitations of ECE-based evaluation and consider methods that minimize bias, such as properly setting the number of bins or using alternative bias-corrected estimators. Including such discussions would strengthen the empirical robustness of this work.", "essential_references_not_discussed": "To properly position the contribution of this work and ensure a fair evaluation of calibration performance, it is essential to reference and discuss the prior research on calibration-aware learning and bias in ECE estimation that has been highlighted in this review.\n\n- The paper should cite and discuss prior works on calibration-aware learning in classification models (e.g., [Kumar et al., ICML2018](https://proceedings.mlr.press/v80/kumar18a/kumar18a.pdf#:~:text=Trainable%20Calibration%20Measures%20For%20Neural,so%20compro%02mise%20the%20many%20legitimately), [Krishnan et al, NeurIPS2020](https://arxiv.org/pdf/2012.07923), [Karandikar et al., NeurIPS2021](https://arxiv.org/pdf/2108.00106), [Popordanoska et al., NeurIPS2022](https://proceedings.neurips.cc/paper_files/paper/2022/file/33d6e648ee4fb24acec3a4bbcd4f001e-Paper-Conference.pdf#:~:text=We%20propose%20a%20tractable%2C%20differentiable%2C,ECEKDE%20scales%20well)), which are highly relevant to the methodology of this paper.\n\n- The validity of ECE-based evaluation should be reconsidered, given recent studies showing significant bias in binning-based ECE estimators ([Gruber et al., NeurIPS2022](https://arxiv.org/pdf/2203.07835), [Sun et al., NeurIPS2023](https://arxiv.org/pdf/2305.10886), [Gupta et al., ICML2021](https://arxiv.org/pdf/2105.04656), [Gupta et al., NeurIPS2020](https://arxiv.org/pdf/2006.10564), [Futami & Fujisawa, NeurIPS2024](https://proceedings.neurips.cc/paper_files/paper/2024/file/9961e42624a6c083279303767c73269d-Paper-Conference.pdf), [Fujisawa & Futami, 2024](https://arxiv.org/pdf/2406.06227)). If ECE is used as a primary evaluation metric, proper adjustments (e.g., optimal bin selection or alternative estimators) should be considered.", "other_strengths_and_weaknesses": "### Strengths\n- Addresses an important issue in LLM alignment: The paper tackles the important problem of calibration degradation due to alignment, which is highly relevant given the widespread use of LLMs in real-world applications.\n- Proposes a well-motivated fine-tuning objective: The method explicitly balances accuracy and calibration, making it a conceptually reasonable approach to mitigating miscalibration.\n- Provides a theoretical framework: While certain connections could be clarified further, the paper presents a structured theoretical analysis, which helps to ground the empirical observations.\n- Extensive empirical evaluation across multiple LLMs: The paper tests its method on various LLM architectures, demonstrating general applicability.\n\n### Weaknesses\nLack of discussion on the limitations of the proposed method: The paper does not provide a sufficient discussion on when and where the proposed method is expected to perform well or poorly. For example:\n\n- What are the computational costs associated with this fine-tuning method, and how do they compare to other calibration techniques?\n\n- Are there trade-offs in terms of sample efficiency, convergence rate, or stability? Providing some empirical evaluation regarding this topic would help clarify the practical applicability of the proposed method.", "other_comments_or_suggestions": "- In Definition 4.2: Is it needed to provide the definition of $\\mathrm{ECE}(\\pi_{\\theta})$?\n\n- In Appendix A.2, line 575: \"Expected Calibration Error (ECE) Naeini et al. (2015),...\": I think you should use the bibtex-style citation here for [Naeini et al. (2015)].\n\n- In Appendix A.2, line 576: (typo?) \"Mlticlass-ECE...\" --> \"Multiclass-ECE...\"\n\n- In Figures 7 and 8 (p.20-21): Why is the plot of (g) for \"Prob. of Option A\"? I think you should show the plot for \"Prob. of Option B\" instead.", "questions_for_authors": "The following questions are critical for improving the clarity and contribution of the paper. \nIf these issues are appropriately addressed, I will consider raising my overall score.\n\n- Comparison with Alternative Calibration Approaches:\n    - Why is fine-tuning with a calibration objective superior to existing post-hoc calibration methods such as temperature scaling?\n    - What is the relationship and significant difference between this paper and calibration-aware training in classification models, such as those by [Kumar et al., ICML2018](https://proceedings.mlr.press/v80/kumar18a/kumar18a.pdf#:~:text=Trainable%20Calibration%20Measures%20For%20Neural,so%20compro%02mise%20the%20many%20legitimately), [Krishnan et al, NeurIPS2020](https://arxiv.org/pdf/2012.07923), [Karandikar et al., NeurIPS2021](https://arxiv.org/pdf/2108.00106), and [Popordanoska et al., NeurIPS2022](https://proceedings.neurips.cc/paper_files/paper/2022/file/33d6e648ee4fb24acec3a4bbcd4f001e-Paper-Conference.pdf#:~:text=We%20propose%20a%20tractable%2C%20differentiable%2C,ECEKDE%20scales%20well)?\n\n- Evaluation Metrics and ECE Bias:\n    - The paper uses binning-based ECE, which is known to suffer from significant estimation bias (e.g., [Futami & Fujisawa, NeurIPS2024](https://proceedings.neurips.cc/paper_files/paper/2024/file/9961e42624a6c083279303767c73269d-Paper-Conference.pdf), [Fujisawa & Futami, 2024](https://arxiv.org/pdf/2406.06227))). \n    - Could you provide revised experimental results using an optimal bin size setting as suggested by these studies? I think this is crucial to confirm whether the reported empirical results are convincing or not.\n    - Does the paper employ uniform-mass binning (UMB) or uniform-width binning? Since the choice of binning strategy significantly impacts bias and variance, providing clarification would be valuable.\n\n- Limitations of the Proposed Method:\n\n    - In which scenarios is the proposed method expected to perform well, and in which cases might it fail?\n\n    - Are there certain types of LLM architectures, alignment techniques, or dataset conditions where this approach is less effective?\n\n    - What are the computational costs of this fine-tuning method? How do they compare to other calibration techniques? Is the ECE term in the proposed objective function differentiable?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lWW0w0yjb7", "reviewer": "Reviewer_USq4", "summary": "This paper addresses poor calibration in Large Language Models (LLMs) after preference alignment procedures like RLHF and DPO. The authors identify that preference-aligned LLMs exhibit overconfidence due to \"preference collapse,\" where models excessively favor certain responses regardless of their correctness. They develop a theoretical framework distinguishing between \"calibratable\" and \"non-calibratable\" regimes based on model accuracy thresholds, and propose two solutions: (1) Calibration-aware Fine-Tuning (CFT) for models in the calibratable regime, which restores calibration without compromising performance, and (2) Regularized CFT (RCFT) for the non-calibratable regime, which uses EM-algorithm-based ECE regularization to balance calibration and accuracy. Experiments across four models show their methods reduce Expected Calibration Error from 14-20% to 2-7% while maintaining or improving language capabilities and alignment with human preferences.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer USq4 for the insightful comments and questions. Below we provide our responses to the questions.\n\n>Q1. One limitation is that their evaluation focuses primarily on multiple-choice settings rather than free-form text generation, which would provide a more complete picture of calibration in real-world LLM deployments.\n\nA. Thank you for the comment. We agree that calibration in free-form generation is important. We focus on multiple-choice settings as they provide a controlled and quantifiable evaluation of calibration. Extending our method to free-form generation is an exciting direction, and we will discuss it as future work.\n\n>Q2. the constant C in the lower bound lacks specific derivation,\n\nA. Thank you for pointing this out. The constant C in the lower bound arises from solving a min-max optimization problem, which makes it difficult to obtain a closed-form expression. While we do not derive an explicit value for C, we will clarify its origin and theoretical role in the revised manuscript. Characterizing or approximating C more precisely is an interesting direction for future work.\n\n>Q3. The EM-algorithm for probability estimation is theoretically sound, though the proof for convergence is not provided.\n\nA. Thank you very much for the question. When the optimization is performed over probability distributions (rather than neural network parameters), the convergence of our EM algorithm follows from standard EM theory, and we take this as given rather than presenting it as a contribution of our work. However, when optimizing over neural network parameters, convergence is no longer guaranteed due to the non-convexity and complexity of the underlying function space.\nWe will include a discussion on the convergence properties of the algorithm in the revised manuscript to clarify these distinctions.\n\n>Q4. The experimental design is generally sound but has several limitations. While they use appropriate models (four different architectures with both DPO/RLHF alignment), metrics (ECE, accuracy, win rate), and testing conditions (in-domain and cross-domain), the analysis lacks statistical significance testing and confidence intervals. The ablation studies are minimal, only comparing LSFT2 vs LECE in the appendix. The regularization parameter λ is fixed at 1 without sensitivity analysis, and the win rate metric only captures binary preference alignment rather than nuanced quality dimensions.\n\nA. Thank you very much for the helpful comments and suggestions. We have now conducted a more comprehensive ablation study on the hyperparameter λ to further support our analysis. The results of this study have been incorporated into the revised manuscript.\n|$\\lambda$|0||| 0.4|||1|||1.8|||ECE_only|||\n|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|\n|Metric|ECE|cwECE|Acc|ECE|cwECE|Acc|ECE|cwECE|Acc|ECE|cwECE|Acc|ECE|cwECE|Acc|\n|Llama-3.1-8B-Tulu| 0.0883 | 0.0808 | 0.8964 | 0.1535 | 0.1014 | 0.8409 | 0.0897 | 0.0771 | 0.8341 | 0.0178 | 0.0106 | 0.4366 | 0.0002 | 0.0081|0.2475|\n|Vicuna-7B| 0.1219 | 0.0774 | 0.8322 | 0.1620 | 0.0991 | 0.7315 | 0.0474 | 0.0459 | 0.6015 | 0.1052 | 0.0799 | 0.3877 | 0.0130 | 0.0270 | 0.2290 |\n|Olmo2-7B| 0.1003 | 0.0992 | 0.8846 | 0.1771 | 0.1008 | 0.8427 | 0.0989 | 0.0806 | 0.8510 | 0.0038 | 0.0113 | 0.4901 | 0.0030 | 0.0043 | 0.2765|\n|Mistral-7B| 0.0976 | 0.0785 | 0.9091 | 0.1316 | 0.0733 | 0.8085 | 0.0979 | 0.0877 | 0.8297 | 0.0366 | 0.0617 | 0.4217 | 0.0021 | 0.0108 | 0.2670 |\n\n>Q5. How might your calibration methods be adapted for cases where we only have black-box API access to LLMs without fine-tuning capabilities?\n\nA. Thank you for the question. Our approach relies on fine-tuning the model, and is therefore not applicable to black-box APIs. Adapting our method to black-box models would go beyond a simple extension—it would effectively require developing an entirely new approach. For improving calibration in black-box settings, techniques such as prompt engineering or encouraging the model to explicitly express its confidence are more suitable. We will include this discussion in the revised version of the paper.\n\n>Q6. Have you considered how quantization or other efficiency techniques might impact calibration properties in aligned models?\n\nA. Thank you very much—this is a great question. Previously, we did not explore the impact of quantization or other efficiency techniques on calibration. These methods can potentially alter the model's confidence estimates and thus affect calibration quality. We agree this is an important and practical direction, especially for deployment scenarios, and we will include it in our discussion of future work.\n___\nTo all reviewers: Additional experiments of alignment ability. Due to limited time, only 2 models for Arena-hard.\n|Alpaca-Eval|DPO| CFT| RCFT |\n|-|-|-|-|\n|  Llama-3.1-8B-Tulu |  21.4|22.6|19.6|\n| Vicuna-7B| 2.6|2.6|3.6|\n| Olmo2-7B | 24.2|22.9|23.1|\n| Mistral-7B| 26.0|26.8|25.2|\n|Arena-hard||||\n| Olmo2-7B|19.4|19.2|20.2|\n| Mistral-7B| 18.9|18.3|18.0|", "claims_and_evidence": "The paper's claims are generally supported by evidence, with experimental results clearly demonstrating ECE reductions from 14-20% to 2-7% across models.  The connection between preference collapse and poor calibration shows correlation but not definitive causation. Win rate metrics support their claim about maintaining alignment capabilities, but this represents just one dimension of alignment quality. While they demonstrate cross-domain generalization, more diverse evaluation scenarios would strengthen their broader applicability claims.", "methods_and_evaluation_criteria": "The calibration-aware fine-tuning methods directly target the identified overconfidence issue through targeted loss functions, which is sensible given the problem definition. Their use of multiple-choice QA datasets is appropriate since they provide clear ground truth for calculating calibration metrics. The confidence and classwise ECE metrics are standard for calibration assessment, making their evaluation protocol methodologically sound.\n\nTheir evaluation balances both in-domain and out-domain generalization, tests multiple model architectures aligned with different methods (DPO and RLHF), and importantly, measures win rate on preference pairs to verify alignment preservation. This comprehensive approach addresses the key concern that improving calibration might compromise alignment quality.\n\nOne limitation is that their evaluation focuses primarily on multiple-choice settings rather than free-form text generation, which would provide a more complete picture of calibration in real-world LLM deployments.", "theoretical_claims": "I verified the paper's theoretical proofs, focusing on:\n\nProposition 4.1: Correctly proves that probabilistic generative models achieve zero ECE by definition, as predicted probabilities match observed frequencies.\nTheorems 4.4 and 4.5 (Upper/Lower Bounds of TCE): The bounds are correctly established by constructing appropriate examples, but the constant C in the lower bound lacks specific derivation, making practical application less clear.\nTheorem 4.6 (Upper bound for ECE): This uses triangle inequality to establish that classwise ECE is bounded by TCE, which is mathematically valid.\n\nThe calibratable/non-calibratable regime distinction follows logically from these bounds, though the paper simplifies this conceptually when moving to practical implementation. The EM-algorithm for probability estimation is theoretically sound, though the proof for convergence is not provided.", "experimental_designs_or_analyses": "The experimental design is generally sound but has several limitations. While they use appropriate models (four different architectures with both DPO/RLHF alignment), metrics (ECE, accuracy, win rate), and testing conditions (in-domain and cross-domain), the analysis lacks statistical significance testing and confidence intervals. The ablation studies are minimal, only comparing LSFT2 vs LECE in the appendix. The regularization parameter λ is fixed at 1 without sensitivity analysis, and the win rate metric only captures binary preference alignment rather than nuanced quality dimensions.", "relation_to_broader_scientific_literature": "This work extends prior work on LLM calibration by Jiang et al. (2021), Xiao et al. (2022), and Chen et al. (2022) who identified miscalibration issues in LLMs, but specifically addresses the previously unexamined problem of how preference alignment techniques (RLHF/DPO) impact calibration.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "See comments above.", "other_comments_or_suggestions": "No.", "questions_for_authors": "1: How might your calibration methods be adapted for cases where we only have black-box API access to LLMs without fine-tuning capabilities?\n\n2: Have you considered how quantization or other efficiency techniques might impact calibration properties in aligned models?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EOgd18YSNk", "reviewer": "Reviewer_hpMa", "summary": "The paper points out the problem of overconfidence of preference aligned Large Language Models (LLMs), and propose two fine-tuning approaches to address the problem: calibration-aware fine-tuning (CFT) and regularized CFT (RCFT).", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer hpMa for the comments and suggestions. Below we provide our answer to your questions.\n\n>Q1. The biggest problem is ... not propose any modifications to preference alignment.\n\nA. Thank you for the comment. We would like to clarify that this is better to be considered as a realistic practical scenario instead of a problem.\n\nTo begin with, consider a practical deployment scenario, which is the primary focus of our work. A practitioner aiming to develop their own model typically starts by downloading an open-source LLM and adapting it to their specific needs. Importantly, for many such models, the intermediate checkpoints (e.g., SFT stages) are not publicly available. As a result, practitioners are often unable to modify or re-run the full preference alignment process. In such cases, post-alignment fine-tuning is the most accessible strategy. Our approach is designed for this realistic setting—providing calibration improvements without requiring access to earlier training stages.\nWe have revised the paper to make this point clear and more explicit to readers.\n\nIn the scenario where the practitioner is able to go through the full training pipeline, this is also not an issue. We provide our answer below.\n\n>Q2. What is the overall post-training pipeline? Justify the long post-training pipeline.\n\nA. CFT is a post-hoc fine-tuning step applied after RLHF, and we will clarify this in the revised manuscript. Following  our previous response, in scenarios where developers will go through the full training pipeline, both replacing RLHF and post-hoc fine-tuning are valid options—what matters most is which method yields better performance.\n\nMethods that aim to replace RLHF—such as PPO-C in [2]—are promising, but they are evaluated under different experimental settings, making a direct comparison difficult within a short timeframe. In our work, we demonstrate the effectiveness of our approach by outperforming the strong baseline of TS. We believe our method is robust and competitive with RLHF-replacement approaches.\n\n>Q3. It is an unfair comparison w/wo domain knowledge.\n\nA. We believe that the comparison remains appropriate for several reasons.\n\nFirst, regarding ECE, the concern about fairness seems to stem from intuition rooted in more standard metrics like accuracy, where additional domain knowledge can indeed lead to higher performance. However, ECE measures the discrepancy between confidence and accuracy, domain knowledge does not inherently bias ECE comparisons.\n\nSecond, we have included experiments on out-domain tasks to further mitigate any potential effects from domain-specific knowledge. \n\nFinally, the ultimate goal of training language models is to obtain models that perform well in real-world settings. If incorporating additional knowledge leads to better-calibrated models, it should be considered a strength rather than a limitation.\n\n>Q4. It is unclear … hard to tell when people should use methods in Sec 5.1 and when to use methods in Sec 5.2.\n\nA: Before addressing it directly, we would like to first clarify a potential misunderstanding regarding Section 5.\n\nIn Theorem 4.6, we prove that ECE ≤ TCE, which implies that minimizing TCE is sufficient for achieving low ECE. This reduces the goal of obtaining a well-calibrated model to finding one that is close to the target probabilistic model, formulated as the following constrained optimization problem:\n$$\\max \\text{ACC}(\\pi) \\quad \\text{s.t.} \\quad \\text{ECE}(\\pi) = 0.$$\n\nIn Section 5, we focus on how to approximate both accuracy and ECE in the context of LLMs—covered in Sections 5.1 and 5.2, respectively. \n\nNow, returning to your question: Sections 5.1 and 5.2 do not present two separate methods, but rather provide a breakdown of our final algorithm.\n \n>Q5. I think the authors should not restrict to only four possible answers (only four-answer setting (A,B,C,D) is allowed), as that is not practical in real applications.\n\nA. We chose to use the four-option format (A, B, C, D) for concreteness and readability, aiming to reduce the burden of mathematical notation and make the presentation more accessible to experimental researchers. This is explicitly stated in our paper (lines 113–114): “While we consider four options in our analysis for concreteness, the framework naturally extends to any number of alternatives.” We will further revise the text to make this generality clearer.\n\n>Q6. The original preference alignment ability.\n\nIn addition to the win rate results presented in our original submission, we have conducted three more experiments related to preference alignment: CFT vs. DPO, AlpacaEval, and Arena-Hard. Due to space constraints, we refer to our response to Q4 of QE3c and the end of our response to USq4 for the detailed results.\n\n>Q7. Related work, suggestions on explaining the two regimes at first and other writings.\n\nA. We have included the 4 mentioned papers in the revised version of the paper and have revised the paper accordingly.", "claims_and_evidence": "The biggest problem is that though the authors state that preference alignment (like RLHF and DPO) causes poor calibration in LLMs, they do not propose any modifications to preference alignment, but instead propose fine-tuning approaches. As we know that Supervised Fine-Tuning (SFT) and RLHF are two different post-training stages, it is very essential for authors to justify why fine-tuning (rather than improving over the current RLHF framework) is a better solution, and should people use the proposed CFT before or after RLHF, or completely replace RLHF with CFT.\n\nMoreover, it is an unfair comparison between an extra domain-knowledge fine-tuned LLM and an LLM not trained with domain knowledge. Intuitively the trained LLM will have more information on the in-domain tasks.", "methods_and_evaluation_criteria": "Theoretical explanation:  \nThe authors propose calibratable and non-calibratable regime. It is better if they can explain at first that the difference of these two regime is that whether it is possible to achieve perfect calibration without sacrificing accuracy.\n\nProposed SFT method:  \nIt is unclear why the proposed domain-specific fine-tuning in sec. 5.1 can help improve LLM calibration.\nIn both Sec 5.1 and 5.2, the authors no longer mention calibratable or non-calibratable regime at all. It is hard to tell when people should use methods in Sec 5.1, and when to use methods in Sec 5.2.\n\nEvaluation criteria:  \nThe authors use four RLHF-trained models and evaluate on a wide range of datasets.", "theoretical_claims": "The theoretical proofs looks good, though I think the authors should not restrict to only four possible answers (only four-answer setting (A,B,C,D) is allowed), as that is not practical in real applications.", "experimental_designs_or_analyses": "The experiments are rather complete on four tasks and the authors use a wide range of metrics: ECE, classwise-ECE, accuracy and win-rate. \n\nHowever, it is unclear which stage the authors apply CFT. If they apply CFT after RLHF, then that means post-training of a pre-trained model will now include three steps: traditional SFT, RLHF, and CFT. This will create a very long training pipeline which is not practical in real practice, and authors need to discuss this limitation.\n\nSince the authors propose to calibrate RLHF-trained models, it is also important to see if the calibration approach will hinder the original preference alignment ability, such as testing their models on benchmarks like Arena-Hard. Otherwise they are simply fine-tuning a model for a specific usage instead of generally improving the RLHF-trained models.\n\nIn fact, there are already previous works that improve LLM calibration through RLHF and maintain preference alignment ability [1][2], and they did a lot of study on whether improving the RLHF framework will hinder other generic capabilities.\n\n[1] When to Trust LLMs: Aligning Confidence with Response Quality. ACL 2024. https://arxiv.org/abs/2404.17287  \n[2] Taming Overconfidence in LLMs: Reward Calibration in RLHF. ICLR 2025. https://arxiv.org/abs/2410.09724", "relation_to_broader_scientific_literature": "The authors echo with previous research that preference aligned LLMs are more overconfident. However, they are proposing an SFT-based approach to improve model calibration. There are of course methods directly improving over traditional SFT [1][2], but the authors in this paper are proposing a CFT method after the RLHF stage, they will need to justify the long post-training pipeline of: SFT -> RLHF -> CFT.\n\n[1] Teaching Models to Express Their Uncertainty in Words. https://arxiv.org/abs/2205.14334.  \n[2] Enhancing confidence expression in large language models through learning from past experience. https://arxiv.org/abs/2404.10315.", "essential_references_not_discussed": "Improving over SFT:  \n[1] Enhancing confidence expression in large language models through learning from past experience. https://arxiv.org/abs/2404.10315. \n\nImproving over RLHF:  \n[2] When to Trust LLMs: Aligning Confidence with Response Quality. ACL 2024. https://arxiv.org/abs/2404.17287  \n[3] Taming Overconfidence in LLMs: Reward Calibration in RLHF. ICLR 2025. https://arxiv.org/abs/2410.09724", "other_strengths_and_weaknesses": "The paper writing is problematic:  \n\n1. The definition of multiple-choice question in Sec. 3 and Sec 4 is too narrow: only four-answer setting (A,B,C,D) is allowed. The authors can relax to a few number of choices, not just four.\n\n2. Many grammar errors in paper writing, just to list a few:\n\"Another researchers teach the LLMs\" -> \"Other researchers teach...\"  \n\"The DPO method is to directly optimize of the policy without explicitly training the reward function\" -> \"The DPO method is to directly optimize the policy ...\"\n\n3. Figure 2. The description of y-axis is unclear, and the authors need to mention which \"value\" it is.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1. What is the overall post-training pipeline when CFT is applied?  \n2. When to use CFT and when to use RCFT?\n3. Will CFT training reduce model's preference alignment ability?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper addresses the poor calibration of Large Language Models (LLMs) after preference alignment (e.g., RLHF/DPO). The authors propose two methods: calibration-aware Fine-Tuning (CFT) and Regularized CFT (RCFT). CFT restores calibration without sacrificing accuracy, while RCFT balances calibration and accuracy for models in "non-calibratable" regimes. Theoretical analysis distinguishes calibratable/non-calibratable regimes based on accuracy thresholds. 

Strengths:
1. The paper tackles a critical issue in LLM alignment. 
2. Methodology: The EM-based CFT/RCFT is well-motivated, and experiments show consistent improvements across models and domains.

Concerns raised by the reviewers that haven't been fully addressed:
1. Theoretical Nuance: While bounds on TCE are proven, the practical distinction between calibratable/non-calibratable regimes (e.g., architecture-dependent thresholds) remains under-explored.
2. Evaluation Scope: Limited ablation studies (e.g., sensitivity to regularization parameter λ).
3. Generalization to a broader case (e.g. beyond multichoice problems of 4 options)

Note that the authors also address several questions raised by the reviewers regarding: baseline comparison on temperature scaling as well as presentations of the paper for better clarity.

Overall the paper is theoretically-grounded paper addressing important issues of miscalibration in LLM alignment post-training.  While minor issues persist, the rebuttal effectively addresses major concerns, and the core claims are validated.
Decision: Accept (poster)