Calibrated Language Models and How to Find Them with Label Smoothing
Authors: Jerry Huang
Peng Lu
QIUHAO Zeng
Venue: ICML 2025 poster (2025)
Keywords: model calibration
label smoothing
large vocabulary large language models
Primary area: deep_learning->large_language_models
TL;DR: Label smoothing helps calibration but is less effective for large vocabulary LLMs, therefore we identify the root causes and provide a custom kernel for better training with label smoothing.

Abstract: Recent advances in natural language processing (NLP) have opened up greater opportunities to enable fine-tuned large language models (LLMs) to behave as more powerful interactive agents through improved instruction-following ability. However, understanding how this impacts confidence calibration for reliable model output has not been researched in full. In this work, we examine various open-sourced LLMs, identifying significant calibration degradation after instruction tuning in each. Seeking a practical solution, we look towards label smoothing, which has been shown as an effective method to regularize for overconfident predictions but has yet to be widely adopted in the supervised fine-tuning (SFT) of LLMs. We first provide insight as to why label smoothing is sufficient to maintain calibration throughout the SFT process. However, settings remain where the effectiveness of smoothing is severely diminished, in particular the case of large vocabulary LLMs (LV-LLMs). We posit the cause to stem from the ability to become over-confident, which has a direct relationship with the hidden size and vocabulary size, and justify this theoretically and experimentally. Finally, we address an outstanding issue regarding the memory footprint of the cross-entropy loss computation in the label smoothed loss setting, designing a customized kernel to dramatically reduce memory consumption without sacrificing speed or performance in comparison to existing solutions for non-smoothed losses.

Calibrated Language Models and How to Find Them with Label Smoothing
Jerry Huang 12
Peng Lu
Qiuhao Zeng?
Abstract
2023). However, such models have demonstrated propen- sity for over-confidence in their predictions (Zhao et al., 2021; Jiang et al., 2021; Xiong al., 2024), eliciting con- cerns over their use in more high-stakes decision decision-making making scenarios. Such observations are not new with respect to neural networks, which have consistently been shown to suffer from over-confident predictions and over-estimate the likelihood of their correctness (Guo et al., 2017; Szegedy et al., 2016; Müller al., 2019; Naeini et al., 2015; Minderer
Recent advances in natural language processing have enabled the fine-tuning of large language models (LLMs) into powerful interactive agents with improved instruction- followwngg ability. How- ever, this can impact confidence calibration for reliable model output, which has not been re- searched in full. In this work, we examine various open-soureed LLMs, where we identify signifi- cant calibration degradation after instruction tun- ing. Seeking a practical solution, we look towards label smoothing, which has been shown as an effective method to regularize for overconfident predictions but has yet to be widely adopted in the supervised fine-tuning (SFT) of LLMs. We pro- main-
et al., 2021). To improve this, methods such as tempera- ture scaling (Guo et al., 2017) and label smoothing (Müller et al., 2019) have been proposed as solutions with varying effectiveness, spurring additional work in ensuring that pre- dictions and confidence remain matching (Lin et al., 2017: Mukhoti et al., 2020; Pereyra al., 2017; Liu et al., 2022).
vide insight into why label smoothing tain calibration throughout the SFT process, but identify settings remain where the effectiveness of smoothing is severely diminished. We posit the cause to stem from the ability to become over- confident, which has direct relationship with the hidden and vocabulary size of models, which we justify theoretically and experimentally. Fi- nally, we address outstanding issue regarding the memory footprint of the cross-entropy loss computation with label smoothing, designing a customized kernel to dramatically reduce memory consumption without sacrificing speed or perfor- mance in comparison existing solutions.
In this work, we focus on label smoothing (LS) for calibra- tion in SFT settings. Motivated by previous works showing the effectiveness of LS for calibration in different settings, we first verify its effectiveness. We demonstrate that while it can be effective, problems begin to emerge in the case of large vocabulary L.MSs (L.--LLMs). To explain this phenomenon, we attempt establish a link between the pre- dictive abilities of such LLMs and their size. We show that in such settings, explicit link between the lower bound of the model entropy and the hidden size of the model causes models to fail to become overconfident (Zhao et al., 202 I), negating the potential benefits and use of label smooth- ing. We further show how alternative techniques, such as temperature scaling and logit capping, explicitly act as a mechanism to steer models toward overconfidence, allowing the benefits of label smoothing to once again emerge.
1. Introduction
Nevertheless, a problematic setting still remains. Growing vocabulary sizes cause large amounts of memory to be con- sumed to materialize the relevant logits and probabilities, making training difficult. While efficient methods (Wijmans et al., 2025; Hsu al., 2024: Py PyTorch, 2024) have been proposed by implementing hardware-level optimization that significantly reduces this memory bottleneck, they often cannot support label smoothing. To address this, we iden- tify specific optimizations that can be made in computing matrices in GPU memory, allowing for the support of label smoothing with minimal increases memory or computa- tional speed. Thus we introduce new kernel that enables to use label smoothing more efficiently as whole.
Tremendous progress has been made in building models that follow natural language instructions (Ouyang et al., 2022: Sanh et al., 2022: Chung et al., 2024) through the use of LLMs tre-trai on large amounts of data as well as high- quality datasets that enable them to learn to interact in human-like manner (Bach et al., 2022; Wang et al., 2022;
Equual contribution 'nniversité de Montréal "Mila- Quebec AI Institute UUniversity Western Ontario. Corresponding author: Peng Lu <peng. lu@umontreal.com
Proceedings of the nnnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Calibrated Large Language Models and How to Find Them with Label Smoothing
To summarize our contributions, we: 1) We point out that common SFT practices significantly degrade LLM model calibration. 2) We demonstrate and justify why la- bel smoothing is an appropriate approach to mitigating this concern. 3) However, we further identify specific issues where label smoothing remains prohibitive, particularly with large vocabulary LLMs, and why existing methods fall short. 4) We demonstrate that optimizations exist, which we in- corporate in custom kernels which enable us to perform label smoothing with significant memory and throughput improvements without performance sacrifices.
model parameterized by 0 to model a conditional distri- bution P(-Ja: 0), where the final operation is softmax. Consider the model to apply a function f(:0) on and (x; E [0, 1] to be the post-softmax output. Then
exp(e(x)::
P(N]æ 0) = ox: ():
(3)
Σ K k= exp(e(x))))
where {(x) IR K is the pre-sof tmax output of the model, commonly referred to as the logits log-probabilities. Mod- els are usually trained by minimizing cross-entropy (CE) loss on a dataset DD = { Un J==1 sampled from an un- known distribution p(æ, y) in order to learn the true condi- tional distribution Py|a (3)2)) computed as
2. Related Work
Model Calibration (Brier, 1950; Murphy, 1972; DeGroot & Fienberg, 1983) is a concept of matching the prediction probabilities yielded for different inputs to the expected ac- curacy on these inputs. In a K K-way classification setting, let X' E IR and Y' E fru ) indicate the input and label space, respectively. Let / be a classifier and ʃ (ú ar) = é be the confidence of prediction, i.e., the maximum of probabil- ities among K dimensions corresponding to its prediction ý. model is perfectly-calibrated when
CCR CD
00)
Un log P(Tx|ar: 0)
N
i=
k=1
K
= Ey(m.))
Σ p()xl>)) log P( (x; (x))z;))
(4)
= -Ep(r,))
[KL
0)] +
= C
p(z (0), (0),
Pÿ = y/é = c) = c Vc [0. 1].
(1)
where 8 is the Kronecker delta with value only when = j. Label smoothing mixes the original distribution with discrete uniform distribution U = [1/K]* E RK using a smoothing rate 3 E [O,1]. The loss then becomes
Qualitatively, model calibration can be derived as [ = 5/c - c) - c]] One metric that has been widely used for measuring calibration is the expected calibration er- ror (ECE) (Naeini et al., 2015). which is weighted average of bin-wise mis-calibration. The ECE divides the confi- dence score of N samples into M uniform confidence bins {Bm } and takes the mean of the gap between accuracy (acc) and confidence (conf) over the bins weighted by the number of samples in the bins.
K
1 N
3 K
0
(1
3)8 Un
+
log P(
N
B
= (1 3) 3) (0) +
KT. [u | 0) +
K
2 - Ep(a..))
[KL [(1 (xx + Bullo 0)]l +
Bml N
ECE- Σ
Jacc (Bm) conf (Bm)l-
(2)
CLS
p)) (0).
(5)
Additional metrics that have been proposed include the Root Mean Square Calibration Error (RMS-CE) (Hendrycks et al., 2019), which places greater emphasis on large calibration deviations, and the Static and Adaptive Calibration Errors (SCE/ACE) (Nixon et al., 2019), which measure miscal- ibration over fixed and data-dependent binning schemes, respectively. These metrics offer complementary perspec- tives to standard calibration eтror measures, enabling more comprehensive assessment of model confidence alignment.
Thus, label smoothing can be understood to regularization term that encourages uniform distribution over the output labels, hence preventing it from over-fitting to the training data and encouraging a model to be less confident on all samples by smoothing the true conditional being learned.
3. Smoothing and Calibration in LLMs
Preliminaries. Define an auto-regressive LLM to be pa- rameterized by parameters O. A model represents embed- ding function g(; 0.) R-N - DxN where I, is the length of a discrete input sequence æ and D is the hidden size of the model that produces an embedding E E RDxN This is followed by a classifier C(e.) E DxVV and a softmax operation to produce a probability distribution over V.
Label Smoothing (LSS) has been demonstrated to be promising paradigm in settings to prevent models from be- coming overconfident (Szegedy et al., 2016; Müller et al., 2019) or when noise exists in the provided labels (Lukasik et al., 2020; Wei al., 2022b; Lu al., 2023). Consider
2

---

Calibrated Large Language Models and How to Find Them with Label Smoothing
A common practice is tune 0 [0=, 0e] on a dataset through supervised fine-tuning (SFT), instruction tun- ing (Wei et al., 2022a; Ouyang et al., 2022; Chung et al., 2024; Dubois et al., 2023). During SFT of an LLM, an input sequence æ of length I. consists of a sequence of discrete vocabulary tokens z; E V Vi E [N]. The first m tokens in a consist of the instruction, while the rest is considered the target output. The goal of SFT is to minimize CE objective over the output portion of the sequence Im+1:: computed as loss over the individual elements of the sequence
Gemma-2B
LLaMA2-7B
1.00
1.00
0.75
0.75
Inderconni inderconfidemt
Indercome
0.50
0.50
0.25
0.25
Overconfident
Overconfident
0.00
0.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
LLaMA3.1-8B
Mistral-7B
1.00
1.00
(x) = Σd 82 log (C(0.)) g( 1.0
(6)
0.75
Underconfident
0.75
Underconn Underconlident
Accuracy
0.50
0.50
In essence, the learning problem is a -i-lass classification problem for each element in the target portion the input sequence, where the prediction for any specific position is influenced by all previous elements in the sequence.
0.25
0.25
Overconfident
Overconfident
0.00
0.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
Qwen2-7B
Qwen2.5-7B
This procedure transforms the original parameter set 0 into a new set, denoted OSFT, which often enables the model to follow human-provided instructions with remarkable ac- curacy and fluency. However, this fine-tuning process can also lead to a deterioration the model's calibration, as illustrated in Figure I, potentially reducing its reliability in estimating uncertainty confidence.
1.00
1.00
Underconfident
0.75
Underconfident
0.75
0.50
0.50
0.25
0.25
Overconfident
Overconfident
0.00
0.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Why does Instruction Tuning Lead to Mis-calibration? Our results raise a question: Why does instruction tuning degrade calibration? To better understand this, SFT can be viewed through the lens of out-of-distribution (OOD) gen- eralization and calibration error. In particular, we assume that the SFT data consists of an in-distribution (ID) dataset whereas the downstream dataset on which generalization and calibration are tested constitutes OOD dataset.
Confidence
SFT
False
True
Figure I. Reliability diagrams of open-sourced pre-trained models with (red) and without instruction-tune -uning (blue) on the MMLU dataset (Hendrycks al., 2021). The horizontal axis represents the model's confidence in each answer choice for each ques- tion, while the vertical axis shows the accuracy on each question. The solid diagonal indicates perfect calibration, separating areas where predictions are deemed over-confident or mnner-confident. Instruction- -tuiig visibly leads to over-confidence. regardless of the instruction- cottucttoo-ttningggg dataset (which differs between models).
Suppose we have an ideal parameter set 0" that minimizes calibration error on an unknown dataset D, defined as Ef 0) c(x)][] where c(z)= = ( [y] the expected label given prediction f(x:; 0). In other words, f(: 0) always produces the calibrated prediction for the label given an input æ such that the output confidence matches the expected label over the subset of samples with the same confidence value. Thus the goal of SFT is to learn a set of parameters 0* that outputs reliable prediction proba- bility on samples from both an unseen OOD domain, which defined by a distribution PooD (a. y), as well as the ID distribution PIn (=) y). Thus the goal is to minimize
final classifier (in the case of an auto-regressive L.M,, this is the language modeling head) has a direct relationship to the diversity of the feature embeddings as follows:
Lemma 3.1 (Oh et al. 2024). Let f(:00) X +
[0, be a real-valued function of the form f(x; 0)
fixliii) 0) where fel:; () is an arbitrary one- dimensional function, and f is in hypothesis class F that has pseudo dimension Paim =dy. Let DiD
(0) (x; 0)- f(; 0* 1
for p E {pooD (x, (2,
(1i1)) ViD (n)) } be a dataset sampled from the ID dis- n=1 tribution. If (a[1],... x(d)) have matching marginals for ID and OOD, and (ar (1), a())) a bi-variate Gaussian for
N
Oh et al. (2024) demonstrate that under such a setting, si- multaneously maintaining accuracy and calibration of the
3

---

Calibrated Large Language Models and How to Find Them with Label Smoothing
Gemma-2B
Gemma2-28
LLaMA2-78
20
3
10
Smoothing
Value
RMS
ECE
SCE
ACE
TÁCE
Avg.
CE
RMS
ECE
SCE
ACE
TACE Avg CE
RMS
ECE
SCE
ACE
TACE AVE CE
0.0
LLaMA3-8B
LLaMA3.2-38
Mistral-7B
a.1
5.0
ECE
SCE
ACE
RMS
BCE
ACE
IACE
CE
RMS
ECE
SCE
ACE
FACE
Aug.
Metric
Figure Effects of instruction-tuninn on calibration, presented under a number of different calibration error metrics (where lower better). Values range from 0 100. Models all fine-tuned Tulu3 (Wang al., 2023) SFT dataset and evaluated MMLU, We can observe that across all models, which have various structural differences, the use of label smoothing is capable of reducing calibration while having negligible effects downstream performance accuracy the task.
every i,j E [d], then for any 5 E 10 and for all f, the following bounds hold with probability least 8:
feature embeddings (the embedding of the context this scenario) becomes less mutually dependent, both calibration eπror and classification error can be minimized. However, significantly prior works have shown that fine-tuning reduce the diversity of such features (Mukhoti et al., 2024; (8) Kumar et al., 2022; Huh et al., 2024), justifying why stan- dard SFT can significantly degrade calibration (Figure 2).
d
Looo (0) (
A
miin (pppote))
N
da
log
dn
No
The Effects of Label Smoothing. To understand the ef- fects of label smoothing from a model calibration perspec- tive, adopt optimization-based viewpoint analyze the implicit constraints imposed by the regularization (Bert- sekas, 1999). This perspective reveals how label smoothing modifies the solution space of the model's predictions, en- couraging not only smoother output distributions but also reduction in the model's confidence on any single class. First, define
E (x.9)
[If (x; 0)
-12
Poop (zr
[Ic(
<DD (0) +
+
(aprn(a))
min
N
dn
1
+0
log
dn
N8
(9) Definition 3.2. The logit distance vector for æ. dlxx), is
where
8" = argmin
Cpooo (mn.)) (0) (pmn(a.y)) (0),
d(x)
max ((x); - l(x)2 k=1 Isisk
E
(11)
oE
(10)
A (ooon(n.g) (0") Cpm(( (0"),
One way ensuring that a model does not over-estimate a specific class is to enforce this as hard constraint, which results in equal logits among all classes and softmax output of preferable to enforce this soft-penalty function P Ø + IR into the objective function minimized during training. Recalling Equation (5), we can relate this soft- penalty to the additional KL-divergence introduced by the label smoothing objective.
and E Epp(() Expo( (e)) (0 [ is a covariance matrix with strictly positive minimum singular value of d-dimensional normalized input ãे (i[1),... [dd), where [xx]]= E[<(i]]) Var(x[i)) and a minn (M) is the smallest singular value of matrix M E xd,
f(x; 0) [I/K]K As such, it is often
The dependence of the bound on the minimal singular value of the covariance matrix indicates that as the set of learnt

---

Calibrated Large Language Models and How to Find Them with Label Smoothing
Proposition 3.3. A linear penalty for Lagrangian term) for the hard constraint d((x) O is bounded from above and below by KL (ullēr (æ; 0)), up to additive constants
Calibration Improvement
HRS C22
522
ACE
d (z);
0.24
KL[ullō (æ; 0)] K <
< KL [ullō (a; 0)]
(12)
The proof (in Appendix A. indicates that label smoothing approximately minimizes, for a linear penalty, the constraint d(x: = 0, encouraging equality among the logits for each class to ensure that over-confidence is penalized. More importantly, however, it is noted that
0.00
18
Proposition 3.4. Define a likelihood model p (yyæ: = Cat (softmax (f (a:; 0))), a categorical distribution with parameters& z softmax (J (x:; 0)) E (()) where A (@) denotes a probability simplex over the parameter space O. The label smoothing objective is equivalent to Maximum A Posteriori (MAP) estimation on the softmax probability vector under the independence assumption p (z|z> = p (z).
LLama 3
Figure 3. Calibration of different LLaMA3 models fine-tuned on the same SFT dataset. As the size of the model decreases, the calibration of the model sees less improvement from the use of LS.
Entropy Gap relative to dimension size WD
A proof is provided in Appendix A.2. This MAP formula- tion above relies on the provided label y for each sample a, without exploiting the potential similarities among dif- ferent samples in the empirical training dataset for more accurate estimation. Lukasz Rajkowski (2019) showed that using MAP estimation can lead to greater separability and diversity of individual samples, under the assumption that the sample is sampled from a normal distribution. Chi et al. (2024) further prove this to be the case with Transformer- based language models. In conjunction with prior claims from Oh et al. (2024), this proposition indicates that label smoothing can in fact learn more diverse input features, further explaining the improvement in calibration.
Normaltzed Entropy Gap
-5
Figure 4, Relative entropy bound for different LLM vocabulary sizes with varying hidden sizes (D). Our visualization shows the normalized entropy gap for varying hidden sizes of the LM head. This gap is calculated by taking the difference between the entropy upper and lower bounds and dividing by the upper bound (log VV). A lower ratio indicates the model is restricted producing concentrated predictions.
4. Label Smoothing for Large Vocabularies
We study the concentration behavior of the LM head by analyzing its relationship between entropy and model size, which provides an explanation of diminishing returns of label smoothing. We start by providing the following lemma, which establishes bound on the LLM logits prior to the language modeling head.
We conduct SFT training with and without LS on a Tulu3 dataset (Wang et al., 2023) for different pre-trained language model families, including Llama (Grattafiori et al., 2024), Gemma (Gemma Team, 2024) and Mistral (Jiang et al., 2023). While we can note the usefulness of label smooth- ing for model calibration shown in Figure 2, it becomes clear that its effectiveness is much less visible in some cases. Take for instance three LLaMA3 models of sizes 1B, 3B and 8B, which we fine-tune on the same instruction dataset (Fig- ure 3). While label smoothing shows an improvement in calibration for the 8B-sized model, this diminishes signifi- cantly the 3B model and IB model, both in the baseline model (where no SFT procedure has been performed) as well as SFT with multiple different datasets. We therefore seek to investigate the underlying causes of our empirical findings, aiming to shed light on a previously unexplained phenomenon.
Lemma 4.1. Let C E D be a matrix with ||C]], ag, and let h RD be a vector such that each entry of satisfies [h, S The The upper bound for ||C hl|2 is:
||CTull, s0c- oh VD.
(13)
Thus, the norm of the final logit vector is also bounded and depends on the dimension dimension, which enables an entropy-based analysis of the LM head's predictions as a function of both the vocabulary size [V] and the embedding dimension D.
5

---

Calibrated Large Language Models and How to Find Them with Label Smoothing
Theorem 4.2. (LM head Entropy Lower Bound). Let p= acTh u C h and ? exp (-P V then the entropy Hu of prediction of the LM head holds that:
size to become over-confident.
DVV
Gemma-2B
P.T/DIV (VV- 1)
Hu log (1 (V 1)7)
(14)
6-
1+
1+ ( 1)
Intuitively, the above lemma and theorem (proofs in Ap- pendix A.3) indicates that for large [V], the minimum tropy is lower bounded by S2 (IV) e linearly with the vocabulary size [V] and decreasing ex- ponentially with vD. We illustrate how this changes in Figure 4, where the normalized entropy gap for different [V] is shown. Given the same [V,, the concentration be- havior of the LM head is primarily influenced by the size of the hidden dimension. As the hidden size increases, the model is increasingly capable of attaining lower entropy, while the bound is smaller for larger |V| at the same D, highlighting why large vocabulary LLMs at smaller sizes are less prone to overconfidence during tuning.
RMS
ECE
SCE
ACE
TACE
Avg. CE
Gemma2-28
pvD). [[V])))VD)
thus increasing
30
Value
20
10
RMS
ECE
SCE
ACE
TACE
Avg. CE
LLaMA3.2-1B
Remark 4.3. Models with smaller D with large vocabulary size VV suffer from lack of concentration ability due to their limited hidden size. By consequence, label smooth- ing cannot help with calibration, as serves to only pe- nalize overconfidence while having no specific benefits for cnder-confidence due to mixing with the maximum entropy (uniform) distribution.
RMS
ECE
SCE
ACE
TACE
Aug. CE
Metric
Smoothing
0.0
0.1
0.2
Figure 5. Effect of label smoothing on large vocabulary models with smaller hidden size (2048). Gemma-2B observes smaller change compared LLaMA3 2-1B, due to having the largest vo- cabulary size. However, Gemma2-2B observes a large change part thanks the softcapping of logits.
However, we note that from this analysis, the entropy bound influenced by V and through the distribution of h. However, h are the logits of the model, these can be manipulated before the softmax directly. Thus follows that under-confidence in models can be attained through manipulation of the || h1l::
5. Efficient Smoothed Cross-Entropy
Despite our analysis on the effectiveness of label smoothing for calibrating LLMs, areas of concern still exist regarding practical applicability. In this section, we introduce such limitations and how the label smoothing setting is distinct from existing solutions that attempt to mitigate them.
Remark 4.4. Fix V and D.. Using a temperature T will modify p. leading to decreased lower bound.
We note that various methods can serve to manipulate this bound. The above remark notes the effects of tempera- ture scaling, which directly uses a constant temperature T scale the logits before the softmax. This enables the manipulation of σ in Lemma 4.1 without changing any additional values, thereby increasing or decreasing the en- tropy bound based on the choice of T. This explains how temperature scaling can enable better model calibration, es- pecially through the choice of > 1 used to divide the logits, thereby decreasing σn which has the downstream effect of increasing the minimum entropy bound of the model. This can (for smaller label class sizes) significantly improve calibration, the initial model has the potential for over-confidence which temperature can serve to mitigate. Similarly, the use logit softcapping, as the Gemma2 family of models, applies a similar change dhe reducing the lower bound and thereby enabling models of smaller
When the vocabulary V is large, the final cross-entropy begins to consume significant amount of memory, as a N X V matrix must materialized to compute the loss. While optimizations have been implemented to manage other model components (Dao al., 2022; Dao, 2024), this logit matrix begins to constitute the vast majority of this foot- print when the vocabulary is large (Grattafiori et al., 2024; Gemma Team, 2024; Yang et al., 2024). In this section, we introduce an efficient smoothed cross-entropy computation, with its core implementation and results that demonstrate its effectiveness.
5.1. Limitations of Existing Solutions
Efficient implementation cross-entropy computation (Wi- jmans et al., 2025; Hsu al., 2024; PyTorch, 2024) often
6

---

Calibrated Large Language Models and How to Find Them with Label Smoothing
makes use of the fact that for a non-smoothed loss, comput- ing the entire legit matrix is unnecessary. Instead, only the relevant row from the classifier head C' is needed, as this is the only raw logit value that is relevant to the loss. Such approaches therefore store into an output only this single logit, reducing overhead by factor of nearly [V]. However, this is not feasible for smoothed losses. Reusing notation from Section 3, consider an LS loss on a sequence
Algorithm 1 Memory-efficient forward pass
Inputs:
E E DER D C E PERxxV, x E R.
Block sizes Ng. VB, and DB.
Smoothing / E [0,1].
Outputs:
LSE = log E, exp C. E) E E
0 0= (1 - 3) (C E,),, + MV E. C. E.RY
8
LSE, 0 = -O0,, ON
for all pairs of blocks Ens Cu. nn do
N
N
Anr = OVgx.Ng for blocks En,d. Cu,d do
LLS-
CLS 2
CI
(1-
(1)Tareet Loss
Anr += C. Ce Endd
3
Σ DEV
C E, - log Σ exp (C. Ei)
(15)
M = (Cen.d == Cv,d)
+
if M + ON V then
VEV
On +- (1 - B) Σ (CT J M
Bndd)
(2)Smoothing Loss
(3)LSE
end if end for
(2) in the above equation indicates that it must be the case that all logits need to be explicitly added to the loss, leaving existing solutions unfeasible for this specific setting. How- ever, we demonstrate that
LSE = log > exp (ATee)
LSE,, = log (exp (LSE,) + exp (LSE,.))
indeed compute these in manner that limits memory consumption usage, without influencing the throughput and applicability of the proposed solution to other scenarios such as no label smoothing.
if 3 + 0 then
On += V A
end if
end for
1. FORWARD PASS
where blocks along the same N dimension range but differ- ent [V] dimension range are written in the same location, in order to reduce memory usage on SRAM. This is im- plemented directly using a lock mechanism, where blocks exchange a single lock per group and update the LSE online.
To naively compute a CE loss as in Equation (15), one could individually compute each loss component. However, notice that many of these computations can be reused, such as the computation of C. E, in both (2) and (3). As such, for more efficient computation, we can compute all components in parallel in on-chip shared memory (SRAM) while making the best use of the GPU cache structure.
Similar to the above, if the label smoothing parameter / is not 0 (indicating that smoothing is used). (2) in Equa- tion (15) can be computed through reuse of intermediate results for (3), by summing over the vocabulary dimension m of Owm and updating an output o E IR-N with the result.
5.2. Implementation
Our forward pass (Algorithm 1) fuses the computation of all three components of Equation (15) to minimize memory and indexing costs, rendering the computation efficient on modern GPUs (Kerr et al., 2017). First, the embeddings E and classifier C are divided into chunks E, of E with size D x NB and C.,, of C with size D VB, operated independently. The standard output O = C E E RIVIXN XN is divided into blocks of size VB x NB which store the products One = CTT, En. Om and E, are further split along the D dimension into d chunks of size DB- Thus chunks En,a and C.n.d of size Dn x Nn and DB x Vn can be used to accumulate On = Ea C,,EE. En.d directly in SRAM before being written into global memory.
To understand how to fuse the computation of (1) into this matrix multiplication, we can first consider that the input sequence E also be split into chunks 2, of size NB- Because Dn contains information about the target labels, on can be directly used to compute the memory addresses of the target classifier chunk Cz.dd Because content from an address for each Cmn,d must be loaded into SRAM to compute Onvs this means that a direct address comparison can be used to create a mask M E NR which has value only where the rows in Cz,,d matches Cm,,d and 0 otherwise. This enables to add (1) to o by only adding the label-corresponding rows to the loss
To compute (3) or the LSE, the above strategy is sufficient by having each block first compute a matrix multiplication, then the log-sum-exp along the vocabulary dimension m for its block, and finally update the LSE with this result. We make use of trick introduced by Wijmans al. (2025),
o, += (1 - 3) Σ [(BTdCCmd) O OM, TN
(16)
where @ is the element-wise matrix multiplication.
Thus for inference, we can directly compute the entire loss
7

---

Calibrated Large Language Models and How to Find Them with Label Smoothing
Table 1. Results of different models w/ or w/o I.S on different datasets. All models are performed with 5-shot evaluation. We report the reliability of models with expected calibration error (ECE) and root mean square calibration error (RMS).
SFT Dataset
Model
MMLU
HELL
ARC-EASY
Acc.
FCB.
RMS
Acc. f
ECR
RMS
Acc.
ECE 4
RMS 4
Mistral-7B+ (3 =
0.579
0.134
0.120
0.302
0.127
0.160
0.803
0.099
0.154
Mistral-7B+ (33 = 0.1)
0.590
0.094
0.104
0.304
0.087
0.124
0.806
0.071
0.131
Alpaca
LLaMA3-8B+ + SFT (3 0)
0.638
0.113
0.113
0.375
0.162
0.085
0.863
0.070
0.127
LLMMA38BB + SIT (3 0.1)
0.636
0.073
0.094
0.374
0.087
0.037
0.864
0.037
0.098
Gemma2-2B + SFT (8 = 0)
0.528
0.343
0.180
0.302
0.127
0.160
0.773
0.131
0.174
Gemma2-2B + SFT (8 = 0.1)
0.532
0.125
0.121
0.304
0.087
0.124
0.764
0.069
0.127
Mistral-7B SFT (5 = 0)
0.600
0.096
0.105
0.369
0.044
0.085
0.843
0.078
0.135
Mistral-7B + SFT
SFT (3 = 0.1)
0.603
0.028
0.071
0.375
0.021
0.067
0.840
0.030
0.094
Tulu3Mixture
LLaMA3-8B- + SFT (3 Q)
0.651
0.050
0,080
0
0.049
0.091
0.857
0.058
0.114
LLaMA3- 8B + SFT (3 =3=00 0.1)
0.646
0.012
0.061
0.356
0.025
0.064
0.858
0.035
0.097
Gemma2-2B- + SFT (3 0)
0.533
0.341
0.177
0.273
0,082
0.128
0.758
0,086
0.142
Cemma2-2B + SFT (3 0.1)
0.531
0.020
0.064
0.271
0.041
0.087
0.755
0.029
0.101
Mistral-7B+ SIT (3 = 0)
0.602
0.071
0.094
0.546
0.041
0.071
0.867
0.066
0.100
Mistral-7B SFT (A = 0.1)
0.602
0.014
0.059
0.552
0.021
0.042
0.857
0.036
0.076
Openifernes
LLaMA3-8B + SFT (3= 0)
0.654
0.038
0.077
0.552
0.063
0.074
0.880
0.065
0.112
LLaMA3-88 1BA-----+44 + SFT (33 = 0.1)
0.646
0.016
0.059
0.554
0.038
0.037
0.880
0.041
0.089
Genme2-2B + SFT (3 0))
0.541
0.353
0.180
0.364
0.125
0.143
0.816
0.131
0.175
Genma2-22 2B + SFT (3 0.13
0.542
0.016
0.063
0.362
0.077
0.096
0.813
0.038
0.096
efficiently without directly needing to materialize the com- plete logit matrix in global memory.
5.3. Experiments
Setup. For all models trained on the Alpaca (Dubois et al., 2023), Tulu3Mixture (Wang et al., 2023) and OpenHermes (Teknium, 2023) SFT datasets, we adhere to the recommended training configuration outlined by Wang et al. (2023). We employ the AdamW optimizer for training and conduct grid search over the learning rates (5e 6, 2e - 5, 5e 5, 2e 4] to determine the optimal setting for each model. To facilitate stable training and pre- vent over-fittin, we use a batch size of 128 and apply a dropout rate 0.1.
5.2.1. BACKWARD PASS
To implement a backward pass. borrowing logic from Wij- mans et al. (2025) is sufficient produce backward pass that is significantly faster than existing alternatives. As Wi- jmans et al. (2025) do not consider label smoothing, their backward implementation requires some modification. As the gradient with respect to LSE is
LSE
E- ((softmax C E). VELLSE))) T C
C- ((softmax(C" E)- V LSE) E) T (17) Cis
V LSE
Calibration Results. Table I provides a comprehensive comparison of the accuracy and calibration performance of various large language models (LLMs) with and with- out label smoothing (LS) across different supervised fine- tuning (SFT) datasets. The evaluation is conducted on three widely used benchmark datasets: MMLU, HellaSwag, and ARC-Easy, ensuring robust assessment of model perfor- mance. Across all experimental settings, applying LS with a smoothing factor of 3 = 0.1 consistently leads to improved calibration, as indicated by lower Expected Calibration Er- ror (ECE) and Root Mean Square (RMS) calibration er- ror, while preserving model accuracy. Notably, LLaMA3-B8 and Mistral-7B achieve the best calibration results, par- ticularly when trained on OpenHermes and Tulu3Mixture These findings underscore the effectiveness of LS as a sim- ple yet powerful technique for enhancing model reliability without sacrificing predictive accuracy.
incorporating the derivative with respect to (1) only requires adding constant to sof tmfx E) depending on whether the element is the true label or not.
While the forward and backward pass can be further fused to compute both the output and gradient simultaneously, this requires computing LSE,, before any sof tmax value S.e. otherwise the computed S.e will use an incomplete log-sum-exp scaling factor (each row S,, depends on LSE,, to have been computed in its entirety). This thus requires a barrier to block subsequent execution code until all pre- ceding computations have been completed by all workers. While can save memory, we observe that specific tricks such as average logit sorting and gradient filtering, introduced by Wijmans et al. (2025), enable faster computation through standalone forward and backward passes with minimal in- creased memory utilization (2MB overhead).
8

---

Calibrated Large Language Models and How to Find Them with Label Smoothing
Grad Norm
Table Memory and time compute losses and gradients. Re- sults are computed on a batch size of 8192 tokens in single sequence, generated from Gemma2-26 model (vocabulary size of 256K and hidden size 2304). Experiments are conducted using PyTorch 2.4.0 and CUDA 12.1. Further see Table in Appendix B.
Method
fod
bed
rud-bed
Memory
Time
Memory
Time
Memory
Time
Smoothing
Ours
1.1 MB
24.2 ms
1J
ms
1J16MMB
72.9 ms
torch. compile
4.000 MB
22.8 12,000
38.3
16,000
62.3
Baseline
24,000 MB
ms
16,000
62.5 ms
28, 28,000 MB
101.9 ms
Smoothing
Ours
1.1 MB
24.0
1,I63MB
49.2
LJ664MB
72.9 ms
Cut Cross Entropy'
1.1 MB
23.6
1.63MB
49.2 ms
1,16 LB
72.4
torch, coupile
4,000 MB
20.6 ms
4,000 MB
33.9 ms
8,000 MB
55.0 ms
Baseline
2B 38.7 mus
1B
55.8
MB
96.0 ms
200
400 Step
600
800
5.4. Benchmarking and Testing
Method
Baseline
Ours
Benchmarking. Table 2/3 provides the primary compari- son of our custom kernel against existing alternatives, both with and without label smoothing. We compare in terms of both memory and time. First, in terms of memory, mea- sured as the peak amount of GPU storage necessary for the computation, our kernels surpass other options, beating the next closest competitor (Liger Kernels) by requiring over 75% less memory. Time-wise, we trail a compiled torch implementation by less than 10 ms while utilizing only R87% the amount of memory as they use. Furthermore, our method can be applied in more general settings. Under such circumstances, we have two observations. First, we can match Cut-Cross-Entropy (CCE) with a very minimal increase in computation speed (less than 0.5% overhead for a forward pass and less than 1% for a combined for- ward/backward). Furthermore, while an efficient compiled torch implementation remains slightly more efficient, la- bel smoothing causes an increase running time as well as memory consumption for this specific implementation, factors that not observed with our method. Accordingly, we can conclude that overall, our method provides a more robust and efficient alternative.
Figure 6. Training curves comparing our implementation against torch. nn. CrossEntropyLoss with / = 0.1. Training uses LLaMA3 2-3B model using the Alpaca dataset.
calibration after the supervised fine-tuning process (SFT) that is commonly used train LLMs. We first identify why SFT can degrade calibration from statistical standpoint before showing how label smoothing can help alleviate this concern. However, we also identify setting where the use label smoothing appears ineffective, particularly in the case of large language models with large vocabularies but smaller hidden sizes, demonstrating how such model construction implicitly impacts the entropy of predictions and leads to difficulty in becoming over-confident during SFT. Finally, we demonstrate practical issue regarding the computational concerns of label smoothing in these settings. To further address this issue, we identify GPU accelerator level optimizations and provide a custom computational kernel, written in Triton, that enables us to maximize the use of accelerator memory/bandwidth and train models, improving both training/inference speed as well as memory consumption without sacrifices in stability.
Training Results. We further compare models trained with our custom kernel and to those using a base- line/reference torch implementation in Figure 6. We demonstrate here that both methods are indistinguishable in terms of loss curves as well as gradient norms, indicating the correctness and stability of the kernel.
Acknowledgements
Jerry Huang was supported by the NSERC Canada Graduate Scholarships Doctoral (CGS-D) program (funding refer- ence number 589326) as well as the Bourse d' d'Éxcellence Hydro-Québec program. This work was made possible in part thanks to computational resources from Calcul Québec² and the Digital Research Alliance of Canada (DRAC)3.
6. Conclusion
In this paper, we present a novel perspective regarding the use of label smoothing as functional mechanism retain
hnttps://ww..n ///ps.////www.ee wwwp://wwwww.lluuu.........mmc ca/ 'nttps://alliaaceess ca/en
"This method does not support label smoothing.
9

---

Calibrated Large Language Models and How to Find Them with Label Smoothing
Impact Statement
DeGroot, M. H. and Fienberg, S. E. The comparison and evaluation of forecasters. Journal of the Royal Statistical Society, Series D (The Statistician), 32(1): 1983.
This paper proposes a method to improve calibration in uuge-vocabulary language models. We anticipate minimal societal impact from this work.
Dubois, Y., Li, C. X., Taori, R., Zhang, T., Gulrajani, I., Ba, J., Guestrin, C., Liang, P., and Hashimoto, T. B. Al- pacafarm: A simulation framework for methods that learn from human feedback. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural In- formation Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023.
References
Bach, S. H., Sanh, V. Yong, Z. X., Webson, A., Raffel, C.,
Nayak, N. V., Sharma, A., Kim, T., Bari, M. S., Févry, T.,
Alyafeai, Z., Dey, M., Santilli, A., Sun, Z., Ben-David, S., Xu, C.. Chhablani, G., Wang, H., Fries, J. A., Al- Shaibani, M. S., Sharma, S., Thakker, U., Almubarak, K., Tang, X., Radev, D. R., Jiang, M. T., and Rush, A. M. Promptsource: An integrated development environment and repository for natural language prompts. In Proceed- ings of the 60th Annual Meeting of the Association for Computational Linguistics, ACL 2022 System Demon- strations, Dublin, Ireland, May 22-27, 2022, pp. 93-104, 2022.
Gemma Team. Gemma 2: Improving open language models at a practical size, 2024.
Grattafiori, A. et The Ilama herd of models, 2024.
Guo, C.,, Pleiss, G., Sun, Y., and Weinberger, K. Q. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pp. 1321 1321-1330, 2017.
Bertsekas, D. Nonlinear Programming. 1999.
Brier, G. W. Verification of forecasts expressed in terms of probability. Monthly Weather Review, 78(1):1-3, 1950.
Hendrycks, D., Mazeika, M., and Dietterich, T. G. Deep anomaly detection with outlier exposure. In 7th Inter- national Conference Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, 2019.
Chi, T., Fan, T., and Rudnicky, A. Attention alignment and flexible positional embeddings improve transformer length extrapolation. In Findings of the Association for Computationall Linguistics: NAACL 2024, Mexico City, Mexico, June 16-21, 2024, pp. 132-148, 2024.
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, 2021.
Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S.,
Webson, A., Gu. S. S., Dai, Z., Suzgun, M.. Chen, X.,
Hsu, P.-L., Dai, Y., Kothapalli, V., Song, Q. Tang, S., Zhu, S., Shimizu, S., Sahni, S., Ning, H., and Chen, Y. Liger kernel: Efficient triton kernels for Ilm training, 2024.
Chowdhery, A., Castro-Ros, A., Pellat, M., Robinson, K., Valter, D., Narang, S., Mishra, G., Yu, A., Zhao, V. Y., Huang, Y., Dai, A. M., Yu, H., Petrov, S., Chi, E. H., Dean, J., Devlin, J., Roberts, A., Zhou, D., Le, Q. V., and Wei, J. Scaling iistructiin-finetune language models. Journal on Machine Learning Research, 25:70:1-70:53, 2024.
Huh, M., Cheung, B., Wang, T., and Isola, P. Position: The platonic representation hypothesis. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024, 2024.
Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., de Las Casas, D., Bressand, F,, Lengyel, G., Lample, G., Saulnier, L.., Lavaud, L. R., Lachaux, M., Stock, P., Scao, T. L., Lavril, T., Wang, T., Lacroix, T., and Sayed, W. E. Mistral 7b, 2023.
Dao. T. Flashattention-2: Faster attention with better paral- lelism and work partitioning. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vi- enna, Austria, May 7-11, 1,111.221 2024, 2024.
Dao, T., Fu, D. Y., Ermon, S., Rudra, A., and Ré, C. Flashat- tention: Fast and memory-effiiien exact attention with io-awareness. In Advances in Neural Information Pro- cessing Systems 35: Annual Conference on Neural In- formation Processing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28- December 9, 2022, 2022.
Jiang, Z., Araki, J., Ding, H., and Neubig, G. How know When language models know? on the calibration of language models for question answering. Trans. Assoc. Comput. Linguistics, 9:962-977, 2021.
Kerr, A., Merrill, D., Demouth, J., and Tran, J. Cutlass: Fast linear algebra in cuda c++, December 2017.
10

---

Calibrated Large Language Models and How to Find Them with Label Smoothing
Kumar, A., Raghunathan, A., Jones, R. M., Ma, T., and Liang, P. Fine-tuning can distort pretrained features and underperform out-of-distribution. In The Tenth Interna- tional Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022, 2022.
Murphy, A. H. Scalar and vector partitions of the probability score: Part i. two-state situation. Journal of Applied Meteorology and Climatology, 11(2):77-2822, 1972.
Naeini, M. P., Cooper, G. F., and Hauskrecht, M. Obtaining well calibrated probabilities using bayesian binning. In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, January 25-30, 2015, Austin, Texas, USA, pp. 2901-2907, 2015.
Lin, T., Goyal, P., Girshick, R. B., He, K., and Dollár, P. Fo- cal loss for dense object detection. In IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017. pp. 2999-3007, 2017.
Nixon, J., Dusenberry, M. W., Zhang, L.,, Jerfel, G., and Tran, D. Measuring calibration in deep learning. In IEEE Conference on Computer Vision and Pattern Recognition Workshops, CVPR Workshops 2019, Long Beach, CA, USA, June 16-20, 2019, pp. 38 41, 2019.
Liu, B., Ayed, I. B., Galdran, A., and Dolz, J. The devil is in the margin: Margin-based label smoothing for network calibration. In IEEF/CVF Conference on Computer Vi- sion and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pp. 30-88, 2022.
Oh, C., Lim, H., Kim, M., Han, D., Yun, S., Choo, J., Hauptmann, A. G., Cheng, Z.-Q., and Song, K. Towards calibrated robust fine-tuning of vision-language models. In Advances in Neural Information Processing Systems 38: Annual Conference Neural Information Processing Systems 2024, NeurlPS 2024, December 10-15, 2024, Vancouver, BC, Canada, 2024.
Lu, P.. Rashid, A., Kobyzev, I., Rezagholizadeh, M., and Langlais, P. LABO: towards learning optimal label reg- ularization via bi-level optimization. In Findings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada, July 9-14, 2023, pp. 5759-5774, 2023.
Lukasik, M., Bhojanapalli, S., Menon, A. K., and Kumar, S. Does label smoothing mitigate label noise? In Pro- ceedings of the 37th International Conference on Ma- chine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 6448-6458, 2020.
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P. F. Leike, J., and Lowe, R. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28. December 9, 2022, 2022.
Minderer, M., Djolonga, J., Romijnders, R., Hubis, F., Zhai, X., Houlsby, N., Tran, D., and Lucie, M. Revisiting the calibration of modern neural networks. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pp. 35682-15694, 2021.
Pereyra, G.,, Tucker, G., Chorowski, J., Kaiser, L.,, and Hin- ton, G. E. Regularizing neural networks by penalizing confident output distributions. In 5th International Con- ference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Workshop Track Proceedings, 2017.
Mukhoti, J., Kulharia, V., Sanyal, A., Golodetz, S., Torr, P. H. S., and Dokania, P. K. Calibrating deep neural networks using focal loss. In Advances in Neural Infor- mation Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
PyTorc. torchtune: Pytorch's finetuning library, April 2024.
Sanh, V., Webson, A., Raffel, C., Bach, S. H., Sutawika,
Mukhoti, J., Gal, Y., Torr, P., and Dokania, K. Fine-tuning can cripple your foundation model; preserving features may be the solution. Transactions on Machine Learning Research, 2024, 2024.
L., Alyafeai, Z., Chaffin, A., Stiegler, A., Raja, A., Dey, M., Bari, M. S., Xu, C., Thakker, U., Sharma, S. S., Szezechla, E., Kim, T., Chhablani, G., Nayak, N. V., Datta, D., Chang. J., Jiang, M. T., Wang. H., Manica, M.. Shen, S., Yong, Z. X., Pandey, H., Bawden, R. Wang, T., Neeraj. T., Rozen, J., Sharma, A., Santilli, A., Févry, T., Fries, J. A., Teehan, R., Scao, T. L.,, Biderman, S., Gao, L., Wolf, T., and Rush A. M. Multitask prompted train- ing enables zero-shot task generalization. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022, 2022.
Müller, R., Korbllith, S., and Hinton, G. E. When does label smoothing help? In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 4696- 4705, 2019.

---

Calibrated Large Language Models and How to Find Them with Label Smoothing
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., and Wojna, Z. Rethinking the inception architecture for computer vision. In 2016 IEEE Conference Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 2818 2826,, 2016.
Yang, A., Yang. B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., al. Qwen2.5 technical report, 2024.
Zhai, S., Likhomanenko, T., Littwin, E., Busbridge, D., Ramapuram, J.. Zhang, Y., Gu, J., and Susskind, J. M. Stabilizing transformer training by preventing attention entropy collapse. In International Conference Ma- chine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 40770-40803, 2023.
Teknium. Openhermes 2.5: An open dataset of synthetic data for generalist Ilm assistants, 2023.
Wang, Y. Mishra, S., Alipoormolabashi, P., Kordi, Y.,
Mirzaei, A., Naik, A., Ashok, A., Dhanasekaran, A. S., Arunkumar, A., Stap, D., Pathak, E., Karamanolakis, G., Lai, H. G., Purohit, I., Mondal, I., Anderson, J., Kuz- nia, K., Doshi, K., Pal, K. K., Patel, M., Moradshahi, M., Parmar, M., Purohit, M., Varshney, N., Kaza, R., Verma, P., Puri, S., Karia, R., Doshi. S., Sampat, K., Mishra, S., A, S. R., Patro, S., Dixit, T., and Shen, X. huperr-aatraaiisttuuiiosss Generalization via declarative instructions on 1600+ NLP tasks. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022 pp. 5085-5109, 2022.
Zhang, Z. and Sabuncu, M. R. Self-distillation instance- specific label smoothing. In Advances in Neural Infor- mation Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6- 12, 2020, virtual, 2020.
Zhao, Z.,, Wallace, E., Feng, S., Klein, D., and Singh, S. Calibrate before use: Improving few-shot performance of language models. In Proceedings of the 38th Inter- national Conference on Machine Learning, ICML 18-24 July 2021, Virtual Event, volume 139 of Proceed- ings of Machine Learning Research, pp. 12697-12706. 2021.
Wang. Y., Ivison, H., Dasigi, P., Hessel, J., Khot, T., Chandu, K. R., Wadden, D., MacMillan, K., Smith, N. A., Beltagy, I., and Hajishirzi. H. How far can camels go? exploring the state of instruction tuning on open resources. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 - 16, 2023, 2023.
Lukasz Rajkowski. Analysis of the Maximal a Posteri- ori Partition in the Gaussian Dirichlet Process Mixture Model. Bayesian Analysis, 14(2):477 -494, 2019.
Wei, J., Bosma, M., Zhao, V. Y., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., and Lc, Q. V. Finetuned language models are zero-shot learners. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022, 2022a.
Wei, J., Liu, H., Liu, T., Niu, G., Sugiyama, M., and Liu, Y. To smooth or not? when label smoothing meets noisy labels. In International Conference on Machine Learn- ing, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 21389123614,, 2022b.
Wijmans, E.,, Huval, B., Hertzberg, A., Koltun, V., and Krähenbühl, P. Cut your losses in large-vocabulas lan- guage models. In The Thirteenth International Confer- ence on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025, 2025.
Xiong, M., Hu, Z., Lu, X., Li, Y., Fu, J., He, J., and Hooi, B. Can 1ims express their uncertainty? an empirical evaluation of confidence elicitation in lmss. In The Twelfth International Conference on Learning Represemations, ICLR 2024, Vienna, Austria, May 7-11, 2024, 2024.
12

---

Calibrated Large Language Models and How to Find Them with Label Smoothing
A. Proofs
A.I. Proof Proposition 3.3
Proposition 3.3. A linear penalty (or Lagrangian term) for the hard constraint d(x) is bounded from above and below by KL (ullō (x; 0)), up additive constants
K /K S KL [ullōr (a;; 0)]
KL[u|lö (z; 0)] - log K < d (xx)
Σ
(18)
==1
Proof. We adapt the proof of Liu et al. (2022). Given the KL divergence
K
1
KL jullo (æ; 0)
log
P (?4 xx; 0) + const
K
we have that
KL. [ullr (a:; 0)]
1
Σ
log
j= ((2:0), const
((=:0);
K
(19)
log
e(==0), l (x:
+ const
K
Considering the property of the LogSumExp (L.SE) function, follows that
K
max l (x: ,) ; I log (a:0), & max e (xx;), , log (K)
j=l
and
KL [ullà (a:; 0)] log K S
1 K
Σ max l (x; (z:00) s KL [ullō (æ: 0)]
(20)
and given the definition of d (x), then the additional penalty KT. [ul ô (x:; 0)] imposed by LS addition to the standard cross-entropy loss _CB is approximately optimizing linear penalty (or Lagrangian) for the constraint
d (z) = 0
encourage equality of the logits.
A.2. Proof of Proposition 3.4
Proposition 3.4. Define a likelihood model p (yla: 0) = Cat (softmax (ʃ (x;; 0))) a categorical distribution with parame- ters z softmax (f (æ; 0)) E A(O) where A (0) denotes a probability simplex over the parameter space θ. The label smoothing objective is equivalent to Maximum Posteriori (MAP) estimation on the softmax probability vector under the independence assumption p (2(xx) (2).
Proof. This proof is an adaptation of Zhang & Sabuncu (2020). Suppose a provide set of examples D = {( 9n) W sampled from X' y. The goal to find set of parameters corresponding labels y e y. Suppose that the likelihood p (y) az =) = Cat(z) be a categorical distribution with parameter è (+) and the conditional prior p (z|æ) Dir (axz) be Dirichlet distribution with instance-specific parameter az.
parameterize a function ʃ that maps inputs æ E to
C,tas,
Due to conjugacy of the Dirichlet prior, a closed-form solution 2, = Σ, C, C,taa -1 where C; corresponds to number of occurrences of the i-th category, can easily obtained.
13

---

Calibrated Large Language Models and How to Find Them with Label Smoothing
Thus the MAP estimation 2, softmax (Sw (ax)) can be amortized with given training set, resulting in an optimization problem of:
1 logp p (z xnn Mnii 0, az)
max
(21)
max
log p (y yn xx: + log (z|an 0, ax))
==1
N
N
K
max
log [softmax (ʃ (I0 0))1 y/n
(Lown)k 1) logl =|
(22)
N
1=1
n=1 Km
Cross Entropy
opsecice-specffic Regularization
where H& denotes the k-th element of a vector. Using the assumption that p (z| = -p(2,, sensible choice of prior would
be a uniform distribution across all possible labels. Choosing [<kz] = |el| Eh2 + 1 for all k E {l...., K} for some hyper-parameter 3, the MAP objective becomes
N
K
LLS
log pm +
[og[=|k-
(23)
N
K
7=1
#=1
With some simple rearrangement of terms,
N LLS
N K ΣΣ Σ
3
log/z|y. yo +
log|z|4
N
N
K
i=1
n=1
km
N
(1
+ 3)
K + 3 K(1+B) B)
8
Σ
log log|z| +
Σ
log[z)K
N
K(1+ B)
Kzyn
Thus the above objective is equivalent to the label smoothing regularization with 1 *(1+)) up to a constant factor of
(1 + $).
A.3. Proof of Lemma 4.1
Lemma 4.1. Let C' E PV be matrix with |C oc, and let h E RD be vector such that each entry of h satisfies
hhl S ohe The upper bound for |Chlll is:
[Chl22 S oc oh VD.
Proof. For any vector R,P, follows that:
Chhl, I |[ ||1l|
Substituting |[C|]2 ac, we obtain:
[Chll2 s 0c |11ll2-
And
D
| <
Σ
oo = oh. on.
Substituting the bound on ||hl| into the inequality for ||Ch/l) we have the norm of logit vector u E RR::
|u l2 ||Ch|l S oc | S oc VD The
14

---

Calibrated Large Language Models and How to Find Them with Label Smoothing
A.4. Proof of Theorem 4.2
Proof. We begin our analysis of the entropy of the prediction distribution of LM head p JR
exp(us)
Pt exp(x)(
(24)
The entropy of p is then:
H(p) Σ pi log(p:).
(25)
Without loss generality, we assume ||u||2 =S pv D. where p = 0.0%- Then we aim to address the following constrained optimization problem:
min H(u), s.t ||u||| S p p.p.vo VD
(26)
We derive the global minimum for it with Lagrangian multiplier and set the corresponding gradients equal to 0 then follow the analysis of Zhai et al. (2023);
C(u, A) H((u) + -| - p²D), dc(u, 2) =0,
(27)
ac(u, A) O.
(28)
Ou
0x
Then we get:
V
exp(u,)
exp(u:)
eex((u))
]
Au;- =
6is
1 1 log
(29)
Z
Z
Z
1=1
Pk [log(p)) + H(u)].
| = D²DD.
(30)
Assume that for the minimizer u" there exists an index such that uz 0, we have:
v
log(p:) + H(u)) = - Σ
Pj
log
Pj
=
Pj log(e")) = -uu.
(31)
==1
1=1
log(pm) + H((l))
log(pn) + H(uu
Vum # Un # 0, Pm
Pn
(32)
Um
Un
Eu
Eu
Pm
Pa
+
Pm = Pu-
Umn
Un
which contradict to |/u||
DD Instead, assume Vn Un + 0, based on Equation (30), we have:
Vun + Ums
Pm
[log(m)) + H(u)]]
Pre
[log(px) + H(u))
(33)
Um
Un
E (u)
= e (1 E (u)
(34)
thn
We could assume that a solution a must contain at least one negative component. To illustrate this, consider u where 24 > component-wise and ||u]|| < pvDD D. We can always shift u by a vector u, where 2m = 2, for all m, n, ensuring that ||u - a|| < PvD D and that u v has at least one negative component. Since all components shift-invariant, follows that softmax(u)
are equal and softmax
softmax(u u v). Additionally, without loss of generality, we assume Eu based the same reasoning. Let Um, thn 0, then based on Equation (34):
(1
E(u)
EE(1) Un
E
> 0
(35)
Um
15

---

Calibrated Large Language Models and How to Find Them with Label Smoothing
As [(x) e" (1 = is monotonously increasing in a E (-oo, 0) and [ax, o)) for 0, it is easy Umm Un- If Uh 0 and um 0, then Umm Eu. As f()) monotonous in for both and O, have solution that must have unique values, one positive and one negative. Let the different elements be b. The minimizer contains
V 1 and one a. Based on || p² D, we have:
PVD
V b= PVD
(36)
VI(V - 1)
The corresponding entropy of the minimizer u:
log 1+ - exp
(-OV
V
DV
PVDIVI cxp
1+ ([ - 1) exp (-PIV|-)) 1V]
(-PV
VV-1
IV
(37)
16

---

Calibrated Large Language Models and How to Find Them with Label Smoothing
B. Additional Results
B.1. Calibration Metries
LLaMA3-8B
10.0
7.5
5.0
2.5
0.0
RMS
ECE
SCE
ACE
TACE
Avg. CE
LLaMA3.2-1B
10.0
7.5
Value
0.0
RMS
ECE
SCE
ACE
TACE
CE
LLaMA3.2-3B
10.0
7.5
5.0
2.5
RMS
ECE
SCE
ACE
TACE
Avg. CE
Metric
Smoothing
0.0
0.1
0.2
Figure 7. Calibration different LLaMA3 models fine-tune tuned on the same SFT dataset. As the size the model decreases, the calibration of the model sees less improvement from the use of LS.
17

---

Calibrated Large Language Models and How to Find Them with Label Smoothing
B.2. Reliability Diagrams
Namess: Gerima-38
Names: Gemma-22
Narmee: Cemmas-2
Namnes: Germmna-28
Names: Gemmma-25
Smoothing:
Smoothing:
Smoothing:
Smootthngg
0.75
200000 loakk
6.75
0.75
Underoomice
0.75
0.75
0.25
0.28
0.28
0.00
Overoonlidem
0.00
Ovenco Widden
0.00
Overoenideent
0.00
Overoont Ident
0.00 Osercorra Fident
0.00
0.25
0.50
0.75 1.00
0.00 0.25
0.90
0.76 1.50
3.00
0.25
0.50
0.75 1.00
0.00
3.25
0.50
0.75 1.00
0.00 0:85
0.00
0.75 1.00
Names: Gemma2-28 28
Names: Gemme2-28
Names: Gemme2-28
2B
Names: Gemmes -28
Narnes: Gemma2 2-
Smoothing: 0.0
Smoothing: 0.1
Smoothin: 0.2
Smoothing: 0.3
Smoothing: 0.4
1.00
1.00
1.00
1.00
1.00
Undersseee
0.50
0.50
0.50
0:50
0.25
0.28
Overcmntiden
0.00
Quenca-fidenn
Overceniident
Overoont Iident
Overcor
0.00
0.25
0.50
0.75
0.00
0.25
0.76 1.50
0:00
0.25
0.50
1.00
3.20 0.50
0.00 0:25 0.50 0.76 1.00
Namee: LLaMA0-78
Names:
LLaMA2-78
Names: LLaMA2-78
Names: LLaMA2-7B
Namee: LLaMA2-78
Smoothing:
Smoothing:
Smoothhng:
Smoothing:
Smoothing:
Undee Dnsteroonic comeeree
0.75
0.75
0.75
0.75
0.75
0.50
0.50
0.25
0.28
0.25
0.00
Opensonlidee
0.00
Ouenco
0.00
Oveiceniideet
0.00
Qvero Idient
0.00
Osercor Ficket
0.00
0.25
0.50
0.75 1:00
0.00
0.25
0.50
0.75 1.50
0.00
0.25
0.50
0.75 1.00
0.00
3.25
0.50 0.75 1.00
0.00 0
0.75 1.00
Nomes: LL LLaMA3-BB
Names: LLaMA3-BB
Names: LLaMA3-8B
Nomes: LL LLaMA3-BB
Nomes: LLMMA3-8B
Smoothing: 0.0
Smoothing: 0.1
Smoothing: 0.2
Smoothing: 0.3
Smoothing! 0.4
1.00
1.00
1.00
1.00
1.00
Ancuracy
36
0.50
0.50
0:50
0.00
Querconffi Widden
Opwiconnliden
Overocnn lideent
Overcon
0.00 0.25 8.50 0.75 1:00
0.00 0.25
0.50
8.75 1.50
0.00
0.25 0.50
1.00
3.25 0.50
0.00
0:85 0.50
0.76 1.00
Names: LLaMA3 2-18
Names: L-
Names:
LLaMA3
Names: LLMMAS
Namee: LLaMAS 2-1B
Smoothing:
Smoothing:
Sioothhing:
Smoothing: 0.3
Smoothing
Uileersinnee den
0.75
0.75
0.75
0.75
0.75
0.50
0.50
0.26
0.00
Opensellitee
0.00
Querco
0.00
Opstcainiddent
0.00
0.00
Oserro Ficknt
0.00
0.25
0.50 0.75 1.00
0.00
0.25
0.50
0.75 1,50
9.00
0.25
0.50
0.75 1.00
0.00
3.28
0.50
0.75 1.00
0.00 0
0.78 1.00
Names: LLMA332--B
Names: LLaMA3,2-SB 3B
Names:
LLaMA3.2-38
Names: LL LLaMAS 2:38
Names: LL 2-
Smoothing: 0.0
Smoothing 0.1
Smoothing: 0.2
Smoothing: 0.3
Smoothing: 0.4
1.00
1.00
1.00
1.03
1.00
U Bigerssnieee
0.50
0.50
0.50
0.50
0.85
0.26
0.85
0.25
0.25
Overcontide::
0.00
Deerccnfldont
Ove Overocntdantt
Overoont
Cverconfidem
0.00
0.25
3.00
6.75 1.00
0.25
0.50
8:75
1.50
0.00 0.35
0.50
1.00
3.56
0.50
0:75
0.00 0.85
0.50 0:75 1.00
Names: Mistral-7B
Names: Mistral-BB Smoothing:
Names: Mistral-7B Smootiing:
Names: Mistrai-7B Smoothing:
Names: Mistral-8B
Smoothing:
Smoolhing:
Piperroon dert
Uncercom dort
0.75
0.78
0.75
0.25
0.75
0.50
0.26
00
Oyerconleem::
0.00
Overco fiden
0.00
Ove pontidddn
0.00
Overoontident
0.00
Overcan Rident
0.00
0.25 6.50 6.75 1.00
0.00
0.25
0.50
0.75 1.50
3.00
0.25
0.50 0:75 1.00
0.00
3.58
0.50 0:25 1.00
0.00 0.2 0.50 1.00
Confidenoe
Smoothng
Figure& Reliability diagrams of models fine-tuned on the Tulu3 SFT Dataset and tested on the MMLU dataset.
18

---

Calibrated Large Language Models and How to Find Them with Label Smoothing
Namea: Gannrg 3B Saroofeeng 2X2
Natresi Gimma 20 Smoothing
Narraa Gamma 2B
Namra Gemmat 2B
Nartesi Gemrrg Seroudinng
Narva Gemmas Smosthieg
Simothing: 0.0
1.00
1.00
1.00
1.00
1.23
0.25
0.80
E30
0.30
699
0.06
635
600
Ovanorr
Overariderr
Dvancoll
430
1arres: LLa7ARo-15
Namms: LL.aMA02-71
Narree LLA43R2-15
Namar 14
harrea ao: LLaWA33-45
Narner: LLaM633-0
1.00
1.00
1.22
1.0
0.75
0.70
UNW 201
Underonéon
0-15
622
0100
Edam
Over:3ndder
canddrr
Wiarr
Dvancop-
Namer: 2-11
Marre: 2-11
Namre: Li-MM4
Narre:: 11
Rames: 2M-
Narrec LLaMAa 2-03
Smoothin
Smeothng:
Smoothing:
1.00
1.00
1.20
0.7
077
0.75
6-73
Uster
0.25
E22
623
0.00
Ovaroe Adr
0:03
Ove Oapdor
Oeato F0drt
6:33
Namrer: Matraa-mt
Varne: Metrsi-111
Names MattaamT
Name:: Daard Se 50
Namee: Oower7 5c
Namar Dward 5-.1
0.7
C775
0.75
672
0.50
0.0
0.00
Ovanoe Adam
0.00
Owacariant
0.03
Ove voa
2.0
660
Oranteldees
Figure 9. Reliability diagrams of models fine-tuned on the OpenHermes-2... SFT Dataset and tested on the MMLU dataset.
B.3. Efficient Smoothed Cross-Entropy
B.3.1. TRAINING
Grad Norm
Value
Method
Baseline
Ours
Figure 10. Training curves for LLaMA3.2-1B.
19

---

Calibrated Large Language Models and How to Find Them with Label Smoothing
B.3.2. BENCHMARKING
Table 3. Memory footprint and time to compute losses and gradients. Results are computed on a batch size of 8192 tokens in a single sequence, generated from Gemma2 with billion parameters (vocabulary size of 256K and hidden size 2304). Experiments conducted on an H100-SXM5 GPU with 80GB of RAM, Py PyToch 2.4.0 and CUDA 12.1.
fud
bwd
fwd+bud
Method
Memory
Time
Memory
Time
Memory
Time
Lower Bound
0.004 MB
1,161 MB
1,161 MB
Smoothing
0
Ours
1.1 MB
24.2 ms
1,163 MB
49.3 ms
M,161MB
72.9 ms
(No Vocab Sorting)
0.09 MB
24.1 ms
1,162
62.8 ms
1,162
86.8 ms
+ (No Grad. Filter)
0.09 MB
24.0 ms
1,162 MB
177.1 ms
1,162 MB
201.5 ms
Fused Version
0.09 MB
100.5 ms
1,162 MB
0.2 ms
1,162 MB
100.6 ms
(No Grad. Filter)
0.09 MB
200.5 ms
1,162 MB
0.2 ms
1,162
200.5 ms
Liger Kernels (Hsu et al., 2024)*
NA
NA
NA
NA
5,349 MB
155.0 ms
torch. compile
4,000 MB
22.8 ms
12,000 MB
38.3 ms
16,000
62.3 ms
Baseline (torch. nn. Crosssssttttrtssssssss)
24,000 MB
41.4s
16,000 MB
62.5 ms
28,000 MB
104.9 ms
Smoothing / 0
Ours
1.1 MB
24.0
1,163 MB
49.2
1,164 MB
72.9
+ (No Vocab Sorting)
0.09 MB
23.9 ms
1,162 MB
62.4 ms
1,162 MB
85.2 ms
(No Grad. Filter)
0.09 MB
23.9 ms
1,162 MB
177.3 ms
1,162 MB
201.4 ms
Fused Version
0.09
100.5 ms
1,162 MB
0.2 ms
1,162 MB
100.6 ms
+ (No Grad. Filter)
0.09 MB
200.5 ms
1,162 MB
0.2 ms
1,162 MB
200.5 ms
Cut-Cross Entropy (Wijmans et al., 2025)*
1.1 MB
23.6 ms
1,163 MB
49.2 ms
1,164MB
72.4 ms
(No Vocab Sorting)
0.09 MB
23.5 ms
1,162 MB
62.5 ms
1,162 MB
85.0 ms
(No Grad. Filter)
0.09 MB
23.5 ms
1,162 MB
177.1 ms
1,162 MB
201.4 ms
Liger Kernels (Hsu et al., 2024)
NA
NA 2,000 MB
NA
5,349 MB
154.8 ms
Chunked Cross Entropy (torchtune) (PyTorch, 2024) (8 chunks)
13,000 MB
30.4 ms
51.0 ms
13,000 MB
82.8 ms
torch. compile
4,000 MB
20.6 ms
4,000 MB
33.9 ms
8,000 MB
55.0 ms
Baseline (torch, nn. CrossEEtrooyyLoss)
24,000 MB
38.7 ms
16,000 MB
55.8 ms
28,000 MB
96.0 ms
B.4. Experimental Details
Training. We conducted a learning rate sweep over learning rates [5e-6, 2e-5, 5e-5, 2e-4] with a summing reduction.
We further tested label smoothing syper-parameters [0.0, 0.2, 0.3, 0.4, 0.5), where 0.0 is no smoothing. We used the open-instruct repository at commit e363290 for our training setup," with modifications to account for our kernel well as specific experimental sypee-parammter' settings and baselines.
Evaluation. Our implementation is based on the MMLU official repository*. We first evaluate models on MMLU and then modify the files here to directly adapt the evaluation dataset the other tasks at hand. We follow MMLU and use the following prompt for all tasks: 'The following are multiple choice questions (with answers) about O 'y''n' format(
20


Reviews:
[{"review_id": "Jxp4bA0JFM", "reviewer": "Reviewer_fRKR", "summary": "This work focuses on mitigating mis-calibration of LLM during SFT by incorporating label smoothing. It argues that label smoothing helps reduce model overconfidence by promoting equality among logits while enhancing learned feature embeddings. Through empirical experiments, this work demonstrates the effectiveness of label smoothing. Additionally, it designs custom kernels to enable label smoothing with significant memory and throughput improvements, without compromising performance.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for providing a thorough evaluation of our work. We are delighted that the reviewer finds the work to have been positioned and set up in a manner that aligns with supporting our claims. We are hopeful that the below responses can sufficiently address their remaining concerns regarding our work as well as provide the necessary details to clarify possible misunderstandings resulting from their absence.\n\n-----\n\n> However, in subsequent arguments, this work only shows that label smoothing encourages equal logits, and it's unclear how this is connected with \"diversity of feature embeddings\" (which itself is not clearly defined in this work).\n\nIndeed, this section could be better detailed, which we may have overlooked as a result of the page limit in the initial submission.\n\nOh et al. (2024) provide a theorem (Theorem 3.1 of their work), which decomposes error into two components, a classification error (i.e. accuracy) and calibration error. They show both errors depend on the reciprocal of the smallest singular value of the input covariance matrix of the input $x$, with a smaller value leading to greater error. Mathematically, this means that the variance within the input features should be more diverse and independent.\n\nNext, our proposition 3.3 shows label smoothing to be a MAP estimation problem. Prior works have shown both empirically (Batmanghelich et al., 2014) and theoretically (Rajkowski, 2019) how this can lead to the learning of more diverse feature sets. First, Chi et al. (2024) show that the logits prior to the LM head are distributed as a Gaussian. Rajkowski (2019) proves that under the assumption that samples that are generated from a gaussian process, MAP estimation divides the data into clusters whose convex hulls are disjoint (Proposition 1 in their work) and into clusters of limited size (Proposition 2), leading to distinct and limited-sized clusters that make the features diverse and easy to distinguish (Corollary 3). Thus the model learns more diverse feature within the representations.\n\nIn light of these details, we are happy to provide additional reference and the associated lemmas and remarks within an updated manuscript.\n\n> If one looks at Figure 6 in the Appendix, it appears that Llama3-3B and 1B both are still over-confident.\n\nWe agree that this could be clarified. Our analysis in Sections 3/4 establish that smaller models have a higher lower entropy bound, which means that their predictions are bound to be less __concentrated__ than those of a larger model. However, this does not limit them to being under-confident or perfectly calibrated, as the lower bound could still indicate that the model has the potential to become overconfident. Thus we can see that as the model increases, so does overconfidence, a signal that the entropy bound could in fact be playing an influence on predictions. \n\n> Authors state that \"alternative techniques explicitly steer models toward overconfidence, allowing the benefits of label smoothing to once again emerge.\" However, no actual experiments/statistics are conducted to support this statement.\n\nWe provide clarification here. We can note the benefit of soft-capping directly from comparisons between Gemma and Gemma2. Figure 8 in Appendix B.2 shows that Gemma is naturally less prone to overconfidence compared to Gemma2, however the base models are both well calibrated. Both models are not publicly transparent about the raw pre-training data, however the fact that both are well calibrated initially prior to fine-tuning is an indication that both start at a roughly equivalent point/level. However, after SFT, Gemma2 models are naturally more overconfident than Gemma models, highlighting how the main difference between the two, the logit soft-capping, can naturally lead to overconfidence, which enables label smoothing to be more effective.\n\n------\n\nWe extend our gratitude towards the reviewer for their engagement with our work. We appreciate their feedback and are hopeful that the provided response above clarifies any remaining uncertainties to highlight the strengths of our work. We hope that the reviewer shares this impression and would be appreciative if such an opinion could be reflected through an improved assessment. We also remain ready to provide any additional details for questions that remain.\n\n------\n\n### References\n\n[1] Oh et al. Towards Calibrated Robust Fine-Tuning of Vision-Language Models. In NeurIPS, 2024.\n\n[2] Batmanghelich et al. Diversifying Sparsity Using Variational Determinantal Point Processes. In arXiv, 2014\n\n[3] Rajkowski, Łukasz. Analysis of the maximal posterior partition in the Dirichlet Process Gaussian Mixture Model. In Bayesian Analysis, 2019\n\n[4] Chi et al. Attention Alignment and Flexible Positional Embeddings Improve Transformer Length Extrapolation. In Findings of the Association on Computational Linguistics: NAACL-HLT, 2024", "claims_and_evidence": "1. This work claims that label smoothing can mitigate mis-calibration by encouraging \"diversity of feature embeddings\". However, in subsequent argument, this work only shows that label smoothing encourages equal logits, and it's unclear how this is connected with \"diversity of feature embeddings\" (which itself is not clearly defined in this work).\n\n2. This work claims that label smoothing is less effective in improving calibration for \"large vocabulary LLMs\" because \"large vocabulary LLMs\" do not have the ability to be overconfident. However, this cannot explain why label smoothing has huge improvement on Gemma2-2B  (and relatively small improvement on Llama3-3B/1B). Additionally, to support the claim, authors shall show that the calibration errors of Llama3-3B/1B are indeed smalled (or less over-confident) compared to Llama3-8B. However, these numbers are not provided. If one looks a Figure 6 in appendix, it appears that Llama3-3B and 1B both are still over-confident. \n\n3. In the introduction section, authors state that \"We further show how alternative techniques, such as temperature scaling and logit capping, explicitly act as a mechanism to steer models toward overconfidence, allowing the benefits of label smoothing to once again emerge.\" However, no actual experiments/statistics are conducted to support this statement.", "methods_and_evaluation_criteria": "The proposed methods are well-suited to the problem, and the evaluation criteria appropriately align with the task.", "theoretical_claims": "See claims and evidence section.", "experimental_designs_or_analyses": "The empirical experiments conducted in this work are methodologically sound and effectively demonstrate: (1) the impact of label smoothing in mitigating model overconfidence in certain LLMs, and (2) the effectiveness of the custom kernels in enhancing memory efficiency and throughput.", "relation_to_broader_scientific_literature": "NA.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "[-] The paper has several typos in its math symbols. E.g., in Eq (1), the index i should be n. \"L is the length of a discrete input sequence\" on page 3 should be N. \n[-] The paper would benefit from a more logical argument, as well as improved clarity and structure in its writing.", "other_comments_or_suggestions": "NA", "questions_for_authors": "NA", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GSPmNO1czv", "reviewer": "Reviewer_C57F", "summary": "In this work the authors examine the effect of Instruction tuning on LLM calibration, i.e. when the model says it is 70% sure about a prediction does it actually get it right ~70% of the time? They find that majority of available LLMs are reasonably well calibrated, but once Instruction tuning is applied, they tend to become miscalibrated and end up overconfident in their predictions.\n\nPrevious work has used label smoothing during instruction tuning as a way to reduce miscalibration. This work aims to explain why and identify settings, like when a large vocabulary is used, where label smoothing is not expected to help.\n\nThey include theoretical results, such as how things like model and vocabulary size establish lower bounds on entropy and how that relates to calibration.\n\nAdditionally, the develop and benchmark a new method for calculating logits and loss for large vocabularies while supporting label smoothing.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "First, we thank the reviewer for their review of our work and their enthusiasm they express regarding their work. We appreciate that they find the claims well founded, the insights to support these claims, and their agreement regarding our new kernel that incorporates label smoothing.\n\n-----\n\n> Evaluations of calibration is done using just 3 datasets, it would be more convincing if there were more.\n\nTo address the comment raised regarding the use of the different datasets, we are happy to provide additional results on different datasets here (https://anonymous.4open.science/r/Anonymous-0F38/README.md). We provide additional results on two datasets, WinoGrand and TruthfulQA, which demonstrate similar results compared to the prior three that are shown in our manuscript.\n\n> It would be more convincing if there were more [...] and there was some notion of confidence in the entire answer instead of just the first token.\n\nWith respect to whether or not there is some notion of confidence in the entire answer instead of just the first token, we do note that while such a concept exists, there may be limitations that are fundamental to the field of confidence calibration. Take for example two sequences A and B. We can measure a level of confidence on the two as the normalized model perplexity over the two options. However, the fundamental limitation is the need for these two sequences as reference to the model, as otherwise the normalizing constant for the choices will be infinite, due to the infinite potential generations from the model.\n\nWe will specifically mention this fundamental limitation of confidence/calibration measurements in an updated version of our manuscript, in order to make this point transparent. Further discussion on similar questions can be found in our response to reviewers LLNm and Sf9v.\n\n> Figure 5 should probably be at the top of the column\n\nWe also appreciate the mention of the placement of figures, which we will adjust.\n\n------\n\nAgain, we would like to show our appreciation to the reviewer for their positive response to our work and for highlighting the fundamental soundness of our methodology. We also acknowledge some minor comments that were left regarding how to discuss specific points regarding the overarching problem being studied, which we believe have been addressed within this rebuttal. We are hopeful that the reviewer feels similarly, and if there remains any additional question or comments we remain ready to provide additional responses to clarify them.", "claims_and_evidence": "Their claims seem well founded.\n\nThere insights into model size and its effect on entropy is validated in their experiments what show that label smoothing is less effective at calibrating smaller models. And their findings that LS helps calibration is most settings is inline with their claims and previous work.\n\nAdditionally their new LS kernel evaluation supports their claims about it being faster and more memory efficient.", "methods_and_evaluation_criteria": "Evaluations of calibration is done using just 3 datasets, it would be more convincing if there were more, especially in ones where the more text had to be generated and there was some notion of confidence in the entire answer instead of just the first token.", "theoretical_claims": "I did not check the correctness of the proofs", "experimental_designs_or_analyses": "There experimental design seemed reasonable.", "relation_to_broader_scientific_literature": "They mention a lot of previous work, including the other works that first found that LS was helpful in terms of calibration. Additionally, they include multiple common metrics for calibration.", "essential_references_not_discussed": "Not that I know of,", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "Figure 5 should probably be at the top of the column", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mcM87S7LBS", "reviewer": "Reviewer_LLNm", "summary": "The authors proposed using label smoothing to improve language model calibration for supervised fine-tuning. They demonstrate through theoretical analysis and experiments that label smoothing is effective, though computation is heavy for large vocabulary LLMs. They also propose a memory-efficient algorithm without sacrificing much accuracy.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are very grateful and would like to thank the reviewer for their positive assessment of our work. We are happy they mention that they mention the suitability of the theory, experiments, methods and evaluation. To clarify their remaining questions and potential areas of concern, we hope the following discussion can provide the details necessary.\n\n-----\n\n> One thing is that the authors need to include their prompts and templates used in experiments in their Appendix [...] or supplementary material.\n\nWe thank the reviewer for the comment and will put all specific details. Some of these details are already provided in Section 3.1. We can provide the following experimental details:\n\nTraining: We conducted a learning rate sweep over learning rates `[5e-6, 2e-5, 5e-5, 2e-4]` with a summing reduction. We further tested label smoothing hyperparameters `[0.0, 0.1, 0.2, 0.3, 0.4, 0.5]` where `0.0` is no smoothing. We used the open-instruct repository (https://github.com/allenai/open-instruct) at commit `e363290` for our training setup, with modifications to account for our kernel as well as specific experimental hyper-parameter settings and baselines.\n\nEvaluation: Our implementation is based on the MMLU official repository (https://github.com/hendrycks/test). We first evaluate our models on MMLU and then further modify the files here to directly adapt the evaluation dataset to the other tasks at hand. \nWe follow MMLU and use the following prompt for all tasks:  `\"The following are multiple choice questions (with answers) about {}.\\n\\n\".format(query)`.\n\nWe didn’t tune the template for fair comparison, for models that required a chat template, we used `tokenizer.apply_chat_template` (supported by HuggingFace models) directly on the sequence.\n\n> It is interesting to see that label smoothing is helpful to improve language model calibration, as previous improvements mostly happen on verbalized confidence. Not sure if label smoothing is only restricted to logit-based confidence, and classification tasks. [...] Please discuss if your method is limited to solving classification tasks in the paper.\n\nThank you for this question; it is certainly worth further discussion for clarifying inherent limitations in confidence calibration research. Implicitly, it is true that calibration is measurable only on classification-style tasks, or at the very least tasks where there is a set of reference answers where there may be an ordering of which answer is more correct than others. Label smoothing is also generally only applicable for logit-based classification tasks, given that it is used generally for classification-based losses. However similar concepts have been introduced in other domains such as for DPO preference based losses (Chowdhury et al., 2024) which we are currently investigating as a possible follow-up direction of interest.\n\nFor further discussion, we invite the reviewer to read our responses to reviewers C57F and Sf9v, who pose similar questions.\n\n> Some references on LLM calibration have been outdated. Below are a few works you may discuss.\n\nWe sincerely appreciate the references and will include them in our related works and discussion. We agree with the relevance of these works in the broader scope of research on LLM calibration and will keep these in mind for future work as well. If there are further requirements in which the reviewer would like to see these works discussed, we are happy to provide further details in how we plan to incorporate them.\n\n> Please use less LLM polishing for your paper.\n\nThank you for your comment. We would like to note that we actually did not use an LLM to polish our phrasing, but we agree strongly with the sentiment from the reviewer. If the phrasing resulted in any part of our work becoming less clear, we are happy to adjust such portions of the paper.\n\n------\n\nOnce again, we would like to express our sincerest gratitude towards the reviewer for their extensive comments towards our manuscript. We again appreciate their positive comments towards our methodology, experiments and results. We also hope they can read our response towards areas in which they felt improvements could be made and are hopeful that this response has properly addressed those areas. If this sentiment is shared, it would be highly thankful if this could be incorporated within an improved scoring of our work. \n\n------\nReferences\n\n[1] Chowdhury et al. Provably Robust DPO: Aligning Language Models with Noisy Feedback. ICML 2024", "claims_and_evidence": "Both the theoretical analysis and experiment results demonstrate the effectiveness of the label smoothing approach.", "methods_and_evaluation_criteria": "The proposed method is reasonable and convincing, as label smoothing ensures the model not to over-estimate a specific class. For evaluation, the authors apply metrics of accuracy, ECE, and RMS to balance between model performance and reliability. They also include various modern models including Alpaca, Tulu3, and Openhermes.", "theoretical_claims": "Looks good to me.", "experimental_designs_or_analyses": "The experiment design is standard and reasonable. Though they mainly focus on a classification setting, they evaluate on metrics of accuracy, ECE, and RMS to balance between model performance and reliability. They also include various modern models including Alpaca, Tulu3, and Openhermes. They also demonstrate the time and memory usage of their method is more efficient than baselines. One thing is that the authors need to include their prompts and templates used in experiments in their appendix.", "relation_to_broader_scientific_literature": "The paper focuses on improving language model calibration in supervised fine-tuning, which is an emerging topic in LLM reliability. It is interesting to see that label smoothing is helpful to improve language model calibration, as previous improvements mostly happen on verbalized confidence. Not sure if label smoothing is only restricted to logit-based confidence, and classification tasks. Please further describe this in the paper.", "essential_references_not_discussed": "Some references on LLM calibration have been outdated. Below are a few works you may discuss:\n\nscaling-based: \nImproving model reliability via inter-model latent agreement. https://arxiv.org/abs/2305.01481  \nprompting-based: \nJust ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback. https://arxiv.org/abs/2305.14975  \nSFT-based: \nTeaching models to express their uncertainty in words. https://arxiv.org/abs/2205.14334  \nRLHF-based: \nTaming Overconfidence in LLMs: Reward Calibration in RLHF. https://arxiv.org/abs/2410.09724", "other_strengths_and_weaknesses": "Please use less LLM polishing for your paper. There are many uncommon usage in academic papers, examples not limited to:\n(1) \"settings remain where ...\", \n(2) \"Seeking a practical solution, we...\", \n(3) \"Growing vocabulary sizes ... consumed to materialize ..., making training difficult.\" \nFor (3), why not just say \"As the vocabulary size becomes larger, there will be increasing computational costs of... and training will become more expensive\"?", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1. Please discuss if your method is limited to solving classification tasks in the paper.\n2. Please include implementation details (e.g., instruction templates for each task) in the paper or supplementary material.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8dCcIR7d96", "reviewer": "Reviewer_Sf9v", "summary": "The paper studies calibration in LLMs after SFT stage. They mention that SFT leads to some significant degradation in the calibration performance of LLM due to the lack of diversity in learned feature embeddings. To address this, they propose to use label smoothing during SFT stage for better calibration. They provide some theoretical justification on the rationale of using label smoothing by relating them with minimizing the logit distance, and is somehow equivalent to the MAP estimation of the softmax predictions. They also provide more ablations on the effectiveness of label smoothing, and pointed out that label smoothing might not work for models with larger vocab and small hidden dim size, due to limited capacity and lack of \"concentration\" behavior. To practically implement the label smoothing on models with large vocab, they also introduces a novel, memory-efficient GPU kernel implemented in Triton, with optimized memory usage and improved throughput.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are immensely grateful for the thorough review that the reviewer has provided for our work. First, we would like to express our appreciation for the comments regarding the well-supported claims, the usefulness of our custom kernel, the appropriateness of the theoretical claims as well as the broader relationship of our work with robustness/safety of LLMs. We also appreciate the comments raised about improving our work, which we hope the responses below can address.\n\n-----\n\n> The calibration issue of the SFT stage is not explained clearly.\n\nWe appreciate this comment from the reviewer; we will further add discussion relating to existing literature that has explored how LLM representations (the representation of inputs within the feature space) [1, 2]. We agree that this merits a more self-contained paragraph or section that better details these claims, and will provide it within an updated manuscript. [3] also discusses this calibration issue in terms of how tuning in general (not limited to SFT) leads to feature diversity issues which we will also discuss in further detail.\n\n> Does TruthfulQA makes sense for evaluating calibration? It would be good to add some comments/discussions on these.\n\nPlease see our responses to Reviewers C57F and LLNm, who ask regarding whether or not evaluation on other tasks is possible. Results on TruthfulQA and Winogrande are provided here (https://anonymous.4open.science/r/Anonymous-0F38/README.md).\n\n> For Proposition 3.3, it might be worth adding the prior distribution here when you argue for MAP.\n\nWe thank you for your suggestion and will include the following discussion in the final version. The conditional prior $p\\left(z\\mid x\\right)=\\mathrm{Dir}\\left(\\alpha_{x}\\right)$ is a Dirichlet distribution with instance-specific parameter $\\alpha_{x}$.\n\n> It might be good to add more ablations on the smoothing factor $\\beta$. Furthermore, is there any adaptive label smoothing that adjusts the smoothing factor based on hidden size, vocab size, or during training progress?\n\nThank you again for this question. Our response is two-fold. First, we test $\\beta$ across a rather large scale (0.0 to 0.5 at increments of 0.1). Generally, smoothing is always beneficial towards calibration, but too much smoothing can be harmful for accuracy. From our empirical results, a choice of 0.1 or 0.2 is generally most useful. \n\nThere is no adaptive label-smoothing method that we are aware of, but we have considered this as a follow-up direction. Furthermore, we are making a better attempt to derive a potential relationship between $\\beta$ and the model, however this is potentially out of scope for the current work.\n\n> It would be nice to understand the calibration issue after SFT under different training mixtures/datasets? \n\nWe appreciate the comment. We point the reviewer to Table 1, which shows consistent results across three different SFT datasets. We have further tested a mixture of the datasets and observed insignificant differences, thus we did not include it due to a lack of novel insights. Nevertheless, we are happy to provide it within the Appendix as an ablation worth discussing.\n\n> Actually, it seems some baselines are missing, such as temperature scaling?\n\nWe appreciate this mention. We have included a temperature scaling baseline here (https://anonymous.4open.science/r/Anonymous-0F38/README.md). As we can observe, temperature scaling generally can vary in effectiveness between datasets, thus the usefulness of label smoothing is highlighted here.\n\n> Why do we need to focus on label smoothing [...] rather than other model calibration techniques.\n\nWe thank the reviewer for their comment. Label smoothing is particularly attractive as it is interpretable in terms of the objective and why it can benefit confidence calibration (see Section 2 and 3), while also being simple to incorporate within an inherent part of the learning regime, namely the training of models. On the other hand, temperature scaling requires a dataset to determine an appropriate temperature.\n\n------\n\nWe appreciate the extensive feedback provided and the multitude of comments highlighting the strengths of our work as well as those asking for additional details to further highlight them. We hope that this rebuttal provides enough detail to address these needs for additional discussion, as well as better present the strengths of our work. If the reviewer shares the same opinion, we would be extremely grateful for an increased score. We are also willing to further engage in discussion to address any remaining questions and comments.\n\n------\n\n### References\n\n[1] Murthy et al. One fish, two fish, but not the whole sea: Alignment reduces language models' conceptual diversity. _arXiv_ 2024\n\n[2] Wang et al. Preference Optimization with Multi-Sample Comparisons. _arXiv_ 2024\n\n[3] Oh et al. Towards Calibrated Robust Fine-Tuning of Vision-Language Models. _NeurIPS_ 2024.", "claims_and_evidence": "The claims made in the paper are generally well-supported. However, I found that the argument for some claims could be significantly improved. For example, in the section \"Why does Instruction Tuning Lead to Mis-calibration?\", it views the SFT stage as solving an OOD problem. While this might be supported by some literature, it is stated as a fact without any self-contained justification here. Furthermore, the calibration issue of the SFT stage is not explained clearly; it simply mentions that SFT hurts the diversity of the features, which is related to LLM calibration. I feel this part could be strengthened by adding more discussion or making it more self-contained.", "methods_and_evaluation_criteria": "Label smoothing is a well-known technique for adding regularization and improving model calibration. Given the observation that SFT leads to overconfidence in LLMs, applying label smoothing does make sense. Another novelty comes from the development of the memory-efficient kernel, especially for models with large vocab, this is nice as it makes the use of label smoothing more scalable. \n\nThe paper evaluates on three widely used benchmark MMLU, HellaSwag, and ARC-Easy, which seems to be a diverse set of tasks. I am not sure whether there exists any dataset specifically tailored for evaluating calibration, etc, not sure whether TruthfulQA makes sense here. It would be good to add some comments/discussions on these.", "theoretical_claims": "I checked the claims in the main paper, most of them look reasonable [Lemma 4.1, Theorem 4.2]. For Proposition 3.3, it might be worth to add the prior distribution here when you argue for MAP.", "experimental_designs_or_analyses": "The experiment setup does make sense.  Some suggestions: \n\n- It might be good to add more ablations on the smoothing factor $\\beta$, the rule of thumb for selection, and the calibration performance span a wide range of $\\beta$. Furthermore, is there any adaptive label smoothing that adjust the smoothing factor based on hidden size, vocab size, or during training progress?\n- I am not sure whether it is within the scope of the paper, but it would be nice to understand the calibration issue after SFT under different training mixtures / datasets, how they correlated?\n- Actually, it seems some baselines are missing, such as temperature scaling? how do they perform?", "relation_to_broader_scientific_literature": "The paper builds upon the well-known label smoothing method and adapt it to the LLMs with large vocab size, the findings should be related with community studying the robustness / safety of LLMs, and also the general community studying calibration of neural networks.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "Strengths: \n- The paper studies an important problem about calibration in LLMs with large vocab size, and the proposed method of using label smoothing does make sense.\n- The development of a custom GPI kernel is nice, makes the use of label smoothing more memory efficient, and makes it more practical. \n- The paper is generally well-structured, and easy to read.\n\nWeakness:\n- The motivation of using label smoothing is not so clear, there are simpler methods like temperature scaling for model calibration, why we need to focus on label smoothing? It will be great to add more discussions on why we choose label smoothing rather than other model calibration techniques.\n- As noted earlier, the explanation in the section \"Why does Instruction Tuning Lead to Miscalibration?\" could be more self-contained and provide a clearer, more detailed explanation of the underlying mechanisms.", "other_comments_or_suggestions": "See the points mentioned in other threads.", "questions_for_authors": "NA", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The submission studies calibration in the context of supervised fine tuning.  In the context of LLMs, label smoothing is expensive, and the submission makes a contribution of an efficient label smoothing procedure.  The submission is supported by reasonable empirical evidence, and by some theoretical proofs.  The reviewers were generally positive, with three out of four recommending acceptance.  Reviewer fRKR felt that the submission was just below the threshold, with some specific concerns about the use of language in the claims of the paper.  The authors are encouraged to take these concerns seriously and consider refining the claims in the submission to better match the empirical and theoretical evidence.
Decision: Accept (poster)