Towards Understanding Why Label Smoothing Degrades Selective Classification and How to Fix It
Authors: Guoxuan Xia
Olivier Laurent
Gianni Franchi
Christos-Savvas Bouganis
Venue: ICLR 2025 Poster (2025)
Keywords: Uncertainty Estimation
Selective Classification
Label Smoothing
Primary area: other topics in machine learning (i.e., none of the above)

Abstract: Label smoothing (LS) is a popular regularisation method for training neural networks as it is effective in improving test accuracy and is simple to implement. ''Hard'' one-hot labels are ''smoothed'' by uniformly distributing probability mass to other classes, reducing overfitting. Prior work has shown that in some cases *LS can degrade selective classification (SC)* -- where the aim is to reject misclassifications using a model's uncertainty. In this work, we first demonstrate empirically across an extended range of large-scale tasks and architectures that LS *consistently* degrades SC. 
We then address a gap in existing knowledge, providing an *explanation* for this behaviour by analysing logit-level gradients: LS degrades the uncertainty rank ordering of correct vs incorrect predictions by regularising the max logit *more* when a prediction is likely to be correct, and *less* when it is likely to be wrong.
    This elucidates previously reported experimental results where strong classifiers underperform in SC.
    We then demonstrate the empirical effectiveness of post-hoc *logit normalisation* for recovering lost SC performance caused by LS. Furthermore, linking back to our gradient analysis, we again provide an explanation for why such normalisation is effective.

Published as a conference paper at ICLR 2025
TOWARDS UNDERSTANDING WHY LABEL SMOOTHING DEGRADES SELECTIVE CLASSIFICATION AND How TO FIX IT
Guoxuan Xia, 1i Olivier Laurent, 233,4
2.3,4 Gianni Franchi & Christos-Savvas Bouganis'
Imperial College London, 1 U2IS, ENSTA, Institut Polytechnique de Paris,
SATIE, Université Paris-Saclay, DTTS, ONERA*
ABSTRACT
Label smoothing (LS) is popular regularisation method for training neural net- works as it is effective in improving test accuracy and is simple to implement. "Hard" one-hot labels are "smoothed" by uniformly distributing probability mass to other classes, reducing overfitting. Prior work has suggested that in some cases LS can degrade selective classification (SC) where the aim is to reject misclassifi- cations using model's uncertainty. In this work, we first demonstrate empirically across an extended range of large-scale tasks and architectures that LS consistently degrades SC. We then address gap in existing knowledge, providing an explana- tion for this behaviour by analysing logit-level gradients: LS degrades the uncer- tainty rank ordering of correct vs incorrect predictions by suppressing the max logit more when a prediction is likely to be correct, and less when it is likely to be wrong. This elucidates previously reported experimental results where strong classifiers underperform in SC. We then demonstrate the empirical effectiveness of post-hoc logil normalisation for recovering lost SC performance caused by LSS Furthermore, linking back to our gradient analysis, we again provide an explanation for why such normalisation is effective. Project page: https: /ttp:///ssttttteeeeeeeeeeeesaaae ai . gi thub. Lo//nderstt anding- Label - www.-ottt-i-----ee--eeee----eeee wwweeeee-eellleeeeee classi fication/
INTRODUCTION
Label smoothing (LS) (Szegedy al., 2016) is common regularisation technique used to improve classification accuracy in deep learning. The one-hot labels used for cross entropy (CH) are linearly combined with uniform distribution over classes, redistributing the probability mass and "smoothing" the "hard" targets, discouraging overfitting. Due to its simplicity and empirical effectiveness, label smoothing features in many recent training recipes (Vaswani al., 2017; Tan et al., 2019; He et al., 2019; Touvron et al., 2021; Liu et al., 2021; 2022b;e; Tan & Le, 2019; 2021), being particularly popular on the ImageNet- Ik (Russakovsky et al., 2015) image classification benchmark.
Within the domain of uncertainty estimation, LS is well explored in the context of model calibration (Müller et al., 2019; Chun al., 2020; Mukhoti et al., 2020; Liu et al., 2022a), where the aim is to align model's output probabilities with its empirical accuracy. The pairing is intuitive as LS encourages models to output lower probabilities, and models are typically more confident than they are accurate (Guo et al., 2017). On the other hand, there is very little research investigating the effects of LS in the context of Selective Classification. Selective Classification (SC) (Hendrycks & Gimpel, 2017; Geifman & El- Yaniv, 2017; Xia & Bouganis, 2022b; Jaeger et al., 2023) is a problem setting where, in addition to the primary classification task, binary rejection decision is made based on the uncertainty estimated by the model, i.e. reject/abstain if uncertain. The aim is to reduce the number of failures served by the classifier by pre-emptively rejecting potential misclassifications. It is well motivated by applications where safety and reliability are important due to the high cost of failure. For example, when uncertain, an autonomous driving system may require driver assistance (Kendall & Gal, 2017). a medical diagnosis system may defer to a doctor (Beam & Kompa, 2021; Kurz et al., 2022), or visual aid system for the visually impaired may abstain from answering query (Whitehead et al., 2022).
Recently Zhu et al. (2022; 2024) empirically observe that LS can lead worse SC for convolutional neural networks (CNNs) performing image classification, yet it is not clear why this occurs. Concurrently, large-scale empirical evaluations of open-source pre-trained models have shown
1 corresponding author -g.xia212 iimprria all.a.l... .ac.uk

---

Published as a conference paper at ICLR 2025
Input
Ground Truth
DeepLab-13 (ResNet- 101)) Cityscapes
Prediction Cross Entropy
Prediction Label-Smootting (a 0.2)
0.
Uncertainty Ranking (MSP) Cross: Entropy
Uncertainty Ranking (MSP) Label-Smoothing (a 0.2)
0.0
Lowest unner (i.e. mest confident) errors x are yellow Correct pixels are greyed out
ResNet-50 ImageNet MSP
CE
I.S
0.1
L.
logit norm
#ccverage@rrssk=1555
15.66
0.06
22.31
Figure I: Top: LS causes overconfidence for semantic segmentation. The LS-trained model predicts much lower (ranked) uncertainty on incorrect segmentations than CE. In particular, for the erroneous region on the left where the model has predicted parts the "sidewalk" as "road", the LS model is highly overconfident. This could have dire consequences in safety-critical application such as autonomous driving. Bottom: LS leads to close to 0% of samples being accepted (coverage) when striet tolerance of 1% error accepted samples (risk) imposed ImageNet. Deployment-times logit normalisation effectively negates the degradation caused by LS.
that many strong classifiers have surprisingly poor SC ability (Galil et al., 2023; Cattelan & Silva, 2024). Cattelan & Silva (2024) additionally present deployment-times logit normalisation sometimes-effective approach to improving SC, but the reason for its effectiveness remains unclear. In this work, aim empirically validate and analytically demystify the effect of LS on SC, tying together and filling in the gaps in knowledge of previous work with the following key contributions. 1. We show empirically, across a range of large-scale architectures (CNN, ViT) and tasks (image classification, semantic segmentation). that training with LS consistently leads degraded SC performance (see Fig. 1), even if it may improve accuracy. Moreover, we find that the degradation worsens with stronger LS. As LS can be found the training recipes of many of the models evaluated in (Galil et al., 2023; Cattelan & Silva, 2024), this suggests LS as one potential cause for previously unexplained negative results where strong classifiers underperform at SC.
2. We address gap the understanding of LS by providing explanation of this behaviour through analysing the logit-level gradients of the LS loss. We show that the amount L. suppresses the logit directly corresponds the true probability of error Pemors with suppression increasing decreasing) the more likely a prediction correct(wrong. This leads to relatively higher uncertainty on correct predictions and lower uncertainty on misclassifications, degrading the ranking of uncertainties and hurting SC compared to vanilla CE.
show that post-hoc logit normalisation (Cattelan & Silva, 2024) is effective in negating elucidate this effectiveness through the lens of our gradient-basee based analysis. Linking back to the imbalanced logit suppression of LS, we find that logit normalisation compensates for this effect by increasing uncertainty as the max logit increases.
degradation from LS (Fig. 1). Moreover,
2 PRELIMINARIES
For a glossary of notation see Appendix A. Consider a K-class neural network classifier with
parameters that models the conditional distribution P(y) æ;; 0) of labels y E y fwk } k= given
inputs z "_ Typically the network has categorical softmax output π (x; 0) [0, 1]
P(wn xx; 0) = TA (x; 0) exp (2)) Σxr expe;(x), == Wz +b,
(1)
2

---

Published as a conference paper at ICLR 2025
on--hot y- by
smotthed g g=00 02
data supervision
Selective Classification
probabiliyy
accept
reject
xf1 of + of apply
reg regularisation
0.4
X
0.2
62 62g
Figure 2: Left: illustration of how label smoothing (LS) alters a training label. LS reduces data supervision and adds regularisation, potentially improving generalisation by reducing overfitting. Right: illustration of selective classification (SC). Uncertain samples (U > T) are rejected/detected, to reduce the number of errors X served by the system. Rejected samples can be discarded or processed separately (e.g. deferred to a human expert). We wish to better separate/rank / X via U.
where v E Rx are the logits output by the final layer with weight matrix W E R RXXI, bias b E R* and pre-logit features z E IR as inputs. The neural network is trained by minimising the cross entropy (CE) loss on finite dataset Dr = (Ir("), ( j drawn from distribution Paata (2. y). such that it approximately learns the true conditional Pana (y)at).
1 Σ Σ k Sym log P( (x (m); 0) N
LCE
(2)
E (x))
Σx Paat (wx log P( ax; 0)
(x)) [KI. [ (2:
T (x;; 0)] + const. L (0).
where 6ip = l = j. and 0 i # is the Kronecker delta and KL[-I1-] is the Kullback-Leibler
divergence. We use #(x) E [0, 1| as shorthand for the true conditional categorical, i.e.
TR Pasu (w) /zr). Predictions ÿ are then made on new unlabelled input data xt using classifier
function f during deployment,
= f(x 0) = arg max_, P(w | ;))
(5)
We also define the probability of the classifier making an error on a given sample as Pertor = 1 - Rg. where in a slight abuse of notation Fg. is the true probability of the predicted class Pastu (z)
2.1 LABEL SMOOTILING (LS)
Label smoothing involves mixing the original one-hot labels (6 in Eq. (2)) with a uniform categorical distribution u = =1/K. 1 using hyperparameter a E [0, 1] (see Fig. 2). The LS loss is thus
())(0:a)= -1 1/ N n E, (1- (1-a)), amo
1 E
log P(anla
(6)
(1 - a) + N Σn KL u (all 0)]
+ const.
(7)
KL (1- + ou ||: 8)]
2 Bpan((a)) data supervision
const. Liree (8: a)
(8)
regularisation
where we see that it can also be viewed as reduced CE supervision from the data combined with a regularisation term encouraging the softmax to be uniform and preventing it from overfitting to the training data (Fig. 2). Eq. (8) also shows that LS can be seen as learning to predict a "softened" version of the true conditional (1 (xx + au, encouraging model to be less confident on all samples.
2.2 SELECTIVE CLASSIFICATION (SC)
A simple downstream task for estimates of predictive uncertainty is to reject (or detect) predictions that may incur high cost (Xia & Bouganis, 2022b; Jaeger al., 2023), using a binary rejection rejection,function,
50 (reject prediction),
if (())
(uncertain)
g( r)
{1 (accept prediction),
if (x) S T (confident),
3

---

Published as a conference paper at ICLR 2025
where (xx) is scalar measure of predictive uncertainty extracted from the prediction model and is a user-set operating threshold. Intuitively, we reject if the model is uncertain, preventing the system from serving failures, which may then be processed separately. We are only concerned with the relative rankings of Us rather than absolute values- we want to be more uncertain on failures. In the case where the prediction task is classification and we wish reject potential misclassifications (X), we can use a selective classifier (Chow, 1970; El-Yaniv & Wiener, 2010) (f.g). g), which is simply the combination of a classifier (Eq. (5)) and the aforementioned binary rejection function g (Eq. (9)). Fig. 2 contains an illustration. To evaluate a selective classifier, we use the 0/1 classification error,
[0, if f(x) =
(correct )
[s)()(a), = 11, if f(x) # y (misclassified x),
(10)
to define the selective risk (EI- Yaniv & Wiener, 2010; Geifman & EI- Yaniv, 2017) as
Risk(J, TT ( (m.y) [g(a; T) (f
(11)
(se [a::
which is the average error on the accepted samples. The denominator of Eq. (11) is the proportion of samples accepted, or the coverage, Cov = (m) 7)]. Our objective is to minimise risk
for a given coverage (lower Serror on accepted samples) and/or maximise coverage for given risk (accept more samples). Note this can be achieved both through improving (fewer errors) and through improving g (better rejection). SC performance is evaluated via the Risk-Coveriy Coverage (RC) curve (Geifman Elmiv, 2017) (see Fig. for examples). The area under the curve (AURCI) provides an aggregate metric over T.' whilst the curve can also be inspected at specific operating points (Whitehead et al., 2022; Xia & Bouganis, 2023), e.g. coverage at 5% risk (Cov@5t), For deployment, can be set using a held-out validation dataset by finding suitable operating point on the RC curve according to an external requirement e.g. risk- % if tolerance for failure is low.
Uxx)= = -MSP(x) = max = max, T& (z; 0) = [0]]
æx;
(12)
iee the (negative of the) Maximum Softmax Probability (MSP) (Hendrycks & Gimpel, 2017) is the model's estimate of the probability of its prediction y. We focus on this uncertainty measure U for SC as it is popular default choice and has been shown to consistently and reliably perform well in the literature (Xia & Bouganis, 2022b; 2023; Jaeger et al., 2023; Feng et al., 2023) compared to max; Panta (wx (z)
alternatives. MSP is a natural choice as P(y x: 0) models Paaaa (y]=z), and U (or any scalar monotonic to it) provides the optimal risk if the true distribution Pa (v/ 2)) is known (Chow, 1970). We omit evaluations on other softmax- noftmax-based U from the main paper as we find them to behave similarly to MSP. In line with previous work (Xia & Bouganis, 2022b; J Jaeger et al., 2023; Zhu et al., 2024), we find that OOD detection (Yang et al., 2021) scores perform poorly at SC, and also omit them (see Appendix F.1 for discussion and additional results).
Over/Underconfidenne Since we are only concerned about the relative ranking of U, we loosely refer to overconfidence as when model's 's estimate of uncertainty for given sample is (relatively) low when P Pator high, as we want a model to be uncertain when it is likely to be wrong. Underconfidence is then the inverse. Both will result in worse SC: overconfidence leads to errors being accepted, un- derconfidence to correct predictions being rejected. Intuitively, over/underconfidence may arise from poomnss of fit, where (x: 0) fails to accurately model the true conditional distribution म (Ix) (Fig. 4).
This is different to the definition used in model calibration (Guo et al., 2017), which, notably, is concerned with marginal properties (averaged over the input distribution) rather than per-sample properties, as well as absolute probabilities rather than relative rankings. It is possible to be highly over/undercooffident in calibration but optimal for SC and vice versa (Zhu et al., 2024), e.g. applying the monotonic transformation U = max [ [=)|<)] |=000 We note that model calibration is an independent task to SC (Jaeger et al., 2023), well explored in the context of LS (Müller et al., 2019), and is not the focus of this work. LS may improve calibration by increasing the uncertainty of both correct and incorrect X predictions, however, this will not necessarily help SC (Zhu et al., 2024).
THE EFFECT OF LABEL SMOOTHING ON SELECTIVE CLASSIFICATION
Zhu et al. (2022; 2024) observe empirically (as part of a broader investigation) that for single value of a LS degrades SC for CNN-based image classification. In this section we aim to validate this
We note that there number of alternative aggregate metrics to AURC (Geifman et al., 2019; Cattelan & Silva, 2024; Traub et al., 2024). We choose omit them, as focus on non-aggregated results in this work.
4

---

Published as a conference paper at ICLR 2025
behaviour wider range of large-scale tasks and architectures. We find that LS leads to consistent degradation in SC, with stronger LS leading to greater degradation. We suggest that it may partially explain the findings of recent large-scale empirical evaluations of open-source pre- trained models (Galil et al., 2023; Cattelan & Silva, 2024) where strong classifiers surprisingly underperform on SC.
Experimental details. We investigate large-scale image classification on ImageNet-Il Ik (Rus- sakovsky et al., 2015) and semantic segmentation (pixel-level classification) on Cityscapes (Cordts al., 2016). For ImageNet, we randomly split the validation set into 10,000 validation and 40,000 evaluation images. For Cityscapes, we randomly split the original validation set into 100 validation and 400 evaluation images. To estimate the risk for the RC curves, we subsample 5000 labelled pixels per image at random. We evaluate on the same pixels between models. The only parameter varied between training runs the level of LS a, and to isolate the effects of L.S, we train all models from scratch using simple recipes. We purposely avoid augmentations such as MixUp (Zhang et al., 2018) and CutMix (Yun et al., 2019) as these directly affect the training labels, which would interfere with our experiments. For ImageNet classification, we train ResNet-50 (He et al., 2016) and ViT-S-16 (Dosovitskiy et al., 2021) using only random resized cropping and flipping for data augmentation. To achieve decent accuracy for VIT training from scratch without advanced augmentations, use sharpness-awan aware minimisation (SAM) (Foret et al., 2021; Chen et al., 2022). For semantic segmentation Cityscapes, we train DeepLab V3+ (Chen et al., 2018) (ResNet-101 backbone) using only random cropping, flipping and colour jitter for augmentations. Extended training details for reproducibility can be found in the Appendix C. We release our code on GitHub.
3.1 LABEL SMOOTHING DEGRADES SELECTIVE CLASSIFICATION
To examine the effects of LS on SC, we plot RC curves in Fig. 3. We use U MSP and vary only the LS level between training runs. We see that training with LS leads to noticeable degradation for selective classification, with higher a worsening the effect. We provide illustrative examples of LS overconfidence on ImageNet in Fig. 9, Although LS leads to slightly better risk at higher coverage, it quickly becomes worse than CE as coverage is reduced. The degradation is especially evident for the low-risk regime, which is relevant to safety-critical scenarios where tolerances for error are strict. For example, if the target is to achieve only 1% erron/risk on ImageNet, then all of our LS models have close to zero coverage, rendering them effectively useless (Figs. 1 and 3). To further highlight the overconfidence caused by LS, we visualise the uncertainty ranking of incorrect pixels for the segmentation of a Cityscapes scene in Fig. 1. The LS-trained model is extremely confident for an erroneous region where it has predicted the "sidewalk" as "road". Whilst the CE model has made the same error, it is much more uncertain. This illustrates potential danger in an autonomous driving scenario if the vehicle is making decisions based on uncertainty estimates.
Upon inspection, many of the SC-underperforming models benchmarked in (Galil al., 2023; Catte- lan & Silva, 2024) are trained using ISS aligning with our results. The models are sourced from repos- itories such as torchvi s on (Paszke et al., 2019) and t mm (Wightman, 2019), where the training recipes are optimised for top-1 accuracy. We note that for ImageNet, LS is such a common technique that it is often used by default, and not even mentioned in papers, e.g. (Tan & Le, 2019; 202 I). Overall, results highlight that only optimising for accuracy may result in negative downstream consequences and that practitioners of SC need to be aware of the effects of their training recipes. We include additional discussion of existing benchmarks and model checkpoints in Appendix F.6.
4 TOWARDS EXPLAINING THE NEGATIVE EFFECT OF LS ON SC
In the following section, we attempt to delve deeper into why our empirical results (and those in (Zhu al., 2024)) occur, shedding light on a previously unexplained phenomenon. It might intuitively seem odd that LS degrades SC performance as the transform applied to the true targets in Eq. (8) (1-a) (x) + au reduces the max probability for all samples but does not change the relative ranking. However, this only carries over to the model when it is knowledgeable about the data, so it is well fit π (x; 0)= z (1 a) a)π (x) + au. For such samples, the model has learnt to be more uncertain. However, we cannot assume that the model is equally well fit on all regions of the data distribution."
We additionally provide small-scale CIFAR, tabular and text experiments in Appendices B.3 to B.5. For the sake of simplicity we avoid the term epistemic uncertainty (see Appendix F:2 for hrief discussion).
5

---

Published as a conference paper at ICLR 2025
ResNet- 50 (Serrar 1, SAURC 1 ImageNet
20
== 0.1 (22.6. 7.25)
Nrike
02 7.62)
-. 7.89)
10
MSP 40 ax
Scoverage
%coverage
Scoverage
SAM (Serrror %AURC
ImageNet
OF 125.9, 7.51)
Wrisk
a.1 (25. 7.68)
a5. (25.0, 7.93)
L5 0.3 (25.1, 8.26)
Nrisk
X
(1-
20
60
80
100
20
75
100
Ncoverage
%coverage
%ccoverage
DeepLabV3+ ResNet-101 (%imlou 1, KAURC Cityscapes
1.5
1.5
CE (71.8 0.481
%rrake
e50. (72.17, 0.75)
%risk from
LS e.2 (72.34, 0.93)
1.0
2.0
0.90) MSP 1 (1 a) ark
X
0.0
0.0
%ccoeragee
%coveraae
%coverage
Figure 3: Risk-coverage plots for different levels of LS for different models and tasks (ImageNet classification and Cityscapes semantic segmentation). Although may improve error rate/accuracy at 100% coverage, label smoothing consistently degrades SC performance.
Through analysing the logit gradients, provide a potential explanation: LS suppresses the logit differently depending on how well fit the model is to given sample suppression is stronger the more likely model is correct(wrong), harming the model's ability to separate / X.
stronger(weaker) (weaker)
4.1 COMPARING LOGIT GRADIENTS BETWEEN CE AND LS
To better understand how LS could lead to degraded SC, we consider how LS affects logit-level training gradients. These the first in the chain rule for backpropagation and so directly contribute to all parameter gradients during training. We take the gradient of Liree (Eqs. (4) and (8)),
dCiue Ov.
Ocine Oek
a- + OOK T
(13)
data supervision regularisation
for a single sample, where in a slight abuse of notation we omit the outer expectation over Paata (x) for convenience. We can then define the suppression gradient on the logits,
a(C
Cirue LEE
OC
aC
a [ 1/K| = atT - aa/K
(14)
O0k
Oek
deg
Dog
'err assume that the empirical loss C (Fqs. (2) and (6) approximates L in order relate our discus- sion P Pence However, the analysis the empirical loss leads similar conclusions (see Appendix D).

---

Published as a conference paper at ICLR 2025
poorly fit- overconfident
poarlyy fit underconfident
1,0
sup. ant softmax nč:: 09
0.8
likely weak suppression
likely x weak suppression
likely / strong suppression
likely x strong suppression on max logit
0.2
0.0
0.2)
-0.2
4/
42
42
Figure 4: How the suppression gradient (Eq. (14)), iee the difference between LS and CE gradients, affects the logits differently. L. affects the max logit differently depending on how well fit the model is for given sample æ. In the left two when U is lower (sharper softmax), the suppression on the max logit is lower when the model is poorly fit and likely to be wrong. In the right two when U is higher (flatter softmax), the suppression is higher when the model is poorly fit and more likely to be correct. Thus, LS degrades the softmax's 's ability to separate / vs X, hurting SC.
which is the difference between the LS and CE gradients. This represents how LS influences training at the logit level in comparison to CE. Notably, it only depends on the target म. Gradient descent in- volves updating weights in the opposite direction the gradient. Hence ang suppresses vi when the true probability T& is higher. The second term- -a/K K uniformly increases the logits for all samples, which does not affect the softmax as it is invariant to uniform logit shifts ((v) ( + 11), n E R.
4.2 IMBALANCED SUPPRESSION DEGRADES UNCERTAINTY RANKING OF / VS
We now consider how the suppression gradient affects the maximum logit Umax.
oug a/K= = a111 - Panor] az/ K,
(15)
max
which shows that the suppression on Umax decreases as the probability of error increases. This directly impacts softmax-based U such as MSP (see Eqs. (1) and (12)), as the exponentiation of the softmax will be dominated by the largest logits, especially the max logit Umax- The max logit will be especially dominant on high-confidence samples which are the last to be rejected.
Eq. (15) suggests that for predictions that share the same value of U, those with higher Porror (more likely x), will have Vmmax suppressed less, whilst predictions with lower Pemor (more likely v) will have Umax suppressed more. Thus, softmax-bast -base U will have the relative ranking of correct and incorrect X predictions degraded- - less uncertain on X more uncertain on /. To further build an intuition, we consider how a model may not be uniformly well-fit over the data distribution during training. As illustrated in Fig. 4, varying levels of fit with regards to the true distribution स leads to differing Pewor for predictions with similar U. resulting in the degradation described above. Finally, as the suppression gradient in Eq. (15) is proportional to a, this explains why stronger LS leads to greater degradation. In summary: for a given predicted U the Perror will vary depending on how well fit the model is. L.S suppresses the max logit more(less) when Perror is lower(higher), degrading the ranking of / vs X for softmax- based U, hurting SC. We note that although our analysis is performed on training gradients, all our experiments are performed on evaluation data, suggesting that effects indeed generalise to unseen data. We also note that our explanation is purely based on the loss, and thus generalises across network architectures and tasks, matching the empirical results in Sec. 3.
LS empirically leads to higher Umax on misclassifications X. To further validate the effects of Eq. (15), we plot the mean=std. of Umax given Tmax for Re on evaluation data in Fig. 5. We see that for LS the distribution of correct / is below incorrect X, whilst for CE they are roughly similar. This provides further empirical support for Eq. (15) which states that when using I.S. Umux is more strongly suppressed for lower Paror- For results on other models see Appendix B.2.
LS empirically leads to increasing overconfidence on less-well-fit data. To further investigate how LS affects uncertainty differently depending on how well-fit/know// well-ft/knnolledeable the model is about data, we artificially introduce distribution-shifii
data that we expect our models to be worse fit on. We use ImageNet-Sketch (Wang et al., 2019), a dataset containing sketches of each ImageNet class and evaluate on the combination of the 50,889 ImageNet- Skeech and 40,000 ImageNet evaluation images in Fig. 6. Even though the regularisation of LSS improves the error rate, the degradation of
7

---

Published as a conference paper at ICLR 2025
CE
LS a= 0.2
LS a 0.2
Your (meana std)
17.5
ResNet-50 ImageNet
relattv:: mean of
15.0
Test
12.5
0.0
10.0
-0.5
5.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.
Figure 5: Distribution of the max logit Umax given the MSP max for correct / and incorrect X predic- tions on evaluation data. Ummax is lower for / for the LS model, whilst the distributions are roughly similar for CF. This empirically matches the imbalanced max logit suppression described in Eq. (15). We calculate the mtd a 0.05-wide sliding window.
ResNet-S0 (%crror 1, NAURC 1)
12
covengge ImageNia -
10
ImageNet+Sketch
ttomles
(52.6,
%rlsk
a5 0.1 (52.1, 24.28)
LS LS (519. 25.41)
(518, 24.85)
Isuuples
174
MSP (1- a/K
17.8
eomplee
6808
2780
18
Gemors
225
S49
rate
3.3
24.1
Isuples
6726
2362
20
40
100
10
20
LSo 0.1
teroors
258
647
%coverage
%ccoverage
enco Tate
3,8
27.4
Figure 6: Left: Evaluation on the combination of ImageNet + ImageNet-sketch. low-uncertainty predictions, the degradation caused by LS is exacerbated when distribution shift is artificially introduced (vs Fig. 3). Right: statistics @ 10% coverage of the combined evaluation set. As the level of LS increases, the number of accepted errors increases, especially from ImageNet-Sketeh This shows that LS leads to increasing overconfidence on less-well-fit data.
see that for
SC at lower coverages from LS is exacerbated by the distribution shifted data. At 10% coverage of the combined evaluation set, I.S leads to accepting many more errors compared CE, with increasing proportion originating from ImageNet-Sketch. That to say, LS leads to increasing This empirically highlights the situation
overconfidence on less-well-fit data (ImageNet-Sketeh) illustrated on the left of Fig. 4 where confident and well-fit samples have their max logit suppressed, whilst confident but poorly fit samples do not, leading to samples with higher Pance being relatively less uncertain (overconfident). For results on other models see Appendix B.2.
LOGIT NORMALISATION IMPROVES THE SC OF LS-TRAINED MODELS
Ideally, we would like to find way to recover from the degradation caused by I.S. A recent empirical study by Cattelan & Silva (2024) has shown that logit normalisation can improve the SC performance of many (but not all) pre-trained models. During deployment the logits are normalised by their p-norm and then the MSP score Tmax is replaced by the normalised max logit,
max; uyg ='= [w= $ 11 + =1lp = [o + =)/(,, |va. + s|p)
(16)
max
where s is a scalar shift and is found via AURC| grid search validation set. We investigate the efficacy of this approach, applying it to our LS-trained models. Figs. and 13 show the SC performance with and without logit normalisation, and we indeed find that applying logit normalisation greatly improves SC performance for models trained using LS, allowing for improved error rate (@ 100% coverage) with good SC. This is further visualised for semantic segmentation in Fig. 8, where logit normalisation successfully mitigates the overconfident errors (bright yellow) caused by L.S. However, also find that logit normalisation does notably
'We follow Cattelan & Silva (2024) and centralise the logits by subtracting the mean -1/K Σ. tkk For experiments, this has little effect hut has other henefits (see Appendix F.3 for discussion).
8

---

Published as a conference paper at ICLR 2025
ResNet-50 ImageNet
CE 123.0. 6.55)
CE
(23.0, 6.5
%rrkk
LS a 0.2 (22.6, 7.02)
2
10
(22.6, 622)
4.
5
20
40
60
80
100
20
40
60
25
75
100
%coverage
Figure 7: RC curves with inference-time logit normalisation. Logit normalisation improves SC performance on LS models, but has little effect on CE models. For other models, see Appendix B.2.
Crass Entropy MSP
Label-Smoothing
DeepLab-3++ (ResNet-1o1 (011-1001 Cityscapes
1.0
0.8
Cross-Entropy Normalized logits (p
Label-Smoothing (a- 0.2) Normalized logits (p
0.4
0.0
Lowest uncertainty-rankked (i.e. most confident) errors are yellow Correct pixels are greyed out
Figure 8: Visualisation of the effect of logit normalisation for segmentation (same scene Fig. 1).
Logit normalisation significantly reduces the overconfidence of the LS-trained model, although has little effect the uncertainty of the CE-trained model. We provide more figures in Appendix G.
improve the SC of CE models. This aligns with (Cattelan & Silva, 2024) where certain models do not benefit from logit normalisation so the authors suggest "falling back" the MSP.
5.1 EXPLAINING THE EFFECTIVENESS OF LOGIT NORMALISATION.
Although Cattelan & Silva (2024) empirically validate this approach on large number of pre-trained models, remains unclear as to why it is so effective, or why it isn't effective sometimes (further discussion in Appendix F.4). Let shed light the interaction between logit normalisation and LS. we examine Eq. (16), we see that resembles the softmax, however, the logits are raised to power p rather than exponentiated. Crucially, whilst the softmax is invariant uniform shifts in the logits,
exp Uk
exp(Uk + n) Σ. exp(x: + Tu(u + m1), An R,
Th's (v)
(17)
Σ, exp
this is not the case for v'. In fact, we have the following inequality for positive" logits v:
Result 1. For all vectors the co-norm and p-norm strictly decreases when summing a with any uniform vector 11, 0:
(R>u))* with least two different values and p [1, +oo,, the ratio of
V'mcc (v) = || ||00|>//||0|
|| + 17 ||/ ///0 + w/1l/p
V'mar (v + 111
(18)
The proof can be found in Appendix E. Recalling that ||e||oc Umax Result thus implies that for given softmax output π (v) π (v++1) and arbitrary corresponding v, the greater the value of r and thus Unax, the lower the value of the normalised max logit e That to say logit normalisation increases uncertainty when the max logit Umax is higher for the same softmax probabilities л.
Recall how in Sec. 4.2 we show how imbalanced logit suppression leads to degradation in SC. In particular Fig. 5 shows how LS leads to higher Umax given Tmax for errors X this implies that, independent of the value of maax information about Pemor has been encoded in Umax- We see now that logit normalisation will increase the uncertainty (ranking) of errors X relative to correct predictions
Ithough this assumption may not necessarily hold, we find that in practice, Tmx and v/a,, dominated by the larger positive logits, meaning the behaviour discussed still empirically (Appendix E.3 and Fig. 22).

---

Published as a conference paper at ICLR 2025
/ using the information in Umax, leading to improved SC for an LS model. That is to say logit normal- isation effectively reverses the effect of the imbalanced logit suppression from LS, improving SC. We note that Cattelan & Silva (2024) find that models that are less confident (on both / and X) tend to benefit more from logit normalisation, again pointing towards L.S. Given our analysis and empirical results, we strongly recommend logit normalisation for LS-trained models when performing SC. On the other hand, Fig. also shows that the distributions of Umax given max for correct of and incor- rect X predictions are very similar for the CE model. This explains why for CE, logit-normalisation does not seem to help, as Umax does not provide useful information about Pemor given Tmxx-
6 RELATED WORK
Prediction with rejection. Selective classification falls into the broader problem setting of prediction
with rejection. In the case of SC, misclassifications are to be rejected (El-Yaniv & Wiener, 2010).
The baseline approach is to use the MSP (Hendrycks & Gimpel, 2017; Geifman & El- Yaniv, 2017)
and there have been a number of proposed training (Moon et al., 2020; Huang et al., 2020a; Ziyin et al., 2019; Zhu et al., 2024) and architectural (Geifman & EI- Yaniv, 2019; Corbière et al., 2019) enhancements, however, recently the effectiveness of some of these enhancements has been called into question (Feng et al., 2023). Another scenario is out-of-distribution (OOD) detection (Yang et al., 2021; Hendrycks & Gimpel, 2017), where data from outside of the training distribution are to be rejected. There is plethora of research in this field (Xia & Bouganis, 2022a; Sun et al., 2021: Liu et al., 2020; Zhang et al., 2023; Liu et al., 2023; Wang et al., 2022; Hendrycks et al., 2022). Recently, combination of SC and OOD detection has been proposed, where the aim is to reject both miselassi- fications and OOD data (Jaeger et al., 2023; Xia & Bouganis, 2022b; Kim et al., 2023). Notably, Deep Ensembles (Lakshminarayanan) et al., 2020) have arisen as a reliable method to improve performance in all three scenarios (Kim et al., 2023; Xia & Bouganis, 2023; Laurent ct al., 2023; 2024). We believe, given the results of this work, that extending the investigation of LS (and other training enhancements) to other scenarios involving prediction with rejection is an important avenue of future work.
Mixup. Mixup (Zhang et al., 2018) and its variants (Yun et al., 2019; Franchi et al., 2021; Pinto et al., 2022; Liu et al., 2022d; Bouniot et al., 2023) are set of regularisation techniques that involve interpolating between random pairs of samples at training time, modifying both inputs and targets. They are commonly found in many ImageNet training recipes (Tan & Le, 2021; Liu et al., 2021; 2022b) and are often used in conjunction with label smoothing (Wightman et al., 2021; Touvron et al., 2021; Liu et al., 2022c). Research into how Mixup affects SC is a particularly salient avenue of future work as (Zhu et al., 2024) also observe empirically that it can have a negative impact.
Label smoothing. Beyond prediction with rejection, LS is well-explored across various contexts. It has been shown to improve model calibration (Müller et al., 2019; Chun et al., 2020; Mukhoti et al., 2020; Liu al., 2022a) as well accuracy when training under label noise (Lukasik et al., 2020). Knowledge distillation (Hinton et al., 2015), where soft labels are provided by a teacher network, is also commonly linked and combined with LS (Müller et al., 2019; Gao et al., 2020; Yuan et al., 2020; Shen et al., 2021; Chandrascgaran et al., 2022) due to their similarity. Interestingly, in a similar vein to our work, pre-training using LS has been shown to harm transfer learning (Kornblith et al., 2021).
CONCLUDING REMARKS
In this work, we elucidate the effect of label smoothing (LS) on selective classification (SC). Our experiments across various tasks and architectures show that LS leads to consistent degradation in a model's s ability to reject misclassifications, even it improves accuracy. By analysing the logit-level gradients of the L.S loss, we provide an explanation for this previously not understood behaviour -.S suppresses the max logit more(less) the more likely a prediction will be correct correct(wrong)) (wrong), degrading model's uncertainty ranking of correct / vs incorrect X predictions. We then investigate post-hoc logit normalisation as a method to improve the degraded SC performance caused by LS. We find to be highly effective and shed light on why- it reverses the effect of the aforementioned imbalanced suppression by increasing the uncertainty when the max logit is higher. We hope that our work encourages more research into understanding how different training techniques may impact model performance in downstream applications such as uncertainty estimation. A further discussion about the practical impact of our work, as well potential future work can be found in Appendix H.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
Guoxuan Xia's PhD is funded jointly by Arm and the EPSRC. This work was performed using HPC resources from GENCI-IDRIS (Grant 2023(AD001011970R33) We thank the anonymous reviewers for their valuable feedback.
REPRODUCIBILITY STATEMENT
To ensure transparency, we use publicly available datasets: CIFAR-10/100, ImageNet, Cityscapes, IMDB, Bank Marketing, and Online shoppers. Our detailed experimental methods are outlined in Appendix C. Finally, the proof supporting our explanation of the effectiveness of logit normalisation provided in Appendix E.
To help replicate our work, we share our source code GitHub based on nncetanty (Lafage al., 2025),' 7, including the configuration files and training code. We also release the most important models on Hugging Face.
ETHICS
Our primary objective is to contribute to enhancing the reliability of machine learning methods, with a particular focus on raising awareness about the negative effects label-smoothing on selective classification. While our work aims to improve the robustness of ML systems, we acknowledge the potential risk that these advancements could be misapplied in harmful ways.
smooth ng- Sel--eleeeee 1
classi cat cn-Code
httpps: / htth//gginnggt www.gingaacc.c..oooooooooooooooo hot.c//////SS2211111 labe" smoothiiooiiooooo bootthing--e ng- Sel seleecin ive
fi fl
11

---

Published as a conference paper at ICLR 2025
REFERENCES
Andrew Beam and Ben Kompa. Second opinion needed: Communicating uncertainty in medical artificial intelligence. NPJ Digital Medicine, 2021.
Quentin Bouniot, Pavlo Mozharovskyi, and Florence d' Alché Buc. Tailoring mixup to data using kemnll warping functions. arXiv preprint arXiv:231 1.01434, 2023. 10
Luís Felipe Prates Cattelan and Damilo Silva. How to fix broken confidence estimator; Evaluating post-hoc meth- ods for selective classification with deep neural networks. In UAI, 2024. 2, 2,4,5, 5, 8, 9, 10, 17, 25, 28, 29, 31, 32
Keshigeyan Chandrasegaran, Ngac-Trung Tran, Yunqing Zhao, and Ngai-Man Cheung. Revisiting label smoothing and knowledge distillation compatibility: What was missing? In ICMI,, 2022. 10
Liang-Chin Chich Chen, Yukun Zhu, George Papandrcou, Florian Schroff, and Hartwig Adam. Encoder-dccoder with atrous separable convolution for semantic image segmentation. In ECCV, 2018. 5
Xiangning Chen, Cho-Jui Hsich, and Boqing Gong- When vision transformers outperform resnets without pre-training or strong data augmentations. In ICI.R, 2022, 5, 24,31 31
C. Chow. On optimum recognition emor and reject tradeoff. IEEE Transactions Information Theory, 1970. 4
Sanghyuk Chun, Seong Joon Oh, Sangdoo Yun, Dongyoon Han, Junsuk Choe, and Young Joon Yoo. An empirical evaluation robustness and uncertainty of regularization methods. ArXiv, ftm/2003.03877 2020. 1, 10
Charles Corbière, Nicolas Thome, Avner Bar-Hen, Matthieu Cord, and Patrick Pérez. Addressing failure
prediction by learning model confidence. NeurIPS, 2019. 10
Marius Cordts, Mohamed Omran, Schastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset semantic urhan scene understanding. CVPR, 2016. 5, 25
Alexey Dasovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image worth 16x16 words: Transformers for image recognition scale. In ICLR, 2021. 5,
Ran El-ianiv and Yair Wiener. On the foundations of noise-free selective classification. Journal of Machine Learning Research, 2010. 10
Leo Feng. Mohamed Osama Ahmed, Hossein Hajimirsadeghi, and Amir H. Abdi. Towards better selective classification. ICLR, 2023. 4, 10,
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Bchnam Neyshabur. Sharpness-aware minimization for efficiently improving generalization. In ICLR, 2021. 5, 24
Gianni Franchi. Nacim Belkhir, Mai Lan Ha, Yufei Hu, Andrei Bursue, Volker Blanz, and Angela Yao. Robust
semantic segmentation with superpixel-mix In BMVC, 2021. 10
Yarin Gal. Uncertainty in Deep Learning. PhD thesis, University of Cambridge, 2016. 27
Ido Galil, Mohammed Dabbah, and Ran El- Yaniv. What can we learn from the selective prediction and uncertainty estimation performance of 523 imagenet classifiers? In ICLR, 2023, 2, 51
Yingbo Gao, Weiyue Wang, Christian Herold, Zijian Yang. and Hermann Ney. Towards a better understanding of label smoothing in neural machine translation. In IJCNLP, 2020. 10
Yonatan Geifman and Ran El- Yaniv. Selective classification for deep neural networks. In NeurlPS, 2017. 1, 1, 10
Yonatan Geifman and Ran El- (aniv, Selectivenet: A deep neural network with an integrated reject option. In ICML, 2019. 10, 27
Yonatan Geifman, Guy Uziel, and Ran El- El-Yaniv Bias-reduced uncertainty estimation for deep neural classifiers.
In ICIR, 2019. 4
Federica Granese, Marco Romanelli, Daniele Gorla, Catuscia Palamidessi, and Pablo Piantanida. Doctor: A simple method for detecting misclassification errors. In NeurlPS, 2021. 27
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In JCMI, 2017. 1,4
12

---

Published as a conference paper at ICLR 2025
Kaiming He X. Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016. 5, 23, 25
Tong He, Zhi Zhang. Hang Zhang. Zhongyue Zhang, Junyuan Xie, and Mu Li. Bag of tricks for image classification with convolutional neural networks. In CVPR, 2019.
Dan Hendryeks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. ICLR, 2017. 4,
Dan Hendrycks, Steven Basart, Mantas Mazeika, Andy Zou, Joe Kwon, Mohammadreza Mostajabi, Jacob Steinhardt, and Dawn Song. Scaling out-of-distribution detection for real-world settings. ICML, 2022. 10
Geoffrey Hinton, Orial Vinyals, and Jeffrey Dean. Distilling the knowledge in neural network. In NewrlPSW, 2015. 10
Gao Huang, Zhuang Liu, and Kilian Q. Weinberger. Densely connected convolutional networks, CVPR, 2017. 22
Lang Huang, Chao Zhang, and Hongyang Zhang. Self-adaptive training: beyond empirical risk minimization. In NeurlPS, 2020a. 10
Xin Huang, Ashish Khetan, Milan Cvitkovic, and Zohar Karnin. Tabtransformer; Tabular data modeling using contextual embeddings. arXiv preprint arXiv:2012.06678, 2020b. 22
Byke Hüllermeier and Willem Waegeman. Alcatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods. Machine Learning, 2021. 27
Paul Jaeger, Carsten Tim Lüth, Lukas Klein, and Till J. Bungert. call to reflect on evaluation practices for failure detection in image classification. In ICLR, 2023. 1,3,4, 4, 10,
Alex Kendall and Yarin Gal. What uncertainties do we need in bayesian deep learning for computer vision? In NeurlPS, 2017.
Jihyo Kim, Jiin Koo, and Sangheum Hwang. unified benchmark for the unknown detection capability of deep neural networks. Expert Systems with Applications, 2023. 10, 27
Andreas Kirsch. Advancing deep active learning & data subset selection: Unifying principles with information-theorg intuitions, 2024. 27
Simon Kornblith, Ting Chen, Honglak Lee, and Mohammad Norouzi. Why do better loss functions lead to less transferable features? In NeurIPS, 2021. 10, 32
Alex Krizhevsky, Learning multiple layers of features from tiny images. Technical report, MIT, 2009. 22,
Alexander Kurz, Katja Hauser, Hendrik Alexander Mehrtens, Eva Krieghoff-Henning. Achim Hekler, Jakob Nikolas Kather, Stefan Fröhling, Christof von Kalle, and Titus Josef Brinker. Uncertainty estimation medical image classification: Systematic review. JMIR Med Inform, 2022.
Adrien Lafage, Olivier Laurent, and Gianni Franchi. Torchuncertainty: Deep learning uncertainty quantification, 2025. 11, 24
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In NeurlPS, 2020. 10
Olivier Laurent, Adrien Lafage, Enzo Tartaglione, Geoffrey Daniel, Jean mare Martinez, Andrei Bursuc, and Gianni Franchi. Packed ensembles for efficient uncertainty estimation. ICLR, 2023. 10, 23
Olivier Laurent, Emanuel Aldea, and Gianni Franchi. A sammelry-aware exploration of bayesian neural network posteriors. In ICLR, 2024. 10
Bingyuan Liu, Ismail Ben Ayed, Adrian Galdran, and Jose Dolz. The devil is in the margin: Margin-based label smoothing for network calibration. CVPR, 2022a.
Wei Liu, Andrew Rabinovich, and Alexander C Berg. Parsenet: Looking wider better. arXiv preprint arXiv:1506.0m 04579, 2015. 25
Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy-based out-of-distribution detection. NeurlPS, 2020. 10, 27
Xixi Liu, Yaroslava Lochman, and Christopher Zach. Gen: Pushing the limits of softmax-bast out-of- distribution detection. In CVPR, 2023. 10
13

---

Published as a conference paper at ICLR 2025
Ze Liu, Yutong Lin. Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021. 10,
Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao. Zheng Zhang. Li Dong. Furu Wei, and Baining Guo. Swin transformer v2: Scaling up capacity and resolution. CVPR. 2022b. 1, 10, 31
Zhuang Liu, Hanzi Mao, Chao Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. convnet for the 2020s. In CVPR, 2022c. 1, 10, 31
Zicheng Liu. Siyuan Li, Wu, Zihan Liu, Zhiyuan Chen, Lirong Wu, and Stan Z Li. Automix: Unveiling the power of mixup for stronger classifiers. In ECCV, 2022d. 10
Iya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019. 24
Michal Lukasik, Srinadh Bhojanapalli, Aditya Menon, and Sanjiv Kumar. Does label smoothing mitigate label
noise? JCML, 2020. 10
Andrew I.. Maas, Raymond F. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng. and Christopher Potts. Learning word vectors for sentiment analysis. Proceedings of the 49th Annual Meeting of the Association for Com- putational Linguistics: Human Language Technologies, pp. 142-150, Portland, Oregon, USA, June 2011. As- sociation for Computational Linguistics. URL http: www. aelw attg//ntholooyy/111111aa 23,25
Jooyoung Moon, Jihyo Kim, Younghak Shin, and Sangheum Hwang. Confidence-awe aware learning for deep neural networks. In ICML, 2020. 10, 27
Sérgio Moro, Paulo Cortez, and Paulo Rita. A data-driven approach to predict the success of bank telemarketing. Decision Support Systems, 2014. 22
Jishnu Mukhoti, Viveka Kulharia, Amartya Sanyal, Stuart Golodetz, Philip Torr, and Puneet Dokania. Calibrating deep neural networks using focal loss. In NewrIPS, 2020. 1, 10
Rafael Müller, Simon Kornblith, and Geoffrey E Hinton. When does label smoothing help? In NeurlPS, 2019. 1,4, 10
Yurii Exgenevich Nesterov. A method of solving convex programming problem with convergence rate O ( gT In Doklady Akademüi Nauk, 1983. 24
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary De- Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Fang. Junjie Bai, and Soumith Chintala. Pytorch: An imperative style. high- peeefrreffrmecce deep learning library. In NeurlPS, 2019. 5, 24, 31
Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In EMNLP, 2014. 25
Francesco Pinto, Harry Yang, Ser Nam Lim, Philip Torr, and Puncet Dokania. Using mixup as a regularizer can surprisingly improve accuracy & out-of-distriution robustness. In NeurIPS, 2022. 10
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and L. Fei-Fei. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 2015. 5,
C OOkn Sakar, S Olcay Polat, Mete Katircioglu, and Yomi Kastro. Real-time prediction of online shoppers' purchasing intention using multilayer perceptron and Istm recurrent neural networks. Neural Computing and Applications, 2019. 22
Sepp Hochreiter, Jürgen Schmidhuber. Long short-term memory. Neural Computation, 1997. 23
Zhiqiang Shen, Zechun Liu, Dejia Xu, Zitian Chen, Kwang-Ting Cheng, and Marios Savvides. Is label smoothing truly incompatible with knowledge distillation: An empirical study. In ICLR, 2021. 10
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, llya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting, Journal of Machine Learning Research, 2014. 24
Andreas Peter Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer. How to train your vit? data, augmentation, and regularization in vision transformers. Transactions on Machine Learning Research, 2022. 31
14

---

Published as a conference paper at ICLR 2025
Yiyou Sun, Chuan Guo, and Yixuan Li. React: Out-of-distribution detection with rectified activations. In NewrIPS, 2021. 10
Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance initialization and momentum in deep learning. In JCML, 2013. 24
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zhigniew Wojna. Rethinking the inception architecture for computer vision. In CVPR, 2016. 1, 30
Mingxing Tan and Quoc Le. EfficientNet: Rethinking model scaling for convolutional neural networks. In ICMI, 2019. 1, 5, 31
Mingxing Tan and Quoe Le. Efficientnetv2: Smaller models and faster training. In ICML, 2021. 1,5, 10, 31
Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoc V. Le. Mnasnet: Platform-aware neural architecture search for mobile. In CVPR, 2019.
Hugo Touvron, Matthicu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herve Jegou. Training data-efficient image transformers &; distillation through attention. In ICML, 2021. 1, 10,
Jeremias Traub, Till J Bungert, Carsten T Lüth, Michael Baumgartner, Klaus Maier-Hein, Lena Maier-Hein, and Paul F Jaeger. Overcoming common flaws in the evaluation of selective classification systems. NeurlPS, 2024. 4
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ł. ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurlPS, 2017. 1
Haohan Wang. Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representations by penalizing local predictive power. In NeurlPS, 2019.
Haoqi Wang. Zhizhong Li. Litong Feng, and Wayne Zhang, Vim: Out-of-distribution with virtual-logit matching. In CVPR, 2022.
Jiaheng Wei, Hangyu Liu, Tongliang Liu, Gang Niu, Masashi Sugiyama, and Yang Liu. To smooth or not? when label smoothing meets noisy labels. In ICML, 2022. 17, 24, 30, 31
Spencer Whitchead, Suzanne Petryk, Vedaad Shakib, Joseph Gonzalez, Trevor Darrell, Anna Rohrbach, and Mar- cus Rohrbach. Reliable visual question answering: Abstain rather than answer incorrectly. In ECCV, 2022. 1,
Ross Wightman. Pytorch image models. models s, 2019.
https:/ /ttp::://tttttt
com/ wow.://wgghtttan//yttrccc----hee imagge
Ross Wightman, Hugo Touvron, and Hervé Jégou. Resnet strikes back: An improved training procedure in timm. NewrIPSW. 2021.
Guoxuan and Christos-Sawes Bouganis. On the usefulness of deep ensemble diversity for out-of-listriiution detection. ArXiv, http/2207.075177 2022a. 10
Guoxuan Xia and Christos- Phristos-Savvas Bouganis. Augmenting softmax information for selective classification with
out-of-distribution data. In ACCV, 2022b. 1, 10, 27
Guoxuan Xia and Christos-Savvas Bouganis. Window-based early-exit cascades for uncertainty estimation: When deep ensembles are efficient than single models. In ICCV, 2023, 10
Jingkang Yang. Kaiyang Zhou, Yixuan Li, and Ziwei Liu. Generalized aut-of-distribution detection: A survey. arXiv preprint arXiv:2110.11334 133.1 2021. 4,
William Yang. Byron Zhang, and Olga Russakovsky, Imagenet-OOD: Deciphering modern
detection algorithms. In /CLR, 2024, 27
Li Yuan, Francis EH Tay, Guilin Li. Tao Wang. and Jiashi Feng. Revisiting knowledge distillation via label smoothing regularization. In CVPR, 2020. 10
S. Yun, D. Han, S. Chun, S. Oh, Y. Yoo, and J. Choc. Cutmix: Rogularization strategy to train strong classifiers with localizable fcatures. In ICCV, 2019. 10
Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In ICLR, 2018. 10, 32
15

---

Published as a conference paper at ICLR 2025
Jingyang Zhang. Jingkang Yang, Pengyun Wang. Haoqi Wang, Yueqian Lin, Haoran Zhang, Yiyou Sun, Xuefeng Du, Kaiyang Zhou, Wayne Zhang, Yixuan Li, Ziwei Liu, Yiran Chen, and Hai Li. Openood v1.5: Enhanced benchmark for out-of-distribution detection. arXiv preprint 77Xi::3366009300 2023, 10
Zhu, Zhang, Z. Cheng, and Liu. Revisiting confidence estimation: Towards reliable failure prediction. IEEE Transactions on Pattern Analysis and Machine Intellignnce, 2024, 4, 5, 10, 27, 29, 32
Fei Zhu, Zhen Cheng, Xu- Yao Zhang, and Cheng-Lin Lin Liu. Rethinking confidence calibration for failure prediction. In Shai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner (eds.). ECCV, 2022. 1,4
Xu-Yao
Liu Ziyin, Zhikang T. Wang, Paul Pu Liang, Ruslan Salakhutdinov, Louis-Philippe Morency, and Masahito Ucda. Deep gamblers: learning to abstain with portfolio theory. In NewrIPS, 2019. 10, 27
16

---

Published as a conference paper at ICLR 2025
TABLE OF CONTENTS SUPPLEMENTARY MATERIAL
Glossary of Notation
18
Additional results
19
B.1
Illustrative examples of ImageNet overconfidence
19
B.2
Complete results for VIT and DeepLabV3+
19
B.3
Small-scale CIFAR experiments
22
B.4
Small-scale tabular data experiments
22
B.5 Small-scale natural language data
23
Reproducibility
23
C.I Image classification
24
C.2 Semantic segmentation
25
C33 Tabular data
25
C.4 Natural language data
25
C.5 Logit Calculations
25
Analysis of empirical loss (one-hot labels)
26
Result and proof
26
F
Discussions
27
F.1 U other than MSP
27
F2 Aleatoric and epistemic uncertainty
27
F3 On the assumption that logits are positive
27
F.4
Explanations of Logit normalisation in (Cattelan & Silva, 2024)
................................
29
F.5
Analysis of Negative Label Smoothing (Wei al., 2022)
30
F6 Existing benchmarks and training recipes with LS
31
Additional segmentation results
31
H
Impact and Future Work
32
17

---

Published as a conference paper at ICLR 2025
A
GLOSSARY OF NOTATION
We summarise the main notations used the paper in Tab. 1.
Table I: Glossary of Notation
Notation
Meaning
p(-)
Probability density function
P(-)
Probability mass function
Input datum in R
y
Label (categorical) from the K possible labels {ue MK
P.bna()) (-)
True data distribution
D. (n) y pa
Training dataset of inputs and labels drawn from Paum (2t, V)
0
Model parameters
1
Logits (pre-softmax model outputs) in R
Vmax
Maximum logit, max; (a; 0)
(ax)
True categorical conditional data distribution, Te Paan (xz)
Tx2;0 0)
Softmax output in [0, of model, T
exp Us expr..
P(wa kx; 0)
(-)MSP/TmuX
Maximum softmax probability, max; Te (a: 0), negate for uncertainty Kronecker delta, 6, if j. and if i *
6g
KL|-II-1
Kullback-Leibler divergence
Ccu(O)
Empirical cross entropy loss minimised over finite data Drr
True entropy loss minimised over Paaa y)-
a
Label smoothing parameter in [0, 1]
Css(0:o) oi)
Label smoothing loss
Test input datum
f(x)
Classifier function
9
Label predicted by model for given imput. 9 = f(=":0) arg max P(wler True probability of label predicted by model ng Passa (5)xz)
ng
Parce
Probability of given label prediction being incorrect, Paror Rg..
(xx)
Scalar uncertainty score
g(x;7)
Binary rejection function, 0 (reject) if U >> otherwise (accept)
Risk(f.g:T)
Average error on accepted samples threshold
Coverage(g::)
Propartion of all data that is accepted for threshold
SCar
Suppression logit gradient, difference between LS and CE gradients, p-normalised logits, v' = av + *)/ | + all, with scalar shift s.
Pen
des
dog
18

---

Published as a conference paper at ICLR 2025
y "lagpole'
label y
label y- hotdoo
"ipstick LS 0.2 pred. thain carry uni r anke 20.0%
70.2%
LS 15a-0a 0.2
1S 0.2 prea: 'meat losr
0.2 pred: Band Ald' unc. Farkc 12.2%
pred: "church unc. rank: 15
runc. fank: 10.0%
Figure 9: Illustrative examples of overconfident errors performed by our LS-trained ResNet-50 on evaluation data. Even though Poror is high in all cases (e.g. due to multiple possible labels). the model predicts low (ranking) uncertainty. Note that even though the CE model is wrong as well, has assigned higher ranking uncertainties, reflecting its superior SC ability shown in Fig.
B ADDITIONAL RESULTS
B.1 ILLUSTRATIVE EXAMPLES OF IMAGENET OVERCONFIDENCE
Fig. 9 shows few examples of overconfident misclassification the ImageNet evaluation data by our LS-trained ResNet-50.
B.2 COMPLETE RESULTS FOR VIT AND DEEPLAB
We include additional experimental results that mirror those found in the main paper on ResNet-50, for ViT-S-16 and DeepL.abV3+:
Fig. 10 and Tab. show full ResNet-50 and VIT-S-16 results on ImageNet + ImageNet- Sketch. We that the behaviour of ViT-S-16 is similar to ResNet-50 (increasing numbers of confident errors from sketch as a increases), but less pronounced.
Figs. 11 and 12 shows the distribution of Umax given Tmax for ViT-S-16 and DeepLab We that similarly to ResNet-50, for LS, the distribution of Umax is higher for errors X. Although it is less obvious, is clear for both ViT-S-16 and DeepLabV3+ that for higher Tmax the standard deviations overlap much less than for CE.
Fig. 13 shows the effectiveness of logit normalisation for ViT-S-16 ImageNet and DecpLabV3+ (ResNet- 101) on Cityscapes. We also provide 2 additional segmentation figures in Appendix G.
Table Statistics @ 10% coverage of combined evaluation set. the level L. a increases, the number of errors increases, especially the number of errors from ImageNet-Sketch.
10% coverage ImageNet Sketch
@ 10% coveruge ImageNet Sketch
ResNet-50
VIT-S-16
ImageNet
Sketch
ImageNet
Sketch
#samples
7940
1148
#samples
8529
559
CE
#errors
94
CE
#errors
90
35
error rate
6.4
error rate
6.3
#sumples
7422
1666
#samples
I.S 0.1
Nermors
174
296
1.S 0.1
#errors 158
102
errrr rate
2.3
17.8
euxor rate
1.9
11.9
#samples
6808
2280
#samples
7927
1161
LS a
#errors
549
LS a= 0.2
#errors 196
3.3
24.1
2.5
13.9
#samples
6726
2362
#samples
7618
1470
LS a = 0.3
#errors
258
647
1.S = 0.3
#errors
254
294
eror rate
27.4
crvor rate
20.0
19

---

Published as a conference paper at ICLR 2025
60
12
ResNet-50 (Sherror %AURC
IkaehhittSkkttch
from CE
40
1526, 23.33)
Wrisk
a.. (92. 24.28)
30
(51.8 24.85)
0.3 (51.9, 25.41)
X
2
10
20
40
60
80
100
10
20
30
100
%ccveraae
%ccoeerage
%ccverrage
60
VIT-S-16-SAN SA-MM6-AMM
(%6rror 4. %AURC )
ImageNNt++Sootc
157.4,
Nrikk
0.1 (56.
4 fram
30
1. 0.2 (56.9, 26
(56.7, 27.08)
MSP (1
al a/K
10
Sccveragee
%coverage
%coverage
Figure 10: RC curves evaluating the combination of ImageNet and ImageNet-sketch. For low- uncertainty predictions, the degradation caused by LS exacerbated when distribution shift is artificially introduced. This shows that LS leads to increasing overconfidence less-well-fit data.
a 0.2
LS a= 0.2
16
VIT-S-16-SAM ImageNet
[meanssta)
24 Test
0.3
22
8
0.0
relative
-0.1
7
0.3
0.4
Alnax
Rooo
Rises
Figure 11: Distribution the max logit Vmax given MSP mmax for correct / and incorrect X predictions separately for ViT-S-16 16. Similarly to the main paper, the distribution of errors X is higher than correct predictions / for the LS-trained model. We calculate the mtdn in a 0.05-wide sliding window.
CE
LS = 0.2
LS = 0.2
DeepLabo3- ResNet-101
(meanaatte)
20
Cityscapes
Test
5
0.6
0.4
0.2
2
0.0
0.2
0.4
0.5
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Aeaar
Reow
Tleas
Figure 12: The same as Fig. 11 but for DeepLab (ResNet-101)
20

---

Published as a conference paper at ICLR 2025
VIT-S 1U-S-16-50 SAM ImageNet
(Seerror 4. NAURC 4
125.9.
CE logit norm
p (25.9, 7.66)
Stisk
0.2
LS logit norm p- (25.0, 7.20
10
%coverage
%coverage
%ccoveraee
DecpLabV3
(%imiou T. NAURC )
Citycrapes
logit norm (5.5, 0.56)
0.49)
2.0
%risk from
2.0
LS 0.2 (5.6, 0.93)
1.5
2.5
1.0
1.0
Socoverage
%ccoverage
%icoverage
Figure 13: RC curve showing the effect of logit normalisation for VIT-S-16 and D3epLabV3+ (ResNet-101). The behaviour is similar to the results in the main paper. Logit normalisation is effective in improving the performance of the LS-trained models, bringing them close to the CE models. However, logit normalisation makes little difference to the CE-trained model.
21

---

Published as a conference paper at ICLR 2025
B.3 SMALL-SCALE CIFAR EXPERIMENTS
We also include experimental results on small-scale 32 x 32 CIFAR- 100 (Krizhevsky, 2009). We train a DenseNet-BC (Huang et al., 2017) (k 12, L 100) to show further generality over model architecture families. Figs. 14 to 16 show results that mirror those found in the main paper for ResNet-50 on ImageNet, although we note that L.S does not improve top-1 error rate in this case.
25
DenseNet (K - 12, 100)
10
10
(Serrror
%AURC 4
20
CIRAR-100
Perisk from CE
123.0 6.23)
Nriske
15u=0.1 (23.8, 7.92)
(23.6, 826)
0.3 (23.4, 10.17)
X
20
60
80
100
20
25
50
75
100
%coverage
Scoverage
%icoverage
Figure 14: RC curves for DenseNet on CIFAR- -S degrades SC
DenseNet
1100 (Neerrrr %AURC
10
CIFAR-100
Prisk Prrsskfrrmm from CE
123.0, 6.23)
Wrtke
15
legi norm
LS -- 02 (23.6, 8.26)
logit norm
5
20
60
80
100
20
60
25
50
75
100
%coverage
%coverage
%cooeraae
Figure 15: DenseNet on CIFAR- - logit normalisation improves SC for LS but not for CE
CE
a 0.2
L5 0.2
20
DenseNet
(meanasdd)
18
CIFAR: 100
[k= 12, 100)
8
0.4
2.6
Test
7
0.2
144
12
20
-0.4
3
-0.6
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.0
1.0
Figure 16: DenseNet on CIFAR-100 - LS leads to higher Umax On errors X
B.4 SMALL-SCALE TABULAR DATA EXPERIMENTS
To increase the scope of our experiments on the degradation of selective classification due to smoothin,, perform small-scale tabular binary classification. Following the setting of TabTransformer (Huang et al., 2020b), we train two-hidden-layer MLPs on Bank Marketing (Moro al., 2014) and Online purchasing shoppers intentions (Sakar al., 2019).
22

---

Published as a conference paper at ICLR 2025
12
SAURC
Online Shoppers
CE (9.3. 1.65:
112.3. 2.90)
L5 15. (92.
LS 112.5, 3.56)
100
Nccoeroge
Scoverage
Figure 17: RC curves for MLPs trained on Bank Marketing and Online Shoppers using a 0.6, hence with a gap between the positive and negative smooth labels of 0.4.
In this binary classification setting, we set a to 0.6 sufficiently reduce the gap between the optimal MSP corresponding to the positive and negative detections of the class at stake. With 0.6 the minimum of the loss with label-smoothing is achieved for a prediction with a score of 0.7 when the hard label equals 1, and a score of 0.3 when the hard label is O, resulting in gap of magnitude of 0.4. In Fig. 17, we see that both cases, the AURC of the LS models are worse than those trained that
with the classical cross-entropy despite their accuracy being very similar. Furthermore, the CE models are able to get much closer to zero risk, similar to the image classification experiments.
However, note that the effect of label smoothing seems less pronounced for small-scale tabular data. We posit that this may be due to the data distributions being simpler and easier to capture, such that the neural networks are generally better fit and more knowledgeable about the data. Thus reducing the level of imbalanced suppression over training data (see Sec. 4). To address this lack of difficulty in fitting the distribution, we reduce the number of training points (as described in Appendix C).
Moreover, training small-scale and simple datasets drastically reduces the quality of our estimation of the Risk Dask-Coeraege curves. Given the limited number of data and the very high accuracy achieved by our MLPs, the number of eтrors used to estimate the risk is limited. The estimation of the "true" risk-coverage curve which would be obtained with the whole distribution and suffers from an important variability. The results may thus differ when starting the optimization process from different initializations, using a different batch composition and order, or due the non- deterministic nature of some algorithms used to compute the backpropagation (Laurent et al., 2023).
therefore imprecise
B.5 SMALL-SCALE NATURAL LANGUAGE DATA
With also provide small-scale natural language processing experiments training LSTM-based models (Sepp Hochreiter, Jürgen Schmidhuber, 1997) combined with two-layer perceptrons. We focus on LSTMs to add another architecture, and given that CNNs and transformers were already used the maage-classification setting (see c.g. Sec. 4).
We train two networks the IMDB Movie Review dataset (Maas et al., 2011) with the classical binary cross-entropy loss and our implementation of the binary cross entropy with label-smoothing. Similarly to results in image classification and tabular data classification, Fig. 18 shows that the LSTMs trained with label smoothing have worse AURC than models trained with cross-entropy despite their lower error-rate. Furthermore, the models also display greater error rates at high-confidence (low-coverage). We provide more extensive details on the training in Appendix C.
C REPRODUCIBILITY
Alongside this document, we provide a code demo to train two ResNet-20 (He et al., 2016) CIFAR-10 (Krizhevsky, 2009) with cross-entropy and label smoothing and compare the
23

---

Published as a conference paper at ICLR 2025
12
IMDB
(32.7, 3.82)
10
4.50)
%rakk
20
40
60
80
100
%icoverage
Figure 18: RC curves for LSTMs trained the IMDB Movie review dataset, using 0.6, hence with a gap between the positive and negative smooth labels of 0.4.
corresponding Risk- Risk-Coverre Coverage curves. We recall that our code based on Torch (Lafage et al., 2025) -is is available on GitHub." We release the most important models on HuggingFace. 10
Here, we provide the full details of our training recipes used in the main paper and in the additional results presented in Appendix B. All our models are trained with PyTorrch (Paszke et al., 2019). We use the native implementation from the ErossEntropyLoss multi-class setting and a custom version of the SCEWithLogite. as the original does not support label smoothing. We use the original implementation of the authors for the negative label smoothing (Wei et al., 2022) experiments.
for label smoothing in the for the binary experiments
NB: The results presented in the final camera-ready ICLR 2025 version of this paper may differ slightly earlier preprint versions. This is due to models being retrained with different random seeds, train-val-test splits, and having minor recipe and hardware differences after a consolidation of the experimental codebase TorchUncertainty. Readers should use the information in this final version for reproducibility purposes.
C.I IMAGE CLASSIFICATION
DenseNet CIFAR100. For the DenseNet trained on CIFAR-100, we randomly split the original test into 2000 images for validation and 8000 for evaluation. We take batches of 64 32 32- 32-pixel images and train single GPU for 300 epochs using stochastic gradient descent and a starting learning rate 0.I, Nesterov (Nesterov, 1983; Sutskever et al., 2013), a momentum of 0.9, and 1 x 10 weight decay. We divide the learning rate ten after 150 and 225 epochs. We use standard augmentations: we apply random crop with a four-pixel padding as well as random horizontal flip. We do perform model selection and keep the last checkpoint.
VIT-S-16 - ImageNet. For our VIT-S-16 trained on ImageNet, we take batches of 2048 images and train on 4 100s for 300 epochs with Adam W (Loshchilov & Hutter, 2019) with the As equal to 0.9 and 0.999. We start with linear warmup for 15 epochs, then use cosine annealing scheduler with 3 x 10-3 as the starting learning rate. The models are trained with non-adaptive sharpness-aware minimisation (SAM) (Foret et al., 2021; Chen et al., 2022) with 0.2. We use dropout (Srivastava et al., 2014) rate of 0.1 but no attention dropout. We transform the training images with a standard random resized crop to >24x 224 pixels using bicubic interpolation and random horizontal flip. For evaluation, we center-crop the images to this resolution. For ImageNet, we do not perform model selection and keep the last checkpoint. However, randomly extract validation set of 10,000 images from the validation set to perform logit normalisation, and evaluation on the remaining 40,000.
htt / Labe smooth Sewnootthig---eeeteeee lective- classi fi cati
/oS.To/ / labe -moothhi www.ttottnnnoootoooooooooo ng- Se active
fittion "htrps://torrd-unnceraiinn..
24

---

Published as a conference paper at ICLR 2025
ResNet-550 ImageNet. Our ResNet-50 is trained 4 A100 with stochastic gradient descent for 120 epochs using a batch size of 1024 images. After five epochs of linear warmup, we use a cosine annealing scheduler starting with learning rate of 0.4 with a momentum of 0.9 and weight decay of 1 x 10-1. We use the same transformation of the images as for the ViT and select the last checkpoint for inference. We use the same validation set for the VIT for logit normalisation.
C.2 SEMANTIC SEGMENTATION
Deeplaby3+. We train Deeplaby3+ CityScapes (Cordts et al., 2016) with ResNet-101 (He et al. 2016) backbone pre-trained on ImageNet (Russakovsky et al., 2015). We use stochastic gradient descent with base learning rate of 0.01, divided by 10 for the backbone weights, and reduced follow- ing the "poly" policy (Liu et al., 2015) with a power of 0.9. The weights are optimised with a momen- tum of 0.9 and weight decay of 10-4 We take batch size of 16 images and train for 30,000 steps. During training, we randomly crop the input images and targets to squares with 768-pixel- We apply random horizontal flip and colour-jitter with the classical parameters: brightness, contrast, and saturation levels of 0.5, For testing, we use the images at their original resolution and do not per- form any test time augmentations. For the RC curves, we randomly sample 5000 predictions per image extracted prior to the final interpolation to compute the coverage and error rates. We keep the pixel- wise locations of the samples when changing the level of label smoothing ensure fair comparisons.
sides.
C.3 TABULAR DATA
We perform the tabular data binary classification on two UCI datasets: Bank Marketing and Online shoppers. These two datasets have an input dimension and a number of samples of 7 and 45,2 for the former and 16 and 12,330 for the latter. In both cases, we select 80% of the data points for the test set and train the models for 10 epochs using Py PyTorch's default Adam optimizer and a batch size of 128. Similarly to the other experiments of this paper, we do not perform model selection and keep the last checkpoint. Our models are MIPs with two hidden layers, whose size depends on the input size as follows: the first hidden layer has 4 times as many neurons as the dimension of the input data, and the second has half the number of neurons of the first layer.
C.4 NATURAL LANGUAGE DATA
For the training on the IMDB (Maas et al., 2011) dataset, we tokenize the data with n k and convert them to 300-dimensional vectors using GloVe 840B (Pennington et al., 2014). The architecture of the models is defined as follows:
an LSTM layer with a 300 input dimension and 256 output dimension, followed by dropout layer of rate 0.2,
linear layer keeping the dimension of the vectors unchanged, on which we apply ReLU, followed by a second dropout layer with the same rate,
two linear layers with ReLU reducing the dimension to 128 and 1, respectively (binary classification).
Finally, we train the models with PyTorch's S Adam default for 10 epochs using the pre-made train test split. The code is available on GitHub.
C.5 LOGIT CALCULATIONS
We perform logit evaluation calculations in double precision (softmax, logit normalisation, entropy etc.), in order to reduce the impact of numerical error.
We perform normalisation in Eq. (16) on logits The value of p is searched over (1.2.3,,45.6677.88 on the corresponding validation data with the optimisation metric as AURCJ. We set s = -1/K Σ. Uk following Cattelan & Silva (2024) (see Appendix F.3 for discussion this choice).
25

---

Published as a conference paper at ICLR 2025
D ANALYSIS OF EMPIRICAL LOSS (ONE-HOT LABELS)
We can perform a similar analysis as in Sec. 3 using the empirical loss (Eqs. (2) and (6)) rather than L in the main paper. We leave this to the Appendix the conclusions are similar to analysing _ however, the presence of one-hot targets makes less convenient to reason about the probability of error Pencr and how well fit the model is to the true conditional distribution (y)ar) for a given data sample. Taking the gradients Eqs. (2) and (6) as in Eq. (13),
OCCE Ouk
0LLS Ovk
|| -1-- yug + a/K Te
16m - Tx),
(19)
data supervision
regularisation
which in turn gives the suppression gradient,
0Lup
0(Lis- - LCE)
ULiS
OLCF = a 1o 1/K],
(20)
Oug which is once again independent of the model output In this case, the label has already been sampled, the model is either right / or wrong X, giving two different suppression gradients for the max logit Umax
Ouk
dey
doy
SC
0C
= 1
= a (0 = -a/K.
(21)
mmas
We can see that the max logit is more strongly suppressed during training when the prediction is correct, which aligns with the analysis of giue and Pemor the main paper. When the model is correct during training, the max logit is suppressed, but when it is incorrect the max logit is not suppressed. Thus the uncertainty ranking of correct / vs incorrect data samples degraded, harming SC. This leads to the same conclusions in Sec. 4 and also aligns with the behaviour in Fig. (right) where Uma is higher for errors given the value of MSP.
E RESULT AND PROOF
Result. For all strictly positive vectors (R>=)) containing least two different values and PE E too,, the ratio of the infinite norm and the p-norm strictly decreases when summing v and any uniform vector 11, 1) strictly positive:
v
10 + 171120
(22)
| lp
0 + 1/ 1|
Proof. Let there be a real > 0. Take p > the dimension the norm and v a vector of dimension
K3 of strictly positive elements en for 1 S k S K, such that there exists 1 S i K such that
2 < max Ug. We have that
kSK
1+
2 1
(23)
max Uk
KSK
and, for at least i, have the same equation, yet with strict incquality. We can adapt Eq. (23) get
max
Uk + 17)
KSK
+ +リ
(24)
max Uk RSK
And when set to exponent p 2 1, we obtain
of max of Similar to Eq. (23), please note that using A i, we get the same equation as Eq. (25), although with a strict incquality. We can now sum on the elements of o to get
7)) max Uk + 17)
(Uh
(25)
RSK
KS K
K
Σ
(uk + 7)
1=1
(26)
max
Uk
7)P
max U
KSK
K
1- K
26

---

Published as a conference paper at ICLR 2025
By taking the inverse (all values strictly positive), get
max of
max(uk (uk + 7)
ks
K
ASK
(27)
K
Σ
(uk + 1)p
And setting the equation the exponent p- and replacing the maxima of the by and ey + 17 with the infinite norm, |/o| and | + 1/ respectively, we obtain the result.
F DISCUSSIONS
F.1 U OTHER THAN MSP
In the main body of the paper we focus solely MSP as our uncertainty score U. Fig. 19 shows how LS affects the SC behaviour of MSP (top) compared two other softmax scores: DOCTOR (Granese et al., 2021) (U see that the behaviour is very similar across all three softmax scores, with increasing the level of LS a degrading all the scores. As MSP is the (marginally) best performing and the most commonly used, we thus choose to focus on it for the main paper.
m 22)) (middle) and entropy (U H())
En Ta log TT&)) (bottom). We
Fig. 20 shows the SC performance of Energy (U log Σ exp vg) (Liu et al., 2020). a popular OOD detection score, compared to MSP with and without LS. We see that Energy performs much worse than MSP SC. This aligns with large body of existing work (Xia & Bouganis, 2022b; Jaeger et al., 2023; Kim et al., 2023; Yang et al., 2024; Zhu et al., 2024) that empirically finds that uncertainty scores designed for OOD detection perform poorly at detecting misclassification and are thus not suitable for SC. Thus we choose not to investigate any OOD detection scores in the main paper. We remark that LS also has a strong negative effect on the SC performance of Energy. This is unsurprising Energy is dominated by the max logit. An important future direction would be to investigate the effect of LS on various OOD detection scores on the task of OOD detection.
Finally, we also choose to omit various maining/architecture-Be et al., 2019; Geifman & EI- Yaniv, 2019; Moon et al., 2020; Zhu et al., 2024), as we aim to focus our investigation solely on understanding the training effects of LS. We note that LS (potentially combined with post-hoc logit normalisation) is considerably simpler to implement than the aforementioned training-based approaches, and has been shown to be stable in many more use cases and so may be more likely to be chosen by a practitioner. Besides, recent work (Feng et al., 2023) has suggested that MSP applied the methods in (Ziyin et al., 2019; Geifman & El- Yaniv, 2019) actually performs better than the Us proposed in them (reject logit/selection head).
approaches such (Ziyin
F.2 ALEATORIC AND EPISTEMIC UNCERTAINTY
We recognise that the conceptual decomposition of predictive uncertainty into aleatoric and epistemic uncertainty (Gal, 2016; Hüllermeier & Waegeman, 2021: Kirsch, 2024) can be applied to the discussion throughout this paper, and that some readers may be confused as to why we do not use simpler, more direct language
these terms. We choose our discussion, reduces the number of concepts to introduce and also reduces any potential confusion.
believe it more efficiently conveys
F.3 ON THE ASSUMPTION THAT LOGITS ARB POSITIVE
Largest positive logits dominate. Result assumes that all elements the logit vector are > which is not necessarily true. However, we also find empirically that both v and Tmux tend to be dominated by the largest positive logits. This is intuitive as exponentiating or raising to power > 1 will amplify the larger logits. This is shown in Fig. 21, where we plot the mean=std of v, S and exp for the sorted logits of ResNet-50 a = 0.2 on the ImageNet evaluation set (p optimal on the validation data for logit normalisation in this case). Thus Tmxx(v) (v) Tmax Tmax(Urop-k)
and 2/max (V) x V/nax (topp=)) where the top-k logits are positive and dominate the computation.
As Result holds for oi "'mv (Utop-k),, we still expect logit normalisation to increase uncertainty
for higher Umax when comparing samples with similar _ Tmax even not all Ug > O. (Note that
omit shift s = -1 Ex Ug as it is as 0 in our experiments).
27

---

Published as a conference paper at ICLR 2025
ResNet-50 (Kerror J, 4) ImageNet
20
5
123.0, 6.551
4. from CE
L5 a= 02 (22.6 6 7.62)
1.0
MSP aK
20
40
60
80
100
20
40
60
75
100
Ricoverage
%coverage
%ccoveraee
ResNet-50 50 (Keerror 4, %AURC 4
ImageNet DOCTOR
123.0, 6.57!
1.5
LS 1.5.000 0.1 (22.6, 7.40)
Sriske
L5 a= (22.7. 8.06)
10
20
40
60
80
100
20
40
60
0
20
60
80
100
Svaraag
%coverage
%coverage
RecNet-50 50 (Serrrrr 1. %AURR 1
ImageNet Entropy
= 0.1 122.6, 8.71)
9.36)
(22.7, 9.58)
5
50
Ncoverage
%ccoerage
%coverage
Figure 19: The effect of LS different softmax scores: MSP (top), DOCTOR (middle), Entropy (bottom). We see that the behaviour very similar, with MSP being the best performer.
We can also consider the scenario where we add 77 to only the top-k logits. This more aptly describes the empirical logit behaviour compared to adding logits vary much less than the higher ranking ones (Fig. 21). Here we would expect Tmax (v) to increase very slightly, but would expect v'nax (v) decrease as the numerator of Eq. (26) would be dominated by the top-k largest logits.
all logits, the values of the lower ranking
Fig. 22 shows how empirically logit normalisation indeed increases uncertainty for higher Umax- We plot the med of V'axx given Umax for samples in different MSP bins. We see clearly that in almost all cases, for samples with similar max the normalised max logit v'mux decreases as the original max logit Umax increases. As shown in Figs. 5, 11, 12 and 16, LS leads to misclassifications having higher Umax than correct predictions V. Thus, logit normalisation is able improve the SC performance of LS-trained models by penalising the confidence of higher max logit values.
Logit centralisation. We follow Cattelan & Silva (2024) and suggest to shift the logits by their mean during normalisation,
u(v) 1 - u(D))lp
p(v)
Σ dk-
(28)
In our experiments, this has little-to-no effect as z (for example for ResNet-50 a = 0.2 on the ImageNet evaluation data μ averages 3 x 10-3 with std. 1 x 10 10-3). This is corroborated by Cattelan & Silva (2024)'s observations their Appendix G, where they find this shift does not affect the vast majority of their models.
28

---

Published as a conference paper at ICLR 2025
ResNet-50 (Nerror 4, ImageNett
14
20
MSP CE (23.0, 6.55)
12
%irik %.iskk from CE
7.62]
10
Energy CE (23.0, 11.23)
Energy
12.96)
10
20
40 %coverage
60
80
100
0
20
60
0
25
50
75
100
%ccoveraee
%icoverage
Figure 20: SC performance of MSP and Energy (OOD detection score). Energy significantly underperforms MSP. This behaviour is in line with existing work that shows that uncertainty scores designed for OOD detection are not suitable for SC.
1c5
1c5
meanittdd
1.0
1.0
sorted logits
sorted logits
sorted logits
Figure 21: Std after logits have been sorted highest to lowest for ResNet-50 a = 0.2 on ImageNet evaluation set. We see that d and exp much larger for the top < 10 logits. Thus these logits dominate Tmax and y
We note that according to the analysis Sec. 5.1 and Appendix F.3, for logit normalisation to effec- tive, "/mxx and max should be dominated by positive logits. Logits are necessarily constrained be positive. They can be arbitrarily shifted without affecting the cross entropy loss (e.g. via bias b of the nal layer), or derived as log probabilities which are necessarily negative. Thus we suggest applying the above shift to ensure that the largest logits are positive. We note that other shifts are possible (for ex- ample ming 0g) achieve the same purpose but we leave the exploration of this future work.
F.4 EXPLANATIONS OF LOGIT NORMALISATION IN (CATTELAN & SILVA, 2024)
The work that introduces logit normalisation (Cattelan & Silva, 2024) is primarily an experimental study, where the focus is extracting empirical takeaways. However, the authors do discuss potential reasons for the effectiveness of the approach in their Appendix B. They observe that models that are generally (on both and X) more uncertain benefit from logit normalisation and suggest that logit normalisation alleviates "underconfidence"" by reducing the influence of lower-ranked logits on the uncertainty prediction (using analysis similar ours Appendix E). We believe their explanation is ultimately incomplete as:
1. They do not clearly delineate the definition of over/underccontidencc used in model calibra- tion with that used in selective classification. This an issue since calibration concerned with absolute marginal (averaged over data samples) properties, whilst selective classifi- cation is concerned with relative conditional (per sample) properties. It is possible to be extremely over/underconnfddntt in the calibration sense, whilst being optimal for SC, or very well calibrated and worse at distinguishing correct vs incorrect samples (Zhu et al., 2024).
2. Although they elucidate some of the mechanics of logit normalisation, showing that logit normalisation reduces the impact of smaller logits uncertainty, they do not link it to
29

---

Published as a conference paper at ICLR 2025
1.00
MSP bin
0.90
0.0.1
0.1-0.2
0.85
0.2-0.3
Vmar (0
0.3-0.4
0.80
0.4-0.5
0.5-0.6
0.75
0.6-0.7
0.7-0.8
0,70
0.8-0.9
0.9-1.0
0.65
4
10
11
12
Viaax
Figure 22: Vmaxx given Umax within bins of similar MSP for ResNet-50 a = 0.2 on the ImageNet evalu- ation set. Generally, V/nxx decreases Umax increases, showing empirically how logit normalisation increases uncertainty for higher Cmax. We calculate the mca of e in 0.2-wide sliding window for samples with Umax within 2std of Umax within the bin (to remove noisy averages).
any observed model behaviour that differs between correct / vs incorrect predictions, instead just speculating that models may be underconfident on certain sets of samples. To explain why an approach improves SC, we need to know how it treats / and X differently to improve the rank ordering of uncertainties between // and X.
On the other hand, our explanation focuses solely the relative ranking of uncertainties between / vs X samples and is able to link the mechanics of logit normalisation to how the behaviour of correct // vs incorrect X predictions differ under label smoothing (Fig. 5 and Result 1).
F.5 ANALYSIS OF NEGATIVE LABEL SMOOTHING (WEI ET AL., 2022)
We consider the framework developed in the main paper to study Generalized Label Smoothing (GLS) with negative smoothing values called Negative Label Smoothing (NLS) suggested by Wei et al. (2022). The definition of GLS is the same as the original from Szegedy et al. (2016), except for the domain of the label smoothing value a, which can take values in (-o0, 1]. Wei et (2022) suggest that using negative values for the label smoothing value can lead to improved performance when training with noisy labels as well as the "clean" " setting. is tempting to consider, as negative ought to reverse the logit suppression described in Eq. (15), However, here show that NLS can lead to unstable training characteristics.
The per-sample cross-entropy between single softmax prediction and the corresponding true categorical distribution म is convex and can be written as follows:
CE( 7)
Σ (1-a)mx AE[1.A]
log Tk.
(29)
K
For a 2 0, which covers vanilla cross entropy (a 0) and regular label smoothing (a > 0) a single global minimum is attained in (1 a) a 1. Here, the gradient of the loss is zero therefore creating a balance and stabilizing training. taking the gradient as in Eq. (13),
aCB IT .)
TH a/K)
(30)
oulput
target
However, in the of negative label smoothing value 0 the target be outside [0, 1], but the softmax output is constrained to [0, 1], thus ग = (1 a) + a 1 not achievable and the gradient will always have some magnitude and there is optimisation minimum.
K
Indeed, if we consider for onc-hot म, the i-th terms s.t. मं = 0 of Eq. (29) will be strictly negative with negative-labes lawiiiv--aael--motteeeee target. Since the log of the softmax probabilities take values
30

---

Published as a conference paper at ICLR 2025
in (-o0, 0], so the loss. Being convex, there cannot exist a local minimum, and therefore, the gradient can never be zero and pushes its value to -oo by reducing the value of the corresponding logit to -00 and pushing the others to oc. We hypothesise that this unbalance explains the instability of the method that users reported on the GitHub repository of the original paper. Unfortunately, we too were not able reproduce the experiments of Wei et al. (2022) and obtain stable enough training runs to perform experiments on NLS and confirm this theoretical analysis.
F.6 EXISTING BENCHMARKS AND TRAINING RECIPES WITH L.
Although we do not exhaustively search all training recipes for all models benchmarked in (Galil et al., 2023; Cattelan & Silva, 2024), we do provide number of examples of evaluated models trained with label smoothing. We also provide links to publicly available training repositories, as not all papers mention label smoothing even when it is used in training. Upon inspection of (Galil et al., 2023; Cattelan & Silva, 2024), these models do in fact seem to underperform at selective classification (and Cattelan & Silva (2024) report that their AURCs benefit from logit normalisation).
EfficientNet (Tan & Le,
EfficientNet-V2 (Tan & Le,
DeiT et http://ww...////o/////o////////a
ma main in/main..ptt..... 01
Swin Transforanfrr (+V2) (Liu et al., 2022b;
L70
ConvNeXt (Liu et al., 2022c); http:s//ggtttii.cccccccc/ot/atss
blob/main/nen htt.:///mimiii...ll. py05
Torchvision (Paszke al., 2019) (various):
tree/main/ wwweeeeeeeeeaaaseiiieeeeeeeeeeee
Galil et al. (2023) state that some of their best performing (at SC) ViT models (Dosovitskiy et al., 2021; Steiner et al., 2022; Chen et al., 2022) are trained with label smoothing (their Tab. However, after inspecting both the original papers and open-source repositories* of the aforementioned work we were unable to find any confirmation of the use of label smoothing.
G ADDITIONAL SEGMENTATION RESULTS
In this section, we complete the picture by providing additional semantic segmentation results to Figs. 1 and 8. As in the main paper, the following segmentation maps are performed by 1esNet-101-basee DeepLab-v3+ trained on Cityscapes with cross-entropy and label-smoothing with a = 0.2. We recall that the models and notebooks used to generate these plots are available on Hugging Face and GitHub, respectively. In these figures, we provide the rank of all the predictions not only errors to show the difference between the model's behavior on its errors but also correct predictions. The rank of the correct predictions is slightly greyed out compared to the rank of the errors X. Fig. 23 presents the full ranks corresponding to the scene used in the main paper.
Figs. 23 and 24 shows that the cross-entropy-basee model predicts label maps in which pixel uncertainty ranks are generally smoother than those provided by the aabel-smoothiingbaae model. The smoothino model exhibits large areas with low uncertainty rankings, which suddenly increases near the boundary of objects. However, the boundaries are sometimes wrongly predicted, leading to (rank-wise) highly confident errors X. The pixels corresponding to the boundaries of objects, where the probubility of error is naturally higher and which are most sensitive to (ground-truth) label errors, have higher confidence (lower uncertainty ranks) according to the LS-based model than the CE-based model. This explains the high-confidence errors in the misclassified boundaries of several objects, either in Figs. and 24, and aligns with the imbalanced
Phttps: :/ thub. wwwpgoog//ooooooooooooooooo.oo.o e resea r oo oontrrnsfrrrer
31

---

Published as a conference paper at ICLR 2025
suppression of Eq. (15). We again see how logit normalisation is able to decrease the confidence of the sobbll--moothingtree trained model on highly confident (bright yellow) errors X.
We also provide Fig. 25 to show that the logit normalisation method is not always successful in solving the ranking of the predictions. In this figure, we see that a large part of the misclassified pixels (lower left) keeps low uncertainty ranks either using the CE-based model the LS-based model while using logit normalisation. However, there are some improvements on the lmoot-sg side, where the original low-ranked boundary errors X are less (rankwise) confident, such as the right border of the median strip. Interestingly, LS-based high confidence errors in Fig. 25 mainly correspond to a change of texture of the median strip likely associated with an uncertainty on the true label and correlate with those of the CE-based model. The choices of match Fig. 13 from optimising AURC4 on the 100 validation images.
H IMPACT AND FUTURE WORK
In this section, we provide some additional discussion about the (practical) impact of our work as well as promising directions for potential future research.
Direct impact. By empirically verifying and analytically elucidating the limited experimental results pertaining to LS in (Zhu et al., 2024) (it is not the focus of that work), we provide strong evidence that the behaviour that LS degrades SC is generalisable (over architectures, data modalities etc.). In particular, we directly analyse the loss, which is common to all settings involving label smoothing. This will help inform practitioners of selective classification when they are designing and deploying systems. Furthermore, by explaining the efficacy of logit normalisation we provide an effective and well-motivated solution to the previously demonstrated problem. We emphasise that when logit normalisation was introduced in (Cattelan & Silva, 2024). it was not clearly explained why logit normalisation was effective on some pretrained models and ineffective on others. In our work, we analytically clarify the mechanism of logit normalisation, opening up the figurative black box, and are able to directly link it to our previous analysis on LS. This provides clear guidance on when and why to use nogiaiooraaiitiinn, giving potential practitioners confidence in the effectiveness of the approach, which is especially important in high-risk safety-critical applications.
Future work. The empirical and analytical results relating LS in our work naturally suggest that other training approaches that alter the lahels such as Mixup (Zhang et al., 2018) 8) may also have similar adverse effects and/or be amenable similar gradient analysis. It also raises the question of how such label augmentations effect problem settings outside of selective classification such as OOD detection or transfer learning. The novel analysis of LS in Sec. may also be useful problem settings beyond SC: Does this aspect of label smoothing (suppressing the max logit less for incorrect predictions) affect generalisation? What about behaviour on OOD data? Could it help explain the behaviour in (Kornblith et al., 2021) where LS results in worse transfer/representatio learning? Can this insight lead to modification of L.S to improve it? Our analysis of the mechanism of logit normalisation in See. 5.1 simply demonstrate that logit normalisation reduces confidence when the max logit
also general is higher. Thus, this knowledge can be potentially applied to other scenarios (e.g. OOD detection). Furthermore, in the case of post-hoc methods, logit normalisation using the p-norm is just one possible option for correcting the effect of LS armed with the insight presented in our work it may be possible to develop superior more principled methods for extracting better uncertainty estimates from logits.
32

---

Published as a conference paper at ICLR 2025
Entropy
Smoothin (a= MSP
DeepLab-V3+ (ResNet-100 Cityscapes
0.8
0.6
Cross-Entropy- Normalizee logits (p - 4)
Soolt-Smoohhigg (a - 0.2) Normalized logits (p 2)
0.0
Lowest uncertainty-ranked most confident)
yellow Correct pixels greyed
Figure 23: We provide all the predicted ranks on our segmentation plots to understand where the model predicted higher relative confidence. The pixel uncertainty rankings provided by the model trained with CE (left) are smoother than those of the model trained with LS 0.2 (right).
Input Image
Ground Truth
1.0
DeepLab-V3- (ResNet 101) Cityscapes
0.8
Prediction Cross Enttr-E
Prediction Label-Smoothing (a 0.2)
Uncertainty Ranking (MSP) Cross-Entropy
Uncertainty Ranking (MSP) Label-S Smoathin (a = 0.2)
0.0
Lowest uncerdainty.rankedd (i.e. most confident)
Correct pixels greyed out
Cross-Entropy MSP
Label- Sbbothing (a= MSP
DeepLab-V3+ (ResNet-110) Cityscapes
0.8
0.6
Cross-Entropy Normalized logits (p = 4)
Label- Saeeel-Saothing (a= 0.2) Normalized logits (p= 2)
0.0
Lowest uncertainty-rankedd (i.e. most confident) errors yellow Correct pixels greyed out
Figure 24: Logit normalisation improves on boundary-related very confident errors but does not completely fix the high (ranked) confidence of the bottom right emrors X.
33

---

Published as a conference paper at ICLR 2025
Input Image
Ground Truth
1.0
DeepLab-V3+ (ResNet: 201) Cityscapes
Prediction Cross- Entrroy
Prediction Laabel-Smootiee 0.2)
Uncertainty Ranking (MSP) Cross-Entropy
Uncertainty Ranking (MSP) Label- Smoothing = 0.2)
0.2
Lowest uncertainty-ranked (i.e. most confident) errors are yellow Correct pixels are greyed out
Cross-Entrope MSP
Label-Smoothing (a = 0.2) MSP
DeepLab-V3+ (ResNet- 101) Cityscapes
Cross-Entropy Normalized logits (p = 4)
Smaething (a 0.2) Normalized logits (p= 2)
0.4
Lawest uncertainty-ranked (i.e. mast confident) errors X are yellow Correct pixels are greyed out
Figure 25: Logit normalisation does not always completely fix selective classification when used model trained with a labol-smo (right).
34


Reviews:
[{"review_id": "cHuYSziZs2", "reviewer": "Reviewer_wn1r", "summary": "This paper investigates the impact of label smoothing (LS) on selective classification (SC), showing that while LS is a popular regularization technique for improving classification accuracy, it degrades SC performance. The authors empirically confirm this degradation across various models and tasks, then analyze the cause at a gradient level, finding that LS suppresses the highest logit differently based on prediction correctness. They propose post-hoc logit normalization as a solution, showing it effectively recovers SC performance degraded by LS.", "questions": "**Q1:** As observed by the authors, LS consistently leads to degraded SC performance, even if it may improve accuracy. What do authors think about the connection between NLS and SC, where NLS refers to Negative Label Smoothing introduced in R1.\n\n**Q2:** In Line 132, it would be beneficial to include the definition of Kronecker delta.\n\n**References:**\n\nR1: To smooth or not? when label smoothing meets noisy labels. ICML 2022.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "**Originality**\n\nThe paper addresses a unique gap by investigating LS's unintended effect on SC. The logit-level gradient analysis provides a fresh perspective on why LS interferes with SC, helping bridge theoretical understanding with observed results.\n\n**Quality** \n\nThe experiments are well-designed, involving diverse datasets (e.g., ImageNet, Cityscapes) and model architectures (e.g., ResNet-50, ViT) to thoroughly validate the findings. The analysis at both empirical and gradient levels strengthens the rigor, making the results compelling.\n\n**Clarity** \n\nThe paper is well-organized, with clear visuals that illustrate LS’s effect on SC. Figures showing SC degradation and the effects of logit normalization make complex points more accessible.", "weaknesses": "My only concern is about the novelty of the contributions.\n\n**Novelty concern**\n\n* As admitted by the authors (Line 212), some of the core conclusions in the main paper are based on previous empirical observations (```for a single value of alpha LS degrades SC for CNN-based image classification```). And the introduced ```broader investigation``` draws the same conclusion as the literature.\n\n* As specified by the authors (Line 416), another main contribution of the paper (```Logit Nomarlisation Improves the SC of LS-Trained Models```) also follows from the literature that ```logit normalization can improve the SC performance of many (but not all) pretrained models```.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ywq6cr5lih", "reviewer": "Reviewer_KVT2", "summary": "This paper shows how label smoothing (LS) can negatively impact selective classification (SC) by combining empirical analysis on large-scale tasks with theoretical insights on logit-level gradients. The authors show that the degradation in SC performance worsens with increased LS strength, and they propose post-hoc logit normalization as an effective method for recovering SC performance lost due to LS.", "questions": "- The experimental analysis is primarily focused on image classification and segmentation tasks, using two specific datasets. Could these findings generalize to other domains, such as text or tabular data?\n- In Figure 3, the degradation impact of LS on SC decreases at high coverage. Is it possible to identify a threshold at which the effect of LS on SC begins to diminish?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The paper provides a well-rounded analysis, both theoretical and experimental, demonstrating how LS degrades SC. By analyzing logit-level gradients, it addresses why LS has this adverse effect on SC, filling an important gap in understanding.\n- Visualizations in the paper effectively illustrate the results, providing an intuitive understanding of how LS impacts SC performance.\n- The findings have practical impact. By showing that LS leads to consistent degradation in SC, the paper suggests that it may partially explain the findings of strong classifiers surprisingly underperform on SC. They further shows that logit normalization can recover the degradation. This could be useful especially in the high-risk tasks like medical or robotics tasks.", "weaknesses": "The experimental analysis is limited to image classification and segmentation tasks, using only two specific datasets. This raises questions about the generalizability of the findings to other domains, such as text or tabular data, where label smoothing and selective classification may behave differently. Expanding the analysis to include diverse data types would strengthen the claim that label smoothing consistently degrades SC performance across various domains.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0Tyle97nvc", "reviewer": "Reviewer_s8Gk", "summary": "This paper explains why logit nomalisation improves SC performance for models trained with label smoothing.", "questions": "N.A", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. I enjoy reading the paper. The presentation and organization look great.\n2. This work provided both empirical and theoretic analysis to answer a research problem and considerably contributed to the field.", "weaknesses": "1. While the authors provided detailed experiments to demonstrate the properties of LS. I think it is also important to quantitatively compare the LS+logit normalisation with other existing solutions under the experimental setting used in this work to demonstrate the effectiveness of such a combination. \n\n2. In figure 7(a), it is hard to find the \"CE logit norm\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DmIkSoPENj", "reviewer": "Reviewer_7wzD", "summary": "Label smoothing (LS) is a popular regularization technique that improves test accuracy, yet previous research has shown that the LS models achieve degraded performance of selective classification (SC)\nBy comparing the formulation of Cross Entropy (CE) model and LS model, this work provides an explicit explanation on why this performance degradation is happening.\nBuilding on the identified reasons for this degradation, the authors further discuss the impact of logit normalization and why it significantly improves SC performance of LS models but not that of CE models.\nThe primary contribution of this paper is the theoretical clarification of a previously unresolved issue. The experimental results are consistent with the theoretical findings presented in the paper.", "questions": "All my confusions have been listed under the weaknesses section. I welcome the authors to provide clarifications on points 1, 3, and 4.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Originality: the paper provides a very detailed and theoretic explanation for the unresolved issue of why label smoothing degrades selective classification.\nQuality & Clarity: their explanations appear to be very reasonable. The theoretical gradient descent expressions align with the actual experimental results, making their work seem like a solid theoretical piece rather than some far-fetched guess.\nSignificance: The authors' explanation addresses a research gap and is likely to be valuable and insightful for future studies in selective classification.", "weaknesses": "While the authors present most of the logic and theory in an understandable and clear manner, unfortunately, the confusion in notation and concepts keeps me oscillating between being confused and understanding. The authors probably need to clarify some logic that they consider obvious but which I find uncertain. These points include:\n\n1. Line 318: I would like to know more clearly the connection between the regularization in Equation 13 and the regularization gradient in Equation 14. Because, in my view, regularization refers to -\\alpha/K, which is an item in L_LS, but the regularization gradient includes \\alpha \\bar{\\pi}_k - \\alpha/K, which is the result of L_LS - L_CE. The relationship between these two needs to be explicitly identified when they have similar names.\n2. Line 319: There is a typo after the second equals sign in the formula.\n3. Line 353: 'This directly impacts softmax-based U such as MSP', but I am not very clear on how. In fact, starting from Part 4.2, I completely lost track of the concepts of uncertainty, various expressions of pi, regularization, and P_error. This makes me feel like I understand but I don't.\n4. Line 371: Again, due to the confusion on the concepts, 'v_max is more strongly suppressed for lower P_error' is not intuitive to me.\n5. The caption for Figure 4 is confusing. Which one is the left, and which one is the right?\n6. The caption for Figure 5 is confusing. What is the purpose of Figure 3? What does it illustrate? It is not clear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)