On the Limitations of Temperature Scaling for Distributions with Overlaps
Authors: Muthu Chidambaram
Rong Ge
Venue: ICLR 2024 poster (2024)
Keywords: calibration
temperature scaling
mixup
label noise
Primary area: learning theory
TL;DR: Temperature scaling is not sufficient for good calibration in distributions that have overlaps between classes, but modified training schemes like Mixup can still handle such distributions.

Abstract: Despite the impressive generalization capabilities of deep neural networks, they have been repeatedly shown to be overconfident when they are wrong. Fixing this issue is known as model calibration, and has consequently received much attention in the form of modified training schemes and post-training calibration procedures such as temperature scaling. While temperature scaling is frequently used because of its simplicity, it is often outperformed by modified training schemes. In this work, we identify a specific bottleneck for the performance of temperature scaling.  We show that for empirical risk minimizers for a general set of distributions in which the supports of classes have overlaps, the performance of temperature scaling degrades with the amount of overlap between classes, and asymptotically becomes no better than random when there are a large number of classes. On the other hand, we prove that optimizing a modified form of the empirical risk induced by the Mixup data augmentation technique can in fact lead to reasonably good calibration performance, showing that training-time calibration may be necessary in some situations. We also verify that our theoretical results reflect practice by showing that Mixup significantly outperforms empirical risk minimization (with respect to multiple calibration metrics) on image classification benchmarks with class overlaps introduced in the form of label noise.

Published as a conference paper at ICLR 2024
ON THE LIMITATIONS OF TEMPERATURE SCALING FOR DISTRIBUTIONS WITH OVERLAPS
Muthu Chidambaram
Rong Ge
Department of Computer Science
Department of Computer Science Duke University
Duke University
muthuêcs duke.
ronggeêcs duke. edu
ABSTRACT
Despite the impressive generalization capabilities of deep neural networks, they have been repeatedly shown to be overconfident when they are wrong. Fixing this issue is known as model calibration, and has consequently received much attention in the form of modified training schemes and post-training calibration procedures such as temperature scaling. While temperature scaling frequently used because of its simplicity, it is often outperformed by modified training schemes. In this work, we identify a specific bottleneck for the performance of temperature scaling. We show that for empirical risk minimizers for general set of distributions in which the supports classes have overlaps, the performance of temperature scaling degrades with the amount of overlap between classes, and asymptotically becomes no better than random when there are a large number of classes. On the other hand, prove that optimizing a modified form of the empirical risk induced by the Mixup data augmentation technique can in fact lead to reasonably good calibration performance, showing that training-timee ttme calibration may be necessary in some situations. We also verify that our theoretical results reflect practice by showing that Mixup significantly outperforms empirical risk minimization (with respect to multiple calibration metrics) on image classification benchmarks with class overlaps introduced in the form of label noise.
INTRODUCTION
The past decade has seen rapid increase in the prevalence of deep learning models across variety of applications, in large part due to their impressive predictive accuracy on unseen test data. However, these models begin to be applied critical applications such as predicting credit risk (Clements et al., 2020), diagnosing medical conditions (Esteva et al., 2017; 2021; Elmarakeby et al., 2021), and autonomous driving (Bojarski al., 2016; Grigorescu al., 2020), is crucial that the models not only accurate but also predict with appropriate levels of uncertainty.
In the context of classification, model with appropriate uncertainty would be correct with probability that is similar to its predicted confidence - for example, among samples on which the model predicts class with 90% confidence, around 90% of them should indeed be the predicted class (a more formal definition is provided in Equation (2.1)).
As concrete (but highly simplified) example, consider the case of applying deep learning model for predicting whether patient has a tife-threatening illness (Jiang et al., 2012). In this situation, suppose our model classifies the patient as not having the illness but does so with high confidence. A physician using this model for their assessments may then incorrectly diagnose the patient (with potentially grave consequences). On the other hand, if the model had lower confidence in this incorrect prediction, a physician may be more likely to do further assessments.
Obtaining such models with good predictive uncertainty is the problem of model calibration, and has seen a flurry of recent work in the context of training deep learning models (Guo et al., 2017: Thulasidasan et al., 2019; Ovadia et al., 2019; Wen et al., 2020; Minderer et al., 2021). In particular, Guo et al. (2017) showed that the simple technique of temperature scaling which introduces only a single parameter to "dampen" the logits of a trained model (defined formally in Section 2) - is powerful procedure for calibrating deep learning models.

---

Published as a conference paper at ICLR 2024
Temperature scaling falls under the class of post-trate trainiraining calibration techniques, which are attractive due to the fact that they can be applied to a black box model without requiring any kind of retraining. However, several empirical works have shown that temperature scaling alone can be outperformed by training-time modifications such as data augmentation (Thulasidasan et al., 2019; Müller et al., 2020) and regularized loss functions (Kumar et al., 2018; Mukhoti al., 2020).
In this work, we try to understand these empirical observations theoretically by addressing the following question:
Can we identify reasonable conditions on the data distribution for which tem- perature scaling provably fails to achieve good calibration, but training-time modifications still succeed?
1.1 MAIN CONTRIBUTIONS AND OUTLINE
We answer this question in the affirmative, showing that temperature scaling cannot handle data distributions with certain class overlap properties, while training-time modifications can. We first define the notions of calibration and temperature scaling relevant to our work in Section 2, and then motivate conditions the models consider in theory in Section 3.1. Namely, we focus on models that interpolate the training data (i.e. achieve zero training error) and satisfy a local Lipschitz-like condition. We also introduce the Mixup (Zhang al., 2017) data augmentation, as well as a modification of it necessary for our theoretical results in Section 3.2.
After establishing the necessary background, show in main results of Section 4 that for classification tasks in which the supports of different classes have overlaps, the performance of temperature scaling degrades with the amount of overlap between classes, while the performance of our modified Mixup procedure remains robust to such overlaps. The key idea behind our theory is that training-time calibration techniques can significantly constrain model behavior regions away from the training data points, and this notion can be extended beyond our Mixup analysis to other augmentation techniques.
Lastly, in Section 5 we show that our theoretical results accurately reflect practice by considering both synthetic data and image classification benchmarks. In Section 5.1 we show that the performance of empirical risk minimization (ERM) combined with temperature scaling quickly degrades on simple 2-class high-dimensional Gaussian dataset as we increase the overlap between the two Gaussians. Similarly, in Section 5.2 we show that the same phenomenon occurs standard datasets when increase the amount of label noise in the data (thereby increasing class overlaps).
1.2 RELATED WORK
Calibration in deep learning. The calibration of deep learning models has received significant attention in recent years, largely stemming from the work of Guo et al. (2017) which empirically showed that modern, overparameterized models can have poor predictive uncertainty. Follow-up works (Thulasidasan et al., 2019; Rahaman & Thiery, 2021: Wen al., 2021) have supported these findings, although the recent work of Minderer et al. (2021) showed that current state-of-the-art architectures can be better calibrated than the previous generation of models.
Methods for improving calibration. Many different methods have been proposed for improving calibration, including: logit rescaling (Guo et al., 2017), data augmentation (Thulasidasan al., 2019; Müller et al., 2020), ensembling (Lakshminarayanan et al., 2017; Wen et al., 2020), and modified loss functions (Kumar al., 2018; Wang al., 2021). The logit rescaling methods, namely temperature scaling and its variants (Kull et al., 2019; Ding et al., 2020). constitute perhaps the most applied calibration techniques, since they can be used on any trained model with the introduction of only a few extra parameters (see Section 2). However, we show in this work that this kind of post-training calibration can be insufficient for some data distributions, which can in fact require data augmentation/moddine
loss functions to achieve good calibration. We focus particularly on Mixup (Zhang et al., 2017) data augmentation, whose theoretical benefits for calibration were recently studied by Zhang al. (2021) in the context of linear models and Gaussian data. Our results provide a complementary perspective to this prior work, as we address a much broader class models and a different class of data distributions.
2

---

Published as a conference paper at ICLR 2024
2 THEORETICAL PRELIMINARIES
Notation. We use [k)] to denote (1, 2...., k) for a positive integer k. We consider k-class classification and use X to denote dataset of N points (Ia, ys) sampled from distribution (XX, Y) whose support Supp(πT) is contained E [k]. We use я and my to denote the respective marginal distributions of π, and use Ty to denote the conditional distribution π (XX Y y). We use d(A, B) to denote the Euclidean distance between two sets A, B R dKL ( T2) to denote the KL divergence between two distributions Tı and T2, and faa for the Lebesgue measure on IR For a function g : R' - RR,, use g" to denote the 1thh coordinate function of g. Lastly, use ((-) to denote the softmax function, i.e. of (x(x)) exp(g" (tr))/ k are sufficiently large, and that N = ([poly(k)) for some large degree polynomial poly(k).
exp(g' (r)). In everything that follows, we assume both N and
Calibration. In a classification setting, we say that a model is calibrated with respect to the ground-truth probability distribution ग if the following conditional probability condition holds:
P(Y (()())) = =((((X)
(2.1)
Equation 12.1) captures the earlier mentioned intuition that, when our model predicts the probability distribution (X)) over the classes [k). the true probability distribution for the classes is also ())())) It is straightforward to translate Equation (2.1) into notion of miscalibration by consider- ing the expectation of the norm of the difference between the left and right-hand sides. In practice, however, it is common in multi-class classificatinn to focus only on calibration with respect the top predicted class argmax, 6² (g(X). This leads to the top-class expected calibration error (ECE), which we define simply as ECE below:
ECE(g) = TE, (x,))~
B|Y E argmax (x(()) max O (((((((()) = =p" 1 pp' vE[k]
(2.2)
yE(k]
Top-class ECE is only one of several common notions used to measure calibration, and is known to have several theoretical and empirical drawbacks (Blasiok et al., 2023). In our theoretical work, we opt instead work with the expected KL divergence Ex Ex~rx ( x))-)] our notion of calibration error. Note that such a notion is difficult to estimate in reality as we do not know TTY X). However, will consider cases where the post-training calibration procedure has access to TY X). In such cases expected KL divergence is a more accurate characterization as minimizing implies the full calibration condition of Equation (2.1), while minimizing Equation (2.2) does not (we will only be calibrated with respect to the top class).
With a notion of miscalibration in hand, we consider methods to improve the calibration of a trained model g. One of the most popular (and simplest) approaches is temperature sealing (Guo et al., 2017), which consists of introducing a single parameter that is used to scale the outputs of g. The value of T is obtained by optimizing the negative log-likelihood on calibration dataset X
T argmin )'(0,s0)
Σ
log tth (e (g(x:)/T)
(2.3)
[Real
For our results, we will in fact consider an even more powerful (and impractical) form of temperature scaling in which we allow access to the ground-truth distribution x:
T argmin Ex-ax (x X), oo (g(x)/T))
(2.4)
fE(0.00)
We will henceforth refer to an optimally cunperature-scaled model with respect to Equation (2.4) as gr. We will show in Section 4 that even when we allow this "oracle" temperature scaling, we cannot hope to calibrate models y that satisfy some empirically regularity properties.
3 TRAINING APPROACHES
3.1 EMPIRICAL RISK MINIMIZATION
In practice, models are often trained via empirical risk minimization (ERM). Due to the large number of parameters in modern models, training often leads to an interpolator that is very confident on every training data point (2t, y:) X, as we formalize below:
3

---

Published as a conference paper at ICLR 2024
Definition 3.1. [ERM Interpolator] For dataset X, say that model g is an ERM interpolator if for every (2 91) E X there exists a universal constant C₁ such that:
min 9 (x:)) (xx) > log k and a
max 9 (2x, - g" < C. n,
(3.1)
Equation (3.1) is slightly stronger than directly assuming 4d (g(z;)) 2 1; however, this type of significant logit separation is commonly observed in practice. Indeed, to further justify Equation (3.1) we train ResNeXt-50 (Xie et al., 2016) models on CIFAR-10, CIFAR- 100, and SVHN and examine the means of the max and second max logit over the training data; results are shown in Table 1.
Logit
CIFAR-10
CIFAR- 100
SVHN
Max
14.5755
18.6198
11.5285
Second Max
0.3650
5,4575
-4.1904
Table I: Comparison of means of max and second max logit over different datasets, with ResNeXt-50 models trained as described in Section 5.
In addition to the interpolation condition Definition 3.1, we also constrain our attention to ERM models g that satisfy mild lopal-Lipsehitz- like condition.
Definition 3.2. [7-Regular] For point (aa, Us) E X. leuting L be universal constant, we define:
B, = {x R ||ا; - x|| 7
(3.2)
(I,))) = (x E B&d 1 - 9 (x)) < L7}
(3.3)
We say that model is 7-regular over set U if there exists a class E [K] and @ (U)N) points (2r,, y) E X with 2, U such that (X E (x,) X E (x.)) 2 O(1kk)
Basically, Definition 3.2 codifies the idea that the model logit nd does not change much in small enough neighborhood of each over a given set U, with high probability. Experiments providing empirical justification for this assumption are provided in Appendix B.I. We will show in Theorem 4.5 that satisfying Definitions 3.1 and 3.2 is sufficient for poor calibration for wide class of data distributions, even when using temperature scaling with access to the ground truth distribution oracle.
3.2 MIXUP
In contrast, we can show that if we consider models minimizing a Mixup-like training objective instead of the usual negative log-likelihood in Equation (2.3), can prevent major calibration issues.
Let DA denote a continuous distribution supported on [0, 1] and let (A) = Ar, + (1 A)æ, (using Zij when À is clear from context) where (I;, vi), (æj, My) X. Then may define the empirical Mixup cross cross-entropy Jmix (g, X, Dx) as:
Jmix(g, (g, H,) = N2
1
Σ [E[N] (E[N]
Σ
EAND, [Alog (g(zts) + (1 A) log di' (g(Z+,))) (3.4)
Essentially, minimizing Equation (3.4) forces a model to linearly interpolate between its predictions 6"h (g(x:)) and 63 (g(2,)) over the line segment connecting the points z; and zy. This already provides some intuition for why Mixup-optimal models can be better calibrated: their predictions can change quickly one moves away from the training data, avoiding issues that stem from 7-regularity of the logits combined with interpolation.
However, the line segment constraints of Jmmin (g. X, Dx) will not be enough to make this intuition rigorous when the data is in IRd with d > since in this case line segments are measure zero sets with respect to faz We will thus augment Mixup to work with convex combinations of d + points as opposed to two, and refer to this new objective as d-Mixup.
In generalizing from Mixup to d-Mixup, it is helpful from theoretical standpoint constrain the set of allowed mixings Mx(X') C [N]++]_ We will consider only mixing points at most some constant distance away from one another, and we will also preclude mixing points that are too highly
4

---

Published as a conference paper at ICLR 2024
correlated.' The precise definition of Ma(XX) can be found in Definition A.1 of Appendix A.2; omit it here due to its technical nature.
Now let Dx denote continuous distribution supported the d-dimensional probability simplex A C pd+1 Defining 2a (A) EjeØ+11 Ajio, for À E supp(Dx,a) and E &Ma(A), we can define the empirical d-Mixup cross-entropy Vmix,a X, Px,a):
X, [Ma( xEMa(x))
Σ
Ex-DA,d
Σ JE[
A3 log " (0(20 (A))
(3.5)
We will henceforth use Xmix,d denote the set of all 2g- The main benefit of introducing the set Ma(X) instead of just generalizing Equation (3.4) to mixing over [N] is that it allows to use a reparameterization trick with which we can characterize the d-Mixup optimal prediction at every mixed point že- We state only informal version of this result below and defer formal statement and proof to Appendix A.2.
Lemma 3.3. [Informal Optimality Lemma] Every g" arginf y Jmix a(g, X, Dx.a) (where the arginf is over all extended Rd -aluudd functions) satisfies (²) = Ex(2) Ex(2) for almost every 2, E Xmix.d, where (2) corresponds to the expected weight of class points over all mixing sets a E MM) from which we can obtain 2.
We note that this lemma is analogous to Lemma 2.3 in the work of Chidambaram et al. (2021), but avoids restrictions on the function class being considered and is non-asymptotic. Since we can characterize optimal predictions over Xmix,d we can define d-Mixup interpolators follows.
Definition 3.4. [d-Mixup Interpolator] For a dataset X, we say that g is d-Mixup interpo- 2² (z) + O(1/k) for almost every Xmix, and y [k). with
lator if 6d (g(2) 9" E arginf, J a Dx.a).
In Theorem 4.6, we will show that d-Mixup interpolators can achieve good calibration on subclass of distributions for which ERM interpolators perform poorly.
Remark 3.5. In practice it is unreasonable to mix conjecture that due the structure of practical models (i.e. neural networks), even mixing two points as in traditional Mixup is sufficient for achieving neighborhood constraints like those induced by d-Mixup. We introduce d-Mixup because make such structural assumptions in our theory.
points when large. However, we
4 MAIN THEORETICAL RESULTS
In this section, we show that even for simple data distributions, ERM interpolators can prevent temperature scaling from producing well-calibrated models, while modifications the training process (Mixup) can potentially address this issue. Prior to proving our main results, we begin first with I-dimensional example that contains the key ideas of our analysis. The full proofs of all results in this section can be found in Appendix A.
4.1 WARM-UP. A SIMPLE 1-D EXAMPLE
Definition 4.1. [a-Overlapping Intervals] Let rly) denote the parity of a nonnegative integer y and let By = [(y 1)/2jk ar 1) for [k), where [0, is parameter of the distribution. Then we define 7 (XX, Y) to be the distribution on IR x [k] such that Ty is uniform over [k] and (XX] = y) is uniform over [8,, 8y
Definition 4.1 corresponds to distribution in which consecutive class-conditional densities are supported on overlapping intervals (whose overlap is determined by the parameter a:) with a spacing of k between each pair of classes (see Figure 1). The spacing of & between pairs of classes is introduced only to simplify the d-Mixup analysis; it not necessary for proving the negative results regarding ERM interpolators, and will not feature when we generalize to Definition 4.4.
'This means we do not mix points with themselves in d- d-Mixup; however, when xx has density, this makes little difference since mix in neighhorhood of any point.
5

---

Published as a conference paper at ICLR 2024
Class
Class 3
Class
Class 4
Figure Visualization of Definition 4.1 for the case ls
Our first result shows that when considering ERM interpolators for distributions of the type in Definition 4.1, so long as the interpolators satisfy 7-regularity for sufficiently large 7, they will be poorly calibrated in the overlapping regions of the data distribution.
Proposition 4.2. Let X' consist of N i.i.d. draws from the distribution π specified in Definition 4.1, with parameter a. Then with probability at least - k (xxp((9((N/k))) over the randomness of X, the set of all models g that are ERM interpolators for X and k/(4 (-rgular over each overlapping region in supp(rr x non-empty (in fact, uncountable). Furthermore, the predictive distribution (Y X) = o² (XT(X)) of the optimally temperature-scaled model 9r for any E S satisfies:
Ex-ax (T x)) [))] @((1 11/k) log
(4.1)
Thus, for a (1), even with oracle temperature scaling every g E is asymptotically no better than random. In contrast, as the separation a 1, the bound in Equation (4.1) becomes vacuous.
Proof Sketch. We can show that non-trivial using Chernoff bound arguments, and then use rreuuatryy to show that there is significant fraction of supp(Tx x) on which every 9 E S predicts incorrect probabilities. The key idea is then that temperature scaling will only improve incorrect predictions for ERM interpolators
uniformly random (i.e. 1/k), whereas the correct prediction in an overlapping region 1/2 for each of the overlapping classes.
On the other hand, for d-Mixup, each point in the overlapping regions can be obtained as mixture of points from the overlapping classes, so we will have non-trivial probabilities for both classes.
Proposition 4.3. Let be as in Proposition 4.2 and p(k) denote polynomial in k of degree least one. Then taking 2 to be uniform, every 1-Mixup interpolator g for X' with the property that (g(())) S (l))())) for every x supp(mx) Xmix,1 and y E [R:] satisfies with probability least 1 3.3 exp( (N((())))))
Ex [dKı (T(( X),, #(Y [x)]] < ((1)
(4.2)
Note that this result independent of the separation parameter a.
Proof Sketch. We can show with high probability that Xmix, covers most of suрр(πx)) uniformly, and then we can use Lemma 3.3 to precisely characterize the 1-Mixup predictions over X
The added stipulation that (g(x)) < 1 11--N1 ((11poly((/) (k)) is necessary, since cannot hope to prove an upper bound g is allowed to behave arbitrarily on supp(=x) Xmix. and we also expect this in practice due the regularity of models considered. A key takeaway from Proposition 4.3 is that the upper bound on the Mixup error we obtain independent of the parameter a of our distribution; will see that this is also the case in practice in Section 5.
4.2 GENERALIZING TO HIGHER DIMENSIONS
By extending the idea of overlapping regions in sT) from our 1-D example, we can generalize the failure of r-rrgular ERM interpolators to higher-dimensional distributions.
Definition 4.4. [General Data Distribution] Given a parameter E [0, 1]. we define to be any distribution whose support contained in Rd x [k] satisfying the following constraints:
1. (Classes roughly balanced) Ty (Y y) O(1kk)
2. (Constant class overlaps) Letting M denote nonnegative integer constant, there exist ((k) classes for which there are classes (v), s2(y), ..., 8m with Ty (supp(TT)) supp(= 2 a, and all other s' E [bc] satisfy x (s(ppp(T)))
Sm(U) (y) for some 1 S m S M
supp(m>)) 0.

---

Published as a conference paper at ICLR 2024
3. (Overlap density is proportional to measure) 7y(X E A) ((ua(A)) and Tad(s) (X A) = (J)a(A)) for every A C supp(my) O (u)p(TT
Definition 4.4 is quite broad in that we make assumptions on the behavior of the class-conditional densities outside of the overlapping regions. We now generalize Proposition 4.2.
Theorem 4.5. Let X' consist of N i.i.d. draws from any distribution я satisfying Definition 4.4, and let r I be such that the sphere with radius in R has volume k/22MN) Then the result of Proposition 4.2 still holds for the set Sa of ERM interpolators for X' which are r-regular over each overlapping region in supp(
To generalize Proposition 4.3, however, need further restrictions significant spacing between overlaapag classes (as in Definition 4.1), and we need to restrict the class-conditional densities such that mixings in Xmix,d are not too skewed towards small subset of classes. The precise formulation of this assumption can be found in Appendix A.2.
Mainly, need have
Theorem 4.6. Let X' consist of N i.i.d. draws from any distribution π satisfying Definition 4.4 and Assumption A.3, and let p(k) be as in Proposition 4.3. Then the result of Proposition 4.3 still holds when considering d'- d-Mixup interpolators for X' where the mixing distribution D Dx,d is uniform over the d-dimensional probability simplex.
EXPERIMENTS
We now verify that the theoretical phenomenon of ERM calibration performance decreasing with distributional overlap also manifests in practice. For all of the experiments in this section, train ResNeXt-50 models (Xie et al., 2016) due to the continued use of the ResNet (He et al., 2015) family of models as strong baselines (Wightman et al., 2021), and we report the mean performance over 5 random initializations with standard deviation error bounds. For metrics, we report negative- log- log-likelihood (NLL), ECE with binning using 15 uniform bins (as was done by Guo al. (2017)), and also top-class adaptive calibration error (ACE) (Nixon et al., 2020) which is ECE but computed using equal-weight bins. We also use confidence histograms and reliability diagrams calibration. The former corresponds to histograms of the predicted probabilities for the positive class (or top-class probabilities in the multi-class setting), while the latter correspond to binning the top-class probability predictions and plotting the average accuracy in each bin. These visualizations are generated using the calibration library of Küppers et al. (2020).
visualize
All models were trained for 200 epochs using Adam (Kingma & Ba, 2015) with the standard pyperaarametess of 31 = 0.9, 32 = 0.999, a learning rate of 0.001, and a batch size of 500 on single A5000 GPU using PyTorch (Paszke et al., 2019). We did not any additional training heuristics such as Dropout (Srivastava et al., 2014). In preliminary experiments, we found that minor changes these hyperparameters did not affect horizon was sufficiently long (so as to achieve interpolation of training data). For each training dataset considered in this section, set aside 10% of the data for calibration.
experimental results so long the training
5.1 SYNTHETIC DATA
We first consider a synthetic data model which closely resembles Definition 4.1: binary classification on overlapping Gaussian data. Namely, we consider class I points be drawn from (0, Id300) (i.e. 300-dimensional mean zero Gaussian), and class 2 to be drawn from N N(,, Id The probability of overlap between the two classes controlled by the choice of the mean f of class 2.
We consider μ E (0.25 1, 0.05 1, 0.00 }} (where 1 here denotes the all-ones vector in 10300 to cover range of overlaps (0.05 corresponds roughly 1/vd here). We sample 4000 training data points and 1000 test data points according to the aforementioned distribution, with class I and class being chosen uniformly.
We compare the performance of ERM with temperature scaling (TS) to that of Mixup without temperature scaling (which means the set aside calibration data is not used). Our implementation of temperature scaling follows that of Guo et al. (2017). For Mixup, we consider the usual Mixup formulation with uniform mixing distribution as well as d-Mixup with 2, (i.e. mixing 3 and
7

---

Published as a conference paper at ICLR 2024
points respectively) with uniform mixing (although we do not enforce the technical conditions in Definition A.1 for d-Mixup). Results are reported in Table
Training Method
Metric
= 0.25
= 0.05
0.01
NLL
0.2593 #0.0341
3.7331 +9.4986
4.2977
#0.5641
ERM TS
ECE
0.0407 +0.0026
0.4047 #0.0173
0.4686
#0.0059
ACE
0.3114 +0.0520
0.2755 #0.0225
0.2972
#0.0167
NLL
#0.0111
0.7487
#0.0356
0.8247
#0.0400
Mixup
ECE
0.1161 #0.009
0.1481
=0.0227
0.1945
#0.0204
ACE
0.1629
+-001192
0.1456
+000218
0.2174
+.00416
NLL
0.2205
#0.0338
0.7197
=0.0150
0.7996
#0.0331
2-Mixup
ECE
0.1490
+0.0226
0.1169
+0.0095
0.1842
+0.0292
ACE
0.1918
+0.0282
0.1192
#0.0175
0.2085
#0.0324
NLL
0.2729
+0.0299
0.6784
#0.0079
0.7444
#0.0254
4-Mixup
ECE
0.1979
#0.0199
0.0549
#0.0164
0.1018
#0.0305
ACE
0.1804 +0.0103
0.1374 +0.0370
0.2279 +0.0408
Table Mean NLL, ECE, and ACE on test data over runs with 1 standard deviation error bounds on 2-class Gaussian data.
Our results are what would expect from theory the NLL of the ERM model increases sharply as we increase the distributional overlap, while the Mixup models have comparatively small increases in NLL. Additionally, we find that the d-Mixup models perform better than the ERM and regular Mixup models as we increase the amount of overlap. Of note the fact that this occurs even though this synthetic dataset consists of only classes, suggesting that it perhaps possible improve our theoretical observations to a non-asymptotic setting with more precise analysis.
reasonable concern with comparing model NLL is that it can correlate strongly with generalization performance. However, we can verify that in fact the underlying issue is one of calibration and of generalization by comparing confidence histograms and reliability diagrams for the models being considered. Figure 2 compares the ERM TS model to the 4-Mixup model; we see that the average positive class confidence for 4-Mixup model concentrates around 0.5, while ERM confidence is bimodal 0 and (i.e. extremely overconfident, even when making mistakes). and that the test accuracy both models is roughly the same. Also, the observed underconfidence at lower probabilities for both models can be attributed to the lack of predictions those probabilities. Additional plots for other models and levels of separation can be found in Appendix B.2.
Confide De Hatogim
Confidence Hatogram
Agg Ariutry Noliii binert tarples
Cullerrre Peletive drosee taiquin
2.0
C.K
10
Centebero
RelaMilty Di Dirmm
Relisblity Diagram
Perfect Caliraaiin
Callrrtan
0.0
14
18
12
0.6
3.0 00
(a) ERM TS
(b) 4-Mixup
the Figure 2: Confidence histograms and reliability diagrams for ERM + TS and 4-Mixup models on
test Gaussian data with μ = 0.01 * 1, using 15 bins. Overall accuracy on the test data, as well as average confidence, are reported as dashed lines on the histograms.
We observe similar phenomenon that of the NLL results when comparing model ECE, although we note that the ACE performance remains roughly the same across different overlap levels (this is outside the scope of our theory, since this is byproduct of the binning procedure used for ACE). The
8

---

Published as a conference paper at ICLR 2024
main takeaway from this set of experiments is that Mixup, and particularly the d-Mixup variants, have a significant regularization effect on model confidence. This regularization effect is more pronounced increase d (as would expect), but comes at a trade-off of additional computational overhead (computing convex combinations of + points per batch).
5.2 IMAGE CLASSIFICATION BENCHMARKS
We can also verify that the phenomena observed in synthetic data translates to the more realistic benchmarks of CIFAR-10, 10, CIFAR-100, and SVHN. To artificially introduce (more) overlaps in the datasets, add label noise to the training data. In order to follow the style of Definition 4.4, our label noise procedure consists of randomly pairing up each class in the data with another distinct class, and then flipping the label of the original class points be the label of the paired up class according to the desired level of label noise.
Results for ERM TS and Mixup terms of test NLL are shown in Table 3. Results in terms of ECE and ACE can be found in Appendix B.3; they follow similar trends to NLL. We additionally trained d' d-Mixup models on the same data, but we found that d > 2 led underconfidence on these datasets, so we report just the results for Mixup. We also analyzed the relative improvement of TS over baseline ERM, as well as Mixup TS; these results can be found in Appendix B.4.
Dataset
Label Noise
ERM+ + TS (NLL)
Mixup (NLL)
0%%
2.0109 +0.1750
0.8190 +0.0349
CIFAR-10
25%
3,2091 #0.2042
1.3102 #0.0311
50%
5,9533 70.9
1.7729
+0.0773
0%
5,4969 10.8198
2.4608
#0.0482
CIFAR-100
25%
6.3767 +:0.4089
2.9753
+0.1004
50%
8,0224 +0.6025
3.5443
#0.0367
0%%
0.6919 +-01178
0.3441
+0.0348
SVHN
25%
1.6797 #0.2442
0.8546 +0.0358
50%
4.1807 #0.3642
1.5030 #0.0367
Table 3: Mean NLL over 5 runs with standard deviation error bounds image classification benchmark test data with varying levels of label noise.
We see in Table 3 the same behavior we observed in Section 5.1; the NLL for the ERM + TS models jumps quickly with increasing label noise while the Mixup NLLL increases comparatively slowly. While in the cases of CIFAR-10 and CIFAR-100 some of this gap can be associated to improved test error of the Mixup models, we find for SVHN that ERM actually outperforms Mixup in terms of test eиror and still has massive gaps in terms of NLL. Furthermore, Mixup also dominates in terms of ECE and ACE across all datasets and virtually every label noise level. Finally, comparing confidence histograms and reliability diagrams in the manner of Section 5.1 shows the same behavior on the classification benchmarks: ERM + TS confidences cluster around 1 while Mixup confidences are more evenly distributed, contributing to improved calibration.
show in Appendix B.3 that
6 CONCLUSION
The key finding of our work is that when considering interpolating models on data distributions with overlaps, temperature--salinngyyyy
methods alone can be insufficient for obtaining good calibration, both in theory and in practice. We show empirically that such interpolating models have very skewed confidence distributions (i.e. always predicting near 1), which re-affirms earlier empirical findings in the literature and suggests that temperature scaling is forced to push such models to be closer to random (as seen in Theorem 4.5). On the other hand, we also show that minimizing the Mixup- augmented loss instead of the standard empirical risk can lead to good calibration even when the data distribution has significant overlaps. Towards this end, we introduce d-Mixup, and show that mixing more than two points with Mixup can in fact be beneficial for calibration due to the added regularization, particularly in high noise/overlap settings. One clear direction suggested by our work is the idea of developing better neighborhood constraints around training points; we study the constraints introduced by Mixup, but we anticipate there are likely better alternatives for improving calibration that rely on non-linear constraints, and leave this future work.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
Rong Ge and Muthu Chidambaram are supported by NSF Award DMS-2031849, CCF-1845171 (CAREER), CCF-1934964 (Tripods), and a Sloan Research Fellowship. Muthu would like thank Kai Xu for helpful discussions during the early stages of this project.
ETHICS STATEMENT
Due to the largely theoretical nature of this work, do not anticipate any misuses of our results or code. That being said, we do note that our work further emphasizes the calibration issues that exist with present day models, which is an important reminder for those using these models for critical use cases.
REPRODUCIBILITY STATEMENT
All of the proofs and full assumptions for the theoretical results in this paper can be found in Appendix A. Code used to generate all the plots in this paper can be found associated Github Repository: betps://giitee com/201 4mch daabdaameeeee scaling ling- Aamitationa
REFERENCES
Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp, Prasoon Goyal, Lawrence D. Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, Xin Zhang, Jake Zhao, and Karol Zieba. End to end learning for self-driving cars, 2016.
Jaroslaw Blasiok, Parikshit Gopalan, Lunjia Hu, and Preetum Nakkiran. A unifying theory of distance from calibration, 2023.
Muthu Chidambaram, Xiang Wang, Yuzheng Hu, Chenwei Wu, and Rong Ge. Towards understanding the data dependency of mixup-style training. CoRR, abs/21 10.07647, 2021. URL https: //a //arxiv.ort org abs/ 221./211100226111
Jillian M. Clements, Di Xu, Nooshin Yousefi, and Dmitry Efimov. Sequential deep learning for credit risk monitoring with tabular financial data, 2020.
Zhipeng Ding, Xu Han, Peirong Liu, and Mare Niethammer. Local temperature scaling for probability calibration. CoRR, abs/2008.05 1005 2020. URL https: http:ps///aaaavv.orrraaaa
05105.
Haitham A Elmarakeby, Justin Hwang. Rand Arafeh, Jett Crowdis, Sydney Gang, David Liu, Saud H AlDubayan, Keyan Salari, Steven Kregel, Camden Richter, et Biologically informed deep neural network for prostate cancer discovery. Nature, 598(7880) 352, 2021.
Andre Esteva, Brett Kuprel, Roberto Novoa, Justin Ko, Susan M Swetter, Helen M Blau, and Sebastian Thrun. Dermatologist-leva classification of skin cancer with deep neural networks. nature, 542(663): 2017.
Andre Esteva, Katherine Chou, Serena Yeung, Nikhil Naik, Ali Madani, Ali Mottaghi, Yun Liu, Eric Topol, Jeff Dean, and Richard Socher. Deep Beaming-enablee medical computer vision. NPJ digital medicine, 4(1):5, 2021.
Sorin Grigorescu, Bogdan Trasnea, Tiberiu Cocias, and Gigel Macesanu. survey of deep learning technigues for autonomous driving. Journal of Field Robotics, 37(3):362-386. 2020.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. Proceedings of the 34th International Conference on Machine Learning Volume 70, ICML' 17. pp. 1321-1330. JMLR.org. 2017.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CoRR, abs/1512.03385, 2015. URL http::t/tt//iiiii www.wiioooororreeeeooe org/.rrgg/a....
/1512 /as///bs//55 03385.
10

---

Published as a conference paper at ICLR 2024
Xiaoqian Jiang, Melanie Osl, Jihoon Kim, and Lucila Ohno- Machado. Calibrating predictive model estimates to support personalized medicine. Journal of the American Medical Informatics Association, 19(2):263-273 274, 2012.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference Leurning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings. 2015. URL http: //aaxxi..rgg/a///// ab41g//bbb/1111...// 6980.
Meelis Kull, Miquel Perelló-Nieto, Markus Kängsepp, Telmo Menezes e Silva Filho, Hao Song, and Peter A. Flach. Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with dirichlet calibration. CoRR, abs/1910. 12656, 2019. URL httttpp//axiiiii org abs/ 1910. 12 12656.
Aviral Kumar, Sunita Sarawagi, and Ujjwal Jain. Trainable calibration measures for neural networks from kernel mean embeddings. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 2805-2814. PMLR, 10-15 Jul 2018. URL https: httpas::/pccceeenppppee mlr. wwr.sppess/e.. /v//////////// /ttp/////////aarleeee wamakumaaaaare 8aa. html.
Fabian Küppers, Jan Kronenberger, Amirhossein Shantia, and Anselm Haselhoff. Multivariate confidence calibration for object detection. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2020.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles, 2017.
Matthias Minderer, Josip Djolonga, Rob Romijnders, Frances Hubis, Xiaohua Zhai, Neil Houlsby, Dustin Tran, and Mario Lucic. Revisiting the calibration of modern neural networks. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Ad- vances in Neural Information Processing Systems, volume 34, pp. 15682-15694, Curran Asso- ciates, Inc., 2021. URL ItL/ neurips..rrerppreee........r..ee wwtpccerpaapppieeeeeeeeeeeeee firerrrfflleeeeeeeeeeeeeeeereere wesle/ee//////////e 2021/file http://e.2.223300000222555522a22
ppf..
Jishnu Mukhoti, Viveka Kulharia, Amartya Sanyal, Stuart Golodetz, Philip H. S. Torr, and Puneet K. Dokania. Calibrating deep neural networks using focal loss, 2020.
Rafael Müller, Simon Kornblith, and Geoffrey Hinton. When does label smoothing help?, 2020.
Jeremy Nixon, Mike Dusenberry, Ghassen Jerfel, Timothy Nguyen, Jeremiah Liu, Linchuan Zhang, and Dustin Tran. Measuring calibration in deep learning, 2020.
Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, D. Sculley, Sehastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. In H. Wallach, H. Larochelle, Beygelzimer, d'Alché- Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL
negrissenerppppppp cc/paper hiles/paaer/201199 http:///1lee/55ab4400000 /ttp:////11.11.......06666666611
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Ed- ward Z. Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, herf-peeformance deep learning library. CoRR, abs/1912.01703. 2019. URL /tatt:://aree www.riii.....oo.eoooooooooo /1912. 01 0.3.
Rahul Rahaman and Alexandre H. Thiery. Uncertainty quantification and deep ensembles. 2021.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way prevent neural networks from overfitting. Journal of Machine Learning Research, 15(56):1929-1955 2014. URL http:/ jmir. wtw.rorrg/oo.o.ootoooo...o.oocrt hegg/appers/viie srivastaval 4a hhml..
11

---

Published as a conference paper at ICLR 2024
Sunil Thulasidasan, Gopinath Chennupati, Jeff A Bilmes, Tanmoy Bhattacharya, and Sarah Michalak. On mixup training: Improved calibration and predictive uncertainty for deep neural networks. Advances in Neural Information Processing Systems, 12:13888-1389 2019.
Deng-Bao Wang, Lei Feng, and Min-Ling Zhang. Rethinking calibration of deep neu- ral networks: Do not be afraid of overconfidence. In M. Ranzato, A. Beygelz- imer, Y. Dauphin, P.S. Liang, and Wortman Vaughan (eds.), Advances in Neural In- formation Processing Systems, volume 34, pp- 11809-11820. Curran Associates, Inc., 2021. URL https: htap::://rroceee http///eooeednggseeeeeeee ngs. neurips cc/paper_fiiiccccccceeee wttpsssssssss.s..eeeeeeeee.. file/ 61 f f3a 6dbc91 20ea (75754442277775222222822222 4e- 4e-Paper.
Yeming Wen, Dustin Tran, and Jimmy Ba. Batchensemble: An alternative approach to efficient ensemble and lifelong learning, 2020.
Yeming Wen, Ghassen Jerfel, Rafael Muller, Michael W. Dusenberry, Jasper Snoek, Balaji Lak- shminarayanan, and Dustin Tran. Combining ensembles and data augmentation harm your calibration, 2021.
Ross Wightman, Hugo Touvron, and Hervé Jégou. Resnet strikes back: An improved training procedure timm, 2021.
Saining Xie, Ross B. Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. CoRR, abs/161 1.05431, 2016. URL http:///aaax org/abs/ /111. 05431.
Hongyi Zhang, Moustapha Cissé, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. CoRR, abs/1710.09412, 2017. URL http::/aaxixs http:///aaaii....aaaa////// org.abs////// 09412.
Linjun Zhang, Zhun Deng, Kenji Kawaguchi, and James Zou. When and how mixup improves calibration. CoRR, abs/2102.06289, 2021. URL https: //tp//aaivv////////////////aaal/ org/abs/g//aaae /2102 06289.
12

---

Published as a conference paper at ICLR 2024
A FULL PROOFS
Here we provide full proofs of all results in Section 4. For convenience, we first recall all of the relevant definitions and assumptions from Sections and 4.
Definition 3.1. [ERM Interpolator] For dataset X, we say that a model g an ERM interpolator for every (z,, Ve) E X there exists a universal constant C₁ such that:
min g (ara) (x:) > log k and max 9² (2;) C. r,
(3.1)
*
Definition 3.2.
[y-Rearlrr] For point (It, yı) X, letting L be universal constant, we define:
(x,) = {a E R² ||ا - 2|| } (x,), = {z E I² 1gth - 9th (x) - Ln}
(3.2)
(3.3)
We say that model is y-regular over set U if there exists a class E [kc] and ( (U)N) points (24, 3) E X with 2; U such that (X (x,))) X E (x,)) 2 1 0(1kk)
Definition 3.4. [d-Mixup Interpolator] For a dataset X, we say that g is d-Mixup interpo- (=)) + O(1/k) for almost every 2 E Xmixx and E [k], with
lator if 6² (g(2) g" E arrinnf J X, Xxa).)
Definition 4.1. [a-Overlapping Intervals] Let Tyy) denote the parity of a nonnegative integer y and let By = L(y 1)/2jk + ( - 1) for [k], where [0, 1] parameter of the distribution. Then we define π Y) to be the distribution on IR x [k] such that Ty is uniform over [k] and (X = y) is uniform over [8y, By 1].
Definition 4.4. [General Data Distribution] Given a parameter a E [0, 1], we define л be any distribution whose support is contained in Rd x [k] satisfying the following constraints:
1. (Classes are roughly balanced) Ty (Y O(1kk)
2. (Constant class overlaps) Letting M denote nonnegative integer constant, there exist (kk) classes for which there are classes 81 (v) 82(y),..., 8m (m))) for some 1 S m S
with Ty (supp(TT)) n supp(Tx: 2 - a, and all other s' E [k] satisfy x (x(ppp(()))
supp()):)) = 0.
3. (Overlap density is proportional to measure) 7 E A) O(ua(A)) and T..ig) (X A) ((μa(A)) for every A C supp(T,)) supp((T..
A.1 PROOFS OF PROPOSITION 4.2 AND THEOREM 4.5
Proposition 4.2. Let X' consist of N i.i.d. draws from the distribution specified in Definition 4.1, with a parameter a. Then with probability at least 1 (eexp(-S2(N))) over the randomness of X, the set S of all models g that are ERM interpolators for X' and -reguaa over each overlapping region in supp(ı, xx) is non-empty (in fact, uncountable). Furthermore, the predictive distribution fr (Y) X) σ (X)) of the optimally temperature- scaled model gr for any g E S satisfies:
Ex- (d X), aY X))] 2 @(11 a 1/k)) k)
(4.1)
Thus, for a = O(1), even with oracle temperature scaling every y E is asymptotically no better than random. In contrast, as the separation a 1, the bound in Equation (4. becomes vacuous.
Proof. We begin by first verifying that the set is large with high probability. Firstly, there are uncountably many strong ERM interpolators g, since the interpolator constraint is only the finitely many points in (so behavior is unconstrained elsewhere on the input space), and with probability these points do not coincide.
In order to show that single class y à with 7(y) 1 (i.e. the class y is odd-numin add-nuumerred.. In this case, the overlapping region is [8, + a,,3, + 1] (see Definition 4.1.. By a Chernoff bound, there are @((1 - a/N/k) class y points in this region with probability exp( 2 2( o))/k))-
is feasible over each overlapping region, let us focus on a
13

---

Published as a conference paper at ICLR 2024
Now conditioning each class y point in the region, the probability that class y + 1 point falls in a ) neighhboioo of x is at most 1/(2N) (since the classcconitional density of the class y + 1 points is uniform over [3, + a, Sy 1 + al, and the class prior is uniform over [k]). Thus, by a union bound the probability that x has no class y + 1 neighbors is at least 1/2. Now by another Chernoff bound, there are @ - N/k) class y points with no (4NN))-eighhoorhhne with probability at least 1 exp(-
neighbors a))/))) which allows us to estbalish k/(NN) )-regularity over [8, + 3, + +J. Repeating this logic for each overlapping region (conditioning appropriately) and taking a union bound shows that the set S is large with probability 1 - k exp( - QQ((1 o.))/))).
With k/ -Aearrreullrr established, our approach is straightforward: we will lower bound the expected KL divergence by considering just the divergence over the regions consisting of the umions of the regularity neighborhoods, over which temperature scaling can best bring the softmax outputs to uniform over i (i.e. each output is 1/k). However, the ground truth conditional distribution over the regularity neighborhoods is uniform over only two classes (since the neighborhoods are in a region of overlap between two classes).
As before, for simplicity we will focus on single class since the argument that follows can be iterated for each subsequent class. To keep notation brief, let us define:
H (+1 A) (x log (Y X)]
(A.1)
In other words, the cross-entropy restricted to a region A. Now consider a class 1 < y < l with r(y) (even parity); over its support [8y, By + 1] we observe that:
1
+
H (T(( = 1:
Σ ME(A]
+(Y = y X = x) log T(Y = 1/ X = 1
x) dx
Av
log 2
(A.2)
So the entropy over the support of class y of the ground truth conditional distribution is a constant divided by k. We will now show that the cross-entropy over the same region for the temperature-scaled predictive distribution f X) is lower bounded by @(11 a) log(k)/kk)
To do so, will constrain our attention as mentioned earlier to the entropy over just the regions spec- ified by (/(4N) (regulartr Let U denote the union of all of the ((4N))confiddnce neighborhoods around class y points in the region [8y. By 1]. Note that we focusing the neighborhoods around class y - 1 points (this corresponds to the num-beebe class in our discussion above), because in these neighborhoods the class y softmax output will be extremely small (due to the interpolation property applied to g on the class y - 1 points). We have that:
1 log d"(g(x)/T) dx
H ( (8,. (8,,8,y11
min Ju 2 T'0
(A.3)
Now we can lower bound the integrand in Equation (A.3) at each point (I y - E X with x, e U as: log 6" (g(x:)/T)
exp()"(x:)/T)
= -log
exp(g(u-1)xx,//
E) (xp(T=(x:)
(x:()"(x:/(T)
2 - log
(A.4)
exp(g(v-1))
+ (k- 10) exp(((e -C))))
Where C2 above is from the ERM interpolation property of g. Applying k/ -reguatryyylartty we can translate the bound in Equation (A.4) all U at the cost of introducing O(k/N) correction to g: (1. and (11/k) correction to T (U). The former error term will be irrelevant the calculation (asymptotically), while the latter error term will show up in the final bound. Letting X denote all points 2; E U with (xi, y - 1) E X' and recalling that ii (U) denotes the I-dimensional Lebesgue measure of the set U, we then obtain:
H (ftr 1 (9,00+2) 2
(U) O(1/k)
exp(g"(z;))/T)
θ
max log E
exp(glu-D(xxxTT) + (k: 1) expe( - C.)TT)
(A.5)
14

---

Published as a conference paper at ICLR 2024
Now we consider two cases; T < and T 2 1. In the former case, since exp N I (x:()) we have that the term inside the log is bounded above by 1 /k. In the latter case, since C2 a constant, the term inside the log is also bounded above by 1/k. Thus we get:
((u( - -((/k)) log
H ( ( 2 @
(A.6)
k
Finally, we observe that U consists of the union of O /k) neighborhoods of size (kk/N), so μi (U) ((1-a) aa. Iterating this argument over each of the [k/2] overlapping regions, we obtain the desired result.
As mentioned, the proof of the more general result follows the proof of Proposition 4.2 extremely closely, and we thus recommend reading the above proof before proceeding to this one (as we skip some details).
Theorem 4.5. Let X' consist of N i..dd draws from any distribution satisfying Definition 4.4, and let E IR be such that the sphere with radius r in has volume k2(2MN) Then the result of Proposition 4.2 still holds for the set Sa of ERM interpolators for X' which r-regular over each overlapping region in supp(Txx).
Proof. As in the proof of Proposition 4.2, there are uncountably many ERM interpolators for X. Now for showing that there subset of such interpolators satisfying uniform r-confidence over overlapping regions with high probability, let us consider class y whose support overlaps with the supports of other classes 81 (v): ..., (y) (as defined Definition 4.4).
Let A C ( denote the overlap between the support of y and an arbitrary one of the aforemen- tioned classes (3). As before, by a Chernoff bound, there are S2((1 - a N/k) class y points falling in A with probability at least 1 - exp( (11 a/))), since by assumption Ty (A) 2 a.
Conditioning on each class y point æ in A, the probability that a point from classes 81 (y),... 8m (v) falls in an r-neighborhood of æ is at most m/ (2MN). As before, this implies by union bound and another Chernoff bound that there are @( - N/k) class points in the overlapping region with r-neighhorhood neighbors with probability least 1 exp(-S2((
N/R).. Repeating this logic and taking another union bound, it follows that the set S is large with probability at least 1- k e 10--0((1a/N//))) a/k))
The rest of the proof carries over similarly from the proof of Proposition 4.2. Let be any class with overlaps, considered above. Similarly, let s be any class overlapping with y whose region of overlap we denote as A, once again as above. Defining H (-) as in Equation (A.1), we see that:
H (Tt (xllupp(r.)) O (+))
(A.7)
Now letting U denote the union of all r-neighborhoods around class points in the overlapping region A, we also get:
H (AT (upp(r)) 2 - -0 T20
min / log d" (T(x)/T) de
(A.8)
As in the proof of Proposition 4.2, we can lower bound the integrand in Equation (A.8) at each point (Ii,, 3) E X with x; E U as:
cxp() explg"(a,/TT) (k: (xexp)ep(rrre C,)/T)
- log 6z (g(x:)/T) - llog
(A.9)
Once again letting Xy denote all points 2; E U with (Zr,8) X, obtain from r-regularity:
Ha (f 11.) 2 0 @0 [allU) O(1/k)
(x:()"(x:)/T)
max log EEE
k
exp(o)(x./T)) + 1 e))pp((((((x))) C.)TT)
(A.10)
15

---

Published as a conference paper at ICLR 2024
Which, by identical logic to the proof of Proposition 4.2, gives:
[Ha(U) ((1/k) logk
Ha Ha 2 θ
(A.11)
k
Observing again that pa(U)) Q(1 - a) and applying this logic to all @ overlapping regions in supp(Tx), ) we obtain the desired result.
A.2 MIXUP OPTIMALITY LEMMA AND PROOFS OF PROPOSITION 4.3 AND THEOREM 4.6
First, we provide proper definition of the mixing set Ma(X), which was intuitively described in Section 3.2 of the main paper.
Definition A.1. Given a dataset X, we define:
Ma(x)= o E [A]]tt] Zo. Toats # Vi E [d]
and (Jo: Bag-1" Fay Fouts)
| Fort [|,5 Fasti S 2d
Vi,j [d]. i
and
|| Ioo S C(() Vi,j [d +1] 11
Where ((X) is a possibly data-dependent constant.
We recall that the definition of X is all such that Eje 1+1] AzJa for some supp(Dx,a) and E M,( Now we prove a formal version of Lemma 3.3, which will rely on the proofs of Proposition 4.3 and Theorem 4.6.
Lemma A.2. Let /x denote the density of Dx,a and define:
Lg
To Tos-1
Foz
Tost1
Tos
Jost1
(A.12)
Az(2) = L (2 Doam) 1 Σ Lo 1z Rourls
(A.13)
SEld]
Ey(z)
Σ
Lecoou (E ((+,, del (Lo 1s2 (No(=))
Σ Aa(s))
(A.14)
rEM)(xX)
96, my
Then any 8 E arginf, g J X, Dx.a) satisfies e² (0 = [x(2)/ /ell/EeEK Ex(2) for almost
every z E Xmix.d-
Proof. First, writing out the expectation in Equation (3.5), we have:
X log gi (1
(1I
E Xysto ))
1
J,, X2D), = [Ma(X\)
S(A) da
oxM
sE[d+]]
jEld+1]
(A.15)
Now we make the substitution = EjE(a+1) )yFo, By the definition of Ma(X') the transformation Lg Equation (A. 12) invertible, so (=) from Equation (A.13) is well-defined. Now applying Fubini's Theorem, may write:
((0 = det (L. (Ao(2))((A
Ar(z); (0 sEld+1)
Σ
(A.16)
X, [Ma(+))
LEccOuv((En, (tipt ((0,2) 2) dz
(A.17)
Mmix.e (EMa(X)
16

---

Published as a conference paper at ICLR 2024
We observe that LoeMa(x) 1 =Econv (1=,) 1 A(0, 2) is strictly convex as function of ((g(2)).
Now we can optimize the aforementioned term as a function of ((g(z)) under the constraint that Eyels: d (g(()) 1. This straightforward to do with Lagrange multipliers, and the solution is of the form (0(())) = (x(x)/ Me (x) where Ey(2) is defined as in Equation (A.14), so the result
follows.
With Lemma A.2, we can prove Proposition 4.3.
Proposition 4.3. Let A' be as in Proposition 4.2 and p(k) denote polynomial in is of degree least one. Then taking 2x,1 to be uniform, every 1-Mixup interpolator g for A' with the property that 6 (g(x)) ([1p(()) for every x supp(Tx) Xmiix, and y E [k] satisfies with probability
least 1 3 exp( (N((N)))))
Ex [dKL (T X), #Y X))] s (1))
(4.2)
Note that this result is independent of the separation parameter
Proof. We first partition supp(Tx) into subregions each of which have measure @(1/k²) (the reason for this choice will be made clear shortly), so that there are θ(k²) total subregions. By Chernoff bound, there are ((N/k²) points in each of these subregions with probability at least 1 - exp( (NN))))) Thus, by union bound, the probability that every subregion contains O(N/k²) points is at least 1-3 exp(-S2(N/k²)). Consequently, with the same probability, (1)/k²). To see this, observe that for to be in Xmix,a, we need only have one point the left of z and one point to the right of 2, both within a constant distance of one another (note that the other conditions in .Ma(X) are vacuously satisfied since we are in dimension 1). By our high probability arguments, there is at most some ((1/k²) Lebesgue measure region in each pair of overlapping class intervals with no points in X, from which it follows that x (supp(xx) Xmix.a) = (
x (supp(TT)) \\ X
Now for y [k], consider E supp(ry) Xmixxdd We will show that every 1-Mixup interpolator g for X' with Dx,1 being uniform satisfies o² (g(=)) e [0(1), 1] when = falls only in the support of and e² (g(=)) [ 1011--011 @(1) in the overlapping region, which will be sufficient for showing the desired result.
We first handle the non-overlapping region of supp(TT)) Here we need to consider the effect of the separation parameter or if a O(1/k) (i.e. the classes have very high overlap), the behavior in this region will be asymptotically irrelevant to the final KL-divergence calculation (since it will be a (log(k) (k)k) term). On the other hand. if a 12(1/k), then our partitioning of the input space was fine enough that we may claim from our high probability arguments that there are O(a.V/)) /k) class points in some subregion [ By + (a)) Similar logic also applies when considering the overlapping region, but this time the aforementioned logic is applied to 1 - a. Thus, what follows, we will assume we in the regime of [9(1/k), - ((1/k)]
Suppose without loss of generality that ryy) = and that y < k (i.e. y is an odd-numbered class), and consider = E 8,,,,,, ax) ( X (this is, mentioned, the non-overlapping part of supp(T,)) )). Clearly 2 can only be obtained from a E MM(X'') with Vo, = y or You = y. since the other class supports are spaced k away and E supp(T,)) (T)) \ supp(xT+1) (i.c. we have to mix with class y). This implies that for g" E arginf, Jmix, (g. X, 1. we have E,(2) = 0 for s + y, y ++1 (where Ey is as in Equation (A.) Now we consider two sub-cases: = E [By, 3, + a/2) and 2 E a/2, By +a).
In the first sub-case, if yo; y then X3 1/2 since d1 (1 X1 ) must necessarily fall closer to whichever class y point produced it, which implies that (2) 2 (2).
For the latter sub-case, this is longer true, since some points in B8, a a/2, By + a) may fall closer to class + 1. However, by our initial arguments there are @(NN/k) class points in [8y: By a/22,, which implies there are @(aN2 /R²) choices of E [ By + a)) with constant weight associated with class y. Since there are O(aN2 total choices of a E MM(X') that can produce 2 E [8, + a/2, B, + a) (since we need to mix with a point the left of 2), get that E2( ((E)+1(2)) in this case. Now from Definition 3.4, it immediately follows that every 1-Mixup interpolator g satisfies (g(z)) E [0(1), 1] for
Ma(X') which can produce
E [8y By + oz).
17

---

Published as a conference paper at ICLR 2024
Let us consider [33 + 2, + 11] X (the overlapping part of supp(ny)). We can apply similar logic to that used in handling the sub-case of 2 E [3, + a/2, By + aj considered above; namely, for every z E [8y + az By + 1], there are (((= (3y + +1+ a - =) choices of a E MM(XX) which can produce with constant weight associated with class y (consider our partitioning of the space), and so as before we get that {y(2) = (((5+1(2)) However, there MMa(X) which can produce with constant weight associated with class y + 1, so we also get Cy+1(2) = (())(z)). Thus, it follows that g satisfies (0((()) E [@(1), 1 (1)) for E [8, a, By + &
also the same order of choices of
Based on the above, we see that H (f = (11) (where we recall that H is defined as in Equation (A.1)). Furthermore, since (" (g(x:) S 1 2 //poyyyy) on supp( Xnix,1, we have H3 (+1 supp(=x)))ana) = ([log(k)/k)) from which the overall result follows.
Before proving Theorem 4.6, we need to formally state the necessary assumptions alluded to in the main text. Once again, these generalize the core facets of the simple data distribution from Definition 4.1.
Assumption A.3. In what follows, X' is understood to be N i.i.d, draws from æ and Xmixx is obtained from X as defined above and in Section 3.2. With this in mind, restrict the class of distributions π from Definition 4.4 such that:
1. (Non-overlupping classes are separated) For y, E [k]. if T (supp((z)) n supp(m.) = 0. then d(supp(TT) supp(()) )) w(1). Additionally, supp(Ty) @ for all y E [k].
2. (Mixed points cover support) With probability at least 1 k-t exp( (Q(NN())) x (x(upp(T) Xmix,d) = (1/k).
3. (Mixed points are balanced across overlapping classes) Let us define for y E [b]] and E Rd:
supp((xxdl 2 M)(X;2) = E Ma(x)) : YEE E
E
SE(d+1]
and
Σ X, 00 d: =
(A.18)
Then with probability at least 1 1 13 exp(-S(N//3)) for every y E [k] and = E Xmix n supp(T,,) we have that [M = R(f(2 (1122,)))) N (/odd1///+))))) //a+11) for some function f. or we have for every a E Ma(A') such that E Eje(tt1) AyM with A E supp(Dx.a):
Σ
Σ A:
(A.19)
FV
The third part of Assumption A.3 appears complicated, but just generalizes one of the key ideas in the proof of Proposition 4.2: either our mixed points always fall closer to one class (i.c. they are far away from the overlapping region), or there are lot of possible mixings that can produce the mixed point. Ideally we would have structured Assumption A.3 to be purely in terms of the shape of æ and not in terms of the high probability behavior of the mixed points Xmixx, but various edge cases in high dimensions led us to structuring the assumption this way.
With this assumption, it is straightforward to prove the generalization of Proposition 4.2.
Theorem 4.6. Let consist of N i.i.d. draws from any distribution æ satisfying Definition 4.4 and Assumption A.3, and let p(k) be as in Proposition 4.3. Then the result of Proposition 4.3 still holds when considering d--ixup interpolators for X' where the mixing distribution Dx.d is uniform over the d-dimensional probability simplex.
Proof. Assumption A.3 allows us to largely lift the proof of Proposition 4.2 to higher dimensions. Indeed, from the second part of Assumption A.3 we already begin with the fact that T (supp(*xx) Xmix,a) (1/k) with probability at least 3.3 exp(p- (app-2((N/k))
18

---

move Published as a conference paper at ICLR 2024
ResNeXt-50 Train Logit Gaps
18
Mean Logit Gap
15-
12-
CITAR10
CIFAR100
SVHN
Euclidean Distance
Figure 3: Mean logit gap change as a function of distance away from the original training point.
Now before, for [k], consider supp((ry) ( Mixix.d By the first part of Assumption A.3 and Definition A.1, z can only be obtained by mixing points from class y and/or its overlapping neighbors (v), -..; 8m (y).
& (u))( for every (y)) (i.e. it not in the overlapping region), then by the third part of Assumption A.3 it immediately follows that (g())) E [0(1), 1] (note that unlike in dimension 1 the expression for Ey in Lemma A.2 is more complicated now, but due assuming uniform mixing density and near orthogonality of mixed points Definition A.1 we can simplify to essentially expressions like in the 1-d case). On the other hand, if Eje(a-1] 1] Ajati E (upp(Ta.(u)) for some
sy)), then we cannot simultancously satisfy:
Σ
Σ
and Σ
Σ
X
(A.20)
Pilee FM
=.)
This implies that the other condition in the third part of Assumption A.3 satisfied for both y and s(()); i.e (M) = (2 Ad //,++1) (see Equation (A.18)). This again implies that (g(2)) [0(1, (1)). As in the final part of the proof of Proposition 4.3, from this we get (f ) = ((1) and we are done from the fact that sd (g(x)) < 1 - 2(1/poly(kk) on supp(Txx) Xmix,d-
B ADDITIONAL EXPERIMENT PLOTS
B.1 MODEL LOGIT BEHAVIOR
To verify that the locally Lipschitz-like model behavior of Definition 3.2 is reasonable in our setting, we consider the following experimental setup. For every training point in each dataset we consider, we uniformly sample 500 points (due compute constraints) from the surfaces of hyperspheres centered at the original training point with radii ranging from to 10. The cutoff radius of 10 was chosen as that was roughly the minimum distance to a neighboring training point of a different class across the considered datasets.
For all sampled points, we compute the trained model (following the training setup of Section 5) logit gap (to account for translation invariance) between the logit for the correct class (or the original point) and the largest logit belonging to a different class. We then compute the mean over all logit gaps each of the different radii considered, and plot how the mean logit gap changes farther away from the original training points. Results are shown in Figure 3.
As be seen in Figure 3, the mean logit gaps for each dataset barely move this radius, suggesting that assuming Lipschitz-like behavior of the logits with high probability should be reasonable in small neighborhoods. However, acknowledge that truly make this claim one
19

---

Published as a conference paper at ICLR 2024
would need to prove guarantees about the sample complexity needed to get appropriate coverage in each neighborhood. Our goal with the experiments this section has only been to provide an initial attempt at such justification; think that exploring how model logit/confidence behavior changes as one moves away from the training data contains many interesting challenges for future work.
B.2 OMITTED PLOTS FOR SYNTHETIC DATA
Figures 6 display the confidence histograms and reliability diagrams for ERM + of the Mixup models (Mixup, 2-Mixup, and 4-Mixup) for the different settings of Gaussian test data in Section 5.1. As was originally seen Table 2, the calibration performance for ERM markedly declines as we move from μ = 0.25 to μ = 0.05.
well as all
The reason for this is visible the confidence diagrams: at the μ 0.25 level, the ERM confidences still very bimodal, but the accuracy in the top bin is close to perfect. On the other hand, when moving to l = 0.05, the accuracy the top bin drops significantly while the accuracy the bottom bin increases, which corresponds to the drop in calibration performance since nearly all predictions fall in these two bins.
Confiderca Himrgrm
Confidence
Agp AICLIR
40g Actusit
Arp: Contidnncs
confddrco
Loeftiance
Reciability
corfect calitatten
erfherall
suppan
Coelisnce
(a) ERM TS
(b) Mixup
Confiterce co Histogram
Confidonce Hotogram
Aup Accutry
Accurscy
nikize af
Falitivs Aroonnt Campla
14
3.1
0:2
2.4
0.6
c6
14
00
aibiiiiiyy egram
Belibility Diagrom
Pertett Collectnn
Notai Galnuran
0.
24
2.6
Carfdbrroe
Cortdencce
(c) 2-Mixup
(d) 4-Mixup
Figure 4: Confidence histograms and reliability diagrams for ERM + and Mixup models on the test Gaussian data with μ = 0.25 1, using 15 bins.
B.3 OMITTED PLOTS AND TABLES FOR IMAGE CLASSIFICATION BENCHMARKS
Tables 4 and 5 correspond to the ECE and ACE versions of 3; we find as mentioned in the main text that Mixup dominates in ECE and ACE well. Unlike in the case of Gaussian data, here both the ECE and ACE monotonically increase with label noise for the ERM + TS models (the only exception being SVHN the 25% label noise level, for which the mean ACE decreases but remains
that
20

---

Published as a conference paper at ICLR 2024
Hastggam
Canfilence Halogram
20
10
Avb costabncc
Carficecc
Reeeive arouur samples
00
12
00
Belibility Dragnem
Rollabiity Diagram
1.
1.0
Orggu
Olo
Gep
3.
622
10
ERM
(b) Mixup
Acausty
Arp. Cosbarict
Cariacnoo
Cample
Semeto
Caridder
Dirgram
Diagramn
Perfect
Feefect Caltrania
Oviut
Ourpct
Gap
00-
11
02
(c) 2-Mixup
(d) 4-Mixup
Figure 5: Confidence histograms and reliability diagrams for ERM + TS and Mixup models on the test Gaussian data with μ 0.05 using 15 bins.
within standard deviation the ACE for the 0% noise level). For Mixup, find that there are few additional instances in which mean ECE and ACE seemingly improve with increasing label noise (CIFAR-10) 100 and SVHN), but again these instances the performances are roughly within the one standard deviation eπror bounds of the lower noise levels. Furthermore, also remark that is is possible for NLL to get worse (as seen in Table 3) while ECE/ACE improve. As an example, one can consider uniformly random classifier when classes are roughly balanced, this classifier has near-zero ECE despite having poor NLL performance.
Figures 7 to 9 contain the confidence histograms and reliability diagrams for the ERM + TS and Mixup models across CIFAR-10, CIFAR- 100, and SVHN. Unlike in the case Gaussian test data, find non-trivial gaps in test accuracy between the ERM and Mixup models CIFAR-10 and CIFAR-100, which certainly contributes to some of the gap in reported negative log- liglliiel However, can
Figure that ERM actually outperforms Mixup in terms of test accuracy in this case (across different label noise settings), and still suffers massive gap in NLL. Additionally, the confidence histograms are telling, as we once again find that the ERM + TS predictions cluster heavily in the largest probability bin, which is again the cause for poor calibration performance.
B.4 ANALYSIS THE IMPACT TEMPERATURE SCALING
Tables to 8 show the relative performance gains in terms of NLL, ECE, and ACE respectively due to temperature scaling for both ERM and Mixup CIFAR-10, CIFAR- 100, and SVHN. In the interest of space, we report the NLL results rounded to the nearest hundredth, and the ECE/ACE results rounded to the nearest tenth after multiplying by factor of 100.
21

---

Published as a conference paper at ICLR 2024
Hastagamm
Canfilence Halogram
20
10
Avb contibncc
Carficecc
Reletie aroour samples
00
12
Relibbiity Dragrem
Rollabiity Diagram
1.
1.0
Orggut
Oup
Gep
30
02
24
10
ERM
(b) Mixup
Acausty
Arp. Coobaritt
Cariacno
Ankert Cample
Sereto
Carfider
Diegram
Diagram
Perfect
Ferfect Caltranan
Ouiut
Ourpct
Gup
0.0
32
02
(c) 2-Mixup
(d) 4-Mixup
Figure 6: Confidence histograms and reliability diagrams for ERM + TS and Mixup models on the test Gaussian data with μ 0.01 using 15 bins.
Dataset
Label Noise
ERM TS (ECE)
Mixup (ECE)
0%
0.2259 +0.0057
0.0709 +0.0103
CIFAR-10
25%
0.3540 +000194
0.1055 +0.0200
50%
0.5540 +-0.1111
0.1760
+0.0569
0%
0.4370 #0.0413
0.0811
=0.0175
CIFAR-100
25%
0.4890 #0.0164
0.0634
=0.0159
50%
0.5846 +0.0162
0.0649
+0.0251
0%
0.0696 #0.0051
0.0918
#0.0083
SVHN
25%
0.2070 +00.2227
0.0892 +0.0330
50%
0.4799 +-0.0155
0.1314 +0.0255
Table 4: Mean ECE over 5 runs with standard deviation error bounds on image classification benchmark test data with varying levels of label noise.
The main observations we make are that temperature scaling can make significant difference for the ERM models, while makes very little difference for the Mixup models. Furthermore, as one would expect, the greatest benefit is seen with respect to NLL, which is exactly the objective which the temperature scaling parameter not make notable difference in our setup, in most cases the post-TS performance is not outside of the one standard deviation error bounds of the pre-TS TS performance, although for ERM the mean ECE/ACE is almost always improved by temperature scaling.
fit. For ECE and ACE, we find that temperature scaling does
22

---

Published as a conference paper at ICLR 2024
Dataset
Label Noise
ERM+TS (ACE)
Mixup (ACH)
0%
0,2562 #0.0188
0.1012 #0.0132
CIFAR-10
25%
0.2909 #0.0108
0.1216 #0.0327
50%
0.3691 1.0.0114
0.2060 +0.0471
0%
0.3702 =0.0250
0.1004#0.0119 #0.0197
CIFAR-100
25%
0.3992 +-00193
0.0986 +-00245
50%
0.4608 +000218
0.1274 +0.0550
0%
0.2436 #0.0409
0.0643
#0.0091
SVHN
25%
0.2049 #0.0189
0.0937 #0.0139
50%
0,2851 10.0108
0.1880 +0.0277
Table 5: Mean ACE over runs with standard deviation error bounds on image classification benchmark test data with varying levels of label noise.
Dataset
Label Noise
ERM
ERM + TS
Mixup
Mixup + TS
0%
2.21 +0.20
2.01 =0.18
0.82 #0.04
0.82 =0.04
CIFAR-10
25%
3.53 +0.23
3.21 =0.20
1.31 #0.03
1.30 #0.03
50%
6.57 #0.81
5.95 #0.72
1.77 #0.08
1.76 #0.07
0%
6.02 #0.92
2.46 #0.05
2.45 #0.05
CIFAR-100
25%
6.97 #0.45
6.38 #0.41
2.98 =0.10
2.98 #0.10
50%
8.78 +0.67
8.02 +0.60
3.54 +0.04
3.54 #0.04
0%
0.76 10.13
0.69 10.12
0.34 10.04
0.33 #0.04
SVHN
25%
1.84 #0.27
1.68 #0.24
0.86 =0.04
0.86 #0.04
50%
4.61 #0.41
4.18 +0.36
1.50 #0.04
1.50 #0.04
Table 6: Mean NLL (rounded to the nearest hundredth) over runs with standard deviation error bounds on image classification benchmarks for TS and non-TS ERM and Mixup.
This is perhaps not super surprising in light of the reliability diagram and confidence histogram visualizations of Figures difference in the ECE/ACE performance due the gap between the top-class logit/probability and the other classes, but this is of course not optimal when optimizing T using the NLL objective. For Mixup, the pre-TS probabilities roughly balanced in the datasets we consider), and they change negligibly after temperature scaling. We also note that while there are instances in which the post-TS Mixup ECE/ACE performance is worse than the pre-TS performance, in these cases the change in ECE/ACE is virtually nothing, and falls well within the one standard deviation error bounds; again, this because the post-T'S probabilities have not changed by much. This stability of Mixup performance under temperature scaling a possibly interesting direction for future exploration.
9. For ERM, we would need a large choice of T make a substantial
already much less spiky (leading good ECE/ACE because classes
We also point out that while in our setting the post-TS ECE/ACE results are not substantially different, been observed in prior work (Guo et al., 2017) that temperature scaling can improve ECE significantly. We reconcile this with our results by noting that our experimental setup treats the case of ERM without additional regularization and trained to the interpolation regime of zero training error (since this is the focus of our theory). In prior work, the models considered were trained with several additional regularizations (data augmentation, weight decay, etc.), which will certainly have a notable impact calibration performance.
Lastly, the results of Tables 6 to also further corroborate our main theoretical ideas: even though temperature scaling can lead improvements for the ERM models, the gap between ERM calibration performance and Mixup performance simply cannot be closed by temperature scaling alone.
23

---

Published as a conference paper at ICLR 2024
Dataset
Label Noise
ERM
ERM+13
Mixup
Mixup TS
0%
23.2 114
#0.6
#1.
6.7 +0.5
CIFAR-10
25%
42.0
35.9
10.6 +20
8.9 #1.6
50%
56.4 +1.0
55.4
17.6 +5.7
16.4 =5.1
45.5 44.1
43.7 +4.1
811
6.8 +1.3
CIFAR-100
25%
$1.2+1.6
48.941.
6.3 +1.
+1.5
50%
61.0+1.6
58.5
6.5 +2.5
6.2 +2.0
0%
7.1 #0.5
=0.5
9.2 +0.8
7.8 +0.6
SVHN
25%
21.4 +2.2
20.7 +2.3
8,9 +3.3
9.5 #3.6
50%
48.8 11.7
48.0 +1.
13.1 12.6
12.0 2.3
Table 7: Mean ECE (multiplied by 100 and rounded to the nearest tenth) over 5 runs with I standard deviation error bounds on image classification benchmarks for and non-TS ERM and Mixup.
Dataset
Label Noise
ERM
ERM + TS
Mixup
Mixup TS
0%
27.1
25.6 #1.9
10.1 #1.3
10.2 =1.1
CIFAR-10
25 %
29.7 +.0.9
29.1 +1.1
12.2 +3.3
9.3 +18
50%
37.4 +00.8
36.9
20.6 +4.77
21.0 +39
0%
38.3 +233
32.0
10.0 +2.0
8..
CIFAR-100
25%
41.2 112
39,9-1.1
9.9 +2.5
10.0 =2.2
50%
47.1 11.4
46.1 #2.
+5.5
11.8 =3.2
0%
24.3 #4.1
24.4
6.4 6..9
7.4 #2.4
SVHN
25%
21.7 +1.8
21.9
9.4 114
9.9 +12
50%
28.8 +1.0
28.5
18.8 +2.8
19.0 3.5
Table 8: Mean ACE (multiplied by 100 and rounded to the nearest tenth) over 5 runs with I standard deviation error bounds on image classification benchmarks for TS and non-TS ERM and Mixup.
24

---

Published as a conference paper at ICLR 2024
Cannidrr
Confatence
deg Arrenee
28
0.-
0.4
14
24
0.6
C8
1.
00
Cerabroe
Genjoxt Rutubility Diagram
Rat
Perrett Criteetenn
Notact Can montoo
18.
00-
2.6
Ceriberoe
(a) 0% Label Noise, ERM TS
(b) 0% Label Noise, Mixup
Confider ta Masogrm
Confiderice Halogram
Pelatine Seeeaiis
5.0
Relabilty Diromm
Relisblity Diagram
Calctrtoon
Gop
0.6
2. 3.0
0.6
00
Contaberco
Cueisnce
(c) 25% Label Noise, ERM + TS
(d) 25% Lahel Noise, Mixup
Confidiecee Hitogram
Confidence Hotogram
1.4
1.0
Confionce
Corficanoe
reladii Ameurt staappie
Fxlectiv Ansount Samplas
04-
3.8
C2
1.4
5.6
C#
14
0.0
Hutatellyy D oggam
Iuttubiiiy Usagrom
Cap
1.
C3
1.4
C6
C6
00
16
(c) 50% Label Noise, ERM
(f) 50% Label Noise, Mixup
Figure 7: Confidence histograms and reliability diagrams for ERM TS and Mixup models CIFAR- 10 at different levels of label noise.
25

---

Published as a conference paper at ICLR 2024
Carnidecc Mitogram
Confatence Hologram
au ECoutry
(a] drnurry
28-
00
2.4
0.4
14
144
0.6
10
00
Genjoxt
Valiabiitty
Butubiltt Diagram
Perrett Criteetton
Notast Cantrrson
18
00-
2.6
CerMbrro
(a) 0% Label Noise, ERM TS
(b) 0% Label Noise, Mixup
Conniderto DE Malogrmm
Confiderie Halogram
Inlatiin drower Kaingiae
Centeberoe
Relabilty Drromm
Relibblity Diagram
Callcatoon Gop
2.
3.0
022
2.4
0.6
00
Confaberco
Cueisance
(c) 25% Label Noise, ERM + TS
(d) 25% Lahel Noise, Mixup
Confiderre ce Histgramm
Confidence Hotogram
1.4
1.0
vinderss
Carficenoe
Fabitive inroutt of
Falecivs Amsount Samplas
04-
3.8
0.2
144
5.6
14
000
Hutaateity D. oggam
Iutubbiity Usagrom
Cap
1.
23
1.4
0.5
0.6
3.0
00
16
(c) 50% Label Noise, ERM
(f) 50% Label Noise, Mixup
Figure 8: Confidence histograms and reliability diagrams for ERM TS and Mixup models CIFAR- 100 at different levels of label noise.
26

---

Published as a conference paper at ICLR 2024
Cannidrr Hissgram
Contatence
deg Arrenee
28
0.-
0.4
14
24
666
10
00
Cerabrre
Genjoxt Rutubility Diagram
Rat
Perrett Cribeetenn
Notact Canl Contrrion
28.
00-
2.6
CerMbrro
(a) 0% Label Noise, ERM TS
(b) 0% Label Noise, Mixup
Confiderr cal Hasogram
Confideriee Halogram
Pelatine Speeaiin
2.0
8.0
10
Centebrro
Relabilty Drromm
Relisblity Diagram
caltration
Calltrtton
Gop
0.6
2.1
3.0
066
1.
00
Contaberc
Cueisnce
(c) 25% Label Noise, ERM + TS
(d) 25% Lahel Noise, Mixup
Confidirre Historamm
Confidence Hotogram
1.4
Confiinnce
Confidence
Ielathi smccune Sen
telashre of Sample
3.8
02
1.4
0.6
14
0.0
Hutaatelty Di oggam
Iuttubiiiy Usagrom
Cop
1.
23
144
C6
C66
14
00
16
(c) 50% Label Noise, ERM
(f) 50% Label Noise, Mixup
Figure 9: Confidence histograms and reliability diagrams for ERM TS and Mixup models SVHN at different levels label noise.
27


Reviews:
[{"review_id": "pAENTEEiwR", "reviewer": "Reviewer_UALj", "summary": "This paper studies the limitations of the widely used temperature scaling for post-hoc uncertainty calibration. The authors find that under the specific assumption, i.e., the datasets follow a general set of distributions in which the supports of classes have overlaps, the temperature scaling method cannot perform well. Since the temperature scaling has been very successful in post-hoc calibration, this paper is interesting for pointing out its limitations and find under some conditions it provably fails to achieve good calibration. Furthermore, the authors find that the performance of temperature scaling degrades with the amount of overlap between classes, and asymptotically becomes no better than random when there are a large number of classes. This paper also studies a specific training-time calibration technique mixup and finds that it can lead to reasonably good calibration performance under the same conditions.", "questions": "Please refer to Weakness section.", "limitations": "", "rating": 8, "confidence": 2, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. This paper is well motivated. The studied point is interesting. It is empirically found that temperature scaling is good for calibrating deep models. However, as pointed out in this paper, it may harm calibration under some conditions.\n\n2. The empirical study shows very supportive results for the theoretic results. Both experiments on synthetic data and real-world data show positive results that temperature scaling cannot work under some conditions.\n\n3. The writing and organization of this paper is very good.", "weaknesses": "1. It is commonly believed that temperature scaling is very effective for post-hoc calibration scaling, although it is found that this technique cannot be used for all the cases. Can you explain what causes this gap between your theoretic results and the commonly observed empirical success?\n\n2. The main experimental results in tables only show the comparison between ERM+TS vs Mixup. I think that the results of ERM baseline and Mixup+TS should be presented at least. Moreover, is the results influenced by the training schemes used for training models (such as learning epochs, learning rate and regularization)?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iqiBA6z903", "reviewer": "Reviewer_dtwE", "summary": "The paper demonstrated how temperature scaling has subpar performance in the case of overlapping classes, and proposed mixed up as an effective alternative to improve model calibration. The paper considers the ERM interpolator set of models, where there's clear separation between the prediction of top-class and the rest. In this case temperature scaling is failing to produce the desired behavior of equal prediction on the overlapping portion of the two classes. On the other hand, training with mixing loss is able to capture the overlapping behavior and significantly improve ECE.", "questions": "- Does it make sense to generate additional classes for the overlapping case (y=1, y=2, y=1&2)? In that case would temperature scaling still work and what is the tradeoff here?\n- Is ERM interpolator the best model class to capture the datasets with overlaps? The properties of the interpolator seem to be naturally mismatched.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well written with clear presentation of references on background, assumption, and key results.\n- There are theoretical results backing up the observations made in the toy examples and experiments.\n- The few experiments shows good evidence supporting the conclusion that mixup is effective under the overlapping classes scenario.", "weaknesses": "- The mixing training introduces an extra degree of freedom (i.e. d-mixing). Based on table 2, we see that mixing is actually negative for NLL when classes are relatively separate, but only show performance improvement as the overlap increase. I also believe that the model performance would not be strictly better as we increase the degree of mixing, not to mention the additional computational complexity. Intuitively, the optimal d should have to do with the structure of overlapping in the dataset. I think it would be beneficial for the authors to have a more in-depth discussion on the choice of mixing in practice. Discounting the additional regularization effect, is it reasonable to only have the regular mixup when only two classes overlap at a time?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2OCHeM4dUj", "reviewer": "Reviewer_JXm6", "summary": "This paper studies when and why temperature scaling may not work. This paper showed theoretically that when the supports of classes have overlaps, , the performance of temperature scaling degrades with the amount of overlap between classes, and asymptotically becomes no better than random when there are a large number of classes. This paper suggests that Mixup data augmentation technique can lead to reasonably good calibration performance, which are supported by the experiments conducted in the paper.", "questions": "1. \"We also trained d-Mixup models on the same data, but we found that the confidence regularization effect for d > 2 led to underconfidence on these datasets, so we report just the results for Mixup.\": do we know why this may happen?\n2. In the image experiments, for CIFAR-100, why having label noise makes NLL worse but ACE / ECE better? This makes the experiments less convincing if we don't have a solid explanations.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "This paper identifies theoretical limitations of a widely used calibration technique, temperature scaling. The paper is technically solid and clearly written, with aligned theory and experiments.\n\nOriginality: The paper offers novel theoretical results on the inherent limitations of temperature scaling based on distributional assumptions. While temperature scaling is widely used, formal characterization of when it provably fails is new. The conditions identified are also intuitive and realistic.\n\nQuality: The theoretical results are informative. [Note: I am not an expert in theory though, and I didn't check the proof]. The experiments cleanly validate the theory on both synthetic data and real image benchmarks. The proposed d-Mixup method is interesting. Overall the paper reflects quality research.\n\nClarity: The problem is motivated well and background provided. The writing clearly explains the theories, assumptions, experiments, and connections between them. Figures aid understanding. The paper is well organized.\n\nSignificance: Calibration is critical for uncertainty aware models, but little theory exists. This paper significantly advances understanding of an important technique. The insights on training procedures are impactful for future work.", "weaknesses": "1. The scope is limited to temperature scaling and Mixup. Discussing connections to other calibration methods could broaden impact.\n2. It would be better to have more real data experiments. In the \"IMAGE CLASSIFICATION BENCHMARKS\", the overlap is introduced rather artificially.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)