Tailoring Mixup to Data for Calibration
Authors: Quentin Bouniot
Pavlo Mozharovskyi
Florence d'Alché-Buc
Venue: ICLR 2025 Poster (2025)
Keywords: mixup
calibration
confidence
robustness
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: We show that taking distance into account in mixup reduces occurence of mismatch between mixed labels and mixed samples, improving confidence, calibration and robustness.

Abstract: Among all data augmentation techniques proposed so far, linear interpolation of training samples, also called Mixup, has found to be effective for a large panel of applications. 
  Along with improved predictive performance, Mixup is also a good technique for improving calibration.
  However, mixing data carelessly can lead to manifold mismatch, i.e., synthetic data lying outside original  class manifolds, which can deteriorate calibration.
  In this work, we show that the likelihood of assigning a wrong label with mixup increases with the distance between data to mix. 
  To this end, we propose to dynamically change the underlying distributions of interpolation coefficients 
  depending on the similarity between samples to mix, and define a flexible framework to do so without losing in diversity. We provide extensive experiments for classification and regression tasks, showing that our proposed method improves predictive performance 
  and calibration of models, while being much more efficient.

Published as a conference paper at ICLR 2025
TAILORING MIXUP TO DATA FOR CALIBRATION
Quentin Bouniot, 1,2,3,4 Pavlo Mozharovskyi & Florence d' d'Alché-Bue' 'LTCI, Télécom Paris, Institut Polytechnique de Paris, France,
TTechnical University of Munich, 'Helmholtz Munich, *Munich Center for Machine Learning (MCML)
ABSTRACT
Among all data augmentation techniques proposed so far, linear interpolation of training samples, also called Mixup, has found to be effective for large panel of applications. Along with improved predictive performance, Mixup is also a good technique for improving calibration. However, mixing data carelessly can lead to manifold mismatch, i.e., synthetic data lying outside original class manifolds, which can deteriorate calibration. In this work, we show that the likelihood of assigning a wrong label with mixup increases with the distance between data to mix. To this end, we propose to dynamically change the underlying distributions of interpolation coefficients depending on the similarity between samples to mix, and define a flexible framework to do so without losing diversity. We provide extensive experiments for classification and regression tasks, showing that our proposed method improves predictive performance and calibration of models, while being much more efficient.
INTRODUCTION
The Vicinal Risk Minimization (VRM) principle (Chapelle et al., 2000) improves on the well-known Empirical Risk Minimization (ERM) (Vapnik, 1998) for training deep neural networks by drawing virtual samples from a vicinity around true training data. This data augmentation principle is known to improve the generalization ability of deep neural networks when the number of observed data is small compared the task complexity. In practice, the method of choice to implement it relies on hand- crafted procedures to mimic natural perturbations (Yaeger et al., 1996; Ha & Bunke, 1997; Simard et al., 2002). However, one counterintuitive but effective and less application-specific approach for generating synthetic data is through interpolation, or mixing, of two or more training data.
The process of interpolating between data have been discussed multiple times before (Chawla et al., 2002; Wang et al., 2017; Inoue, 2018; Tokozume et al., 2018), but Mixup (Zhang et al., 2018) represents the most popular implementation and continues to studied in recent works (Pinto et al., 2022; Liu et al., 2022b; Wang et al., 2023). Ever since its introduction, it has been a widely studied data augmentation technique spanning applications to image classification and generation (Zhang et al., 2018), semantic segmentation (Franchi et al., 2021; Islam al., 2023), natural language processing (Verma et al., 2019), speech processing (Meng et al., 2021), time series and tabular regression (Yao et al., 2022a) or geometric deep learning (Kan et al., 2023), to that extent of being now an integral component of competitive state-of-the-art training settings (Wightman et al., 2021). The idea behind Mixup can be seen as efficient approximation of VRM, by using linear interpolation of data points from within the same batch to reducing computation (Zhang et al., 2018).
The process of Mixup as a data augmentation during training can be roughly separated in three phases: (i) selecting tuples (most often pairs) of points to mix together, (ii) sampling coefficients that will govern the interpolation to generate synthetic points, (iii) applying a specific interpolation procedure between the points weighted by the coefficients sampled. Methods in the literature have mainly focused on the first and third phases, i.e., the process of sampling points to mix through predefined criteria (Hwang et al., 2022; Yao et al., 2022a;b; Palakkadavath et al., 2022; Teney et al., 2023) and on the interpolation itself, by applying sophisticated and application-speet 2019; Franchi et al., 2021; Venkataramanan et al., 2022; Kan et al., 2023). On the other hand, these interpolation coefficients, when they exist, are always sampled from the same distribution throughout training. Recent works have shown that mixing carelessly different points can result in manifold
functions (Yun et al.,

---

Published as a conference paper at ICLR 2025
intrusions, i.e. mixed examples colliding with other real data manifolds (Guo et al., 2019; Baena et al., 2022; Chidambaram et al., 2021), or label noise, i.. mixed labels differing from ground-truth assignments, which can hurt generalization (Yao et al., 2022a; Liu et al., 2023), while mixing similar points helps in diversity (Chawla et al., 2002; Dablain et al., 2022). Furthermore, several previous work have highlighted trade-off between good predictive performance and calibration in Mixup (Thulasidasan et al., 2019; Pinto et al., 2022; Wang et al., 2023).
In this work, we show that the distance between the data used in the mixing impacts the likelihood of assigning a noisy label, which occurs from manifold mismatch, i.e. mixed samples lying outside the class manifolds of the original data used in the mixing operation. Manifold mismatch can then lead to manifold intrusion, if the mixed sample lies in a different class manifold, or label noise if the mixed sample is closer to a different class manifold. Then, we propose an efficient and flexible framework to take the similarity between the points into account by influencing the interpolation coefficients. high similarity should result in a strong interpolation, while a low similarity should lead to small changes, Consequently, controlling the interpolation through the distance of the points to mix improves calibration by reducing manifold mismatch. Our contributions in this paper are:
We show that the likelihood of assigning a noisy label is increasing with the distance between points, and taking into account distance when mixing can improve calibration. Mixing only similar data leads to better calibration than including dissimilar ones.
We present a flexible framework to dynamically change the distributions of interpolation coefficients. We apply a similarity kernel that takes into account the distance between points to select a parameter for the distribution tailored to each pair to mix. The underlying distribution of interpolation is warped to be stronger for similar points and weaker otherwise.
We quantitatively ascertain the effectiveness of our Similarity Kernel Mixup with extensive experiments on multiple datasets, from image classification to regression tasks, and multiple deep neural network architectures. Our approach achieves better calibration while improving accuracy. Additionally, we highlight the efficiency of our method, obtaining competitive results with less computation per iteration, and reaching the best performance globally faster.
RELATED WORK
2.1 DATA AUGMENTATION BASED ON MIXING DATA
Non-linear interpolation Non-linear combinations are mainly studied for dealing with image data. Instead of a naive linear interpolation between two images, the augmentation process is done using more complex non-linear functions, such as cropping, patching and pasting images together (Takahashi et al., 2019; Summers & Dinneen, 2019; Yun et al., 2019; Kim et al., 2020; Hendrycks et 2020) or through subnetworks (Ramé et al., 2021; Liu et al., 2022b; Venkataramanan et al., 2022). Not only are these non-linear operations focused on images, but they generally introduce significant computational overhead compared to the simpler linear one (Zhu al., 2020; Li et al., 2022). The recent R- R-Mixup (Kan et al., 2023), on the other hand, considers other Riemannian geodesics rather than the Euclidean straight line for graphs, but is also computationally expensive. Furthermore, Oh & Yun (2023) has recently shown that learning with linear mixup leads to more meaningful decision boundaries.
Linear interpolation Mixing samples online through linear interpolation represents the most efficient and general technique compared to the ones presented above (Zhang et al., 2018; Inoue, 2018; Tokozume al., 2018). Among these different approaches, combining data from the same batch also avoids additional samplings. Several follow-up works extend Mixup from different perspectives. Manifold Mixup (Verma et al., 2019) interpolates data in the feature space, Remix (Chou al., 2020) separates the interpolation in the label space and the input space. Notably, AdaMixUp (Guo et al., 2019) learns to predict a mixing policy to apply to avoid manifold intrusion using two additional parallel models with intrusion losses. This leads to a more complex training and includes more parameters to learn. Furthermore, the predicted range is very narrow around 0.5, reducing diversity of the mixed samples, and the training includes non-mixed samples, making the approach similar to the recent Regmixup (Pinto et al., 2022), described below. Local Mixup (Baena ct al., 2022) is
Code is available https:////// thtbeeb/////hbbue//e/e/e com wwt.co//ooooccccccc qbouni /t //smmmmm.... im_kernel L_mneelmmiiee
xup _mxxup
mi
2

---

Published as a conference paper at ICLR 2025
another approach interested the manifold intrusion problem, by weighting the loss function with the distance between the points using K-nearest neighbors graphs. However, this results in completely discarding the interpolation with points that are far away, losing potential augmentation directions and, therefore, in diversity. Moreover, these two methods are focusing on improving generalization, while we are interested in the more general problem of manifold mismatch to improve calibration. Selecting points A recent family of methods applies an online linear combination on specifically selected pairs of examples (Yao al., 2022a;b; Hwang et al., 2022; Palakkadavath et al., 2022; Teney et al., 2023), across classes (Yao et al., 2022b) or across domains (Yao et al., 2022b; Palakkadavath et al., 2022; Tian et al., 2023). These methods achieve impressive results on distribution shift and out-of-distribution generalization (Yao et al., 2022b), but recent theoretical developments have shown that much of the improvements are linked to a resampling effect from the restrictions the selection process, and are unrelated to the mixing operation (Teney et al., 2023). These selective criteria also induce high computational overhead. One related approach is C-Mixup (Yao et al., 2022a), that fits Gaussian kernel on the labels distance between points in regression tasks. Then points to mix together are sampled from the full training set according to the learned Gaussian density. However, all the data before training, which is difficult when there is lot of data and no explicit distance between them. k-Mixup (Greenewald et al., 2023) selects the pairs to mix based on Optimal Transport distance between two different batch of data.
the Gaussian kernel is computed
2.2 CALIBRATION IN CLASSIFICATION AND REGRESSION
Calibration is a metric to quantify uncertainty, measuring the difference between a model's '' confidence in its predictions and the actual probability of those predictions being correct. We refer the reader to Appendix C for a presentation of common calibration metrics used.
In classification Modern deep neural network for image classification overconfident leading to miscalibration (Guo et al., 2017). One can rely on temperature scaling (Guo et al., 2017) to improve calibration post-hoc, or using different techniques during learning such as ensembling (Lakshminarayanan et al., 2017; Wen et al., 2021; Laurent et al., 2022), explicit penalties (Pereyra et al., 2017; Kumar et al., 2018; Moon et al., 2020; Cheng & Vasconcelos, 2022), implicit ones (Müller et al., 2019; Lin et al., 2017; Mukhoti et al., 2020; Liu et al., 2022a), including notably through mixup (Thulasidasan et al., 2019; Pinto al., 2022; Noh et al., 2023). One should note that the ordering of calibration results can change after temperature scaling (Ashukba et al., 2020).
now known to be
Calibration-driven Mixup methods The problem of the trade-off between predictive performance and calibration with Mixup has been extensively studied in previous works (Thulasidasan et al., 2019; Zhang et al., 2022; Pinto et al., 2022; Wang et al., 2023). Notably, Wang et al. (2023) observed that calibration using Mixup can be degraded after temperature scaling. Therefore, they proposed another improvement of mixup, MIT (Wang et al., 2023), by generating two sets of mixed samples and then deriving their correct label. AugMix (Hendrycks et al., 2020), NFM (Lim et al., 2022) and NoisyMix (Erichson et al., 2024) add data-augmentation and noise before the mixing operation to improve robustness. RegMixup (Pinto et al., 2022) considers Mixup as regularization term, using stronger interpolation on every pair. Finally, RankMixup (Noh et al., 2023) uses the interpolation coefficients from multiple mixed pairs as additional supervisory signal for ranking of confidence. We propose a more efficient method to achieve a good trade-off between accuracy and calibration with Mixup. In regression The problem of calibration in deep learning has also been studied for regression tasks (Kuleshov et al., 2018; Song et al., 2019; Laves et al., 2020; Levi et al., 2022), where it is more complex as we lack a simple measure of predictive confidence. In this case, regression models are usually evaluated under the variational inference framework with Monte Carlo (MC) Dropout (Gal & Ghahramani, 2016) 6) to quantify confidence.
In our work, we use a similarity kernel to mix more strongly similar data and avoid mixing less similar ones, leading to preserving label quality and confidence of the network. As opposed to all other methods discussed above, we also show the flexibility of our approach by its effectiveness on both classification and regression tasks. We detail our framework, the similarity used and the intuition behind below.
3 SIMILARITY KERNEL MIXUP
First, we define the notations and elaborate on the learning conditions that will be considered throughout the paper. Let D = {(x;, w)) (X, XN x VN RR4xNN x R be the training
3

---

Published as a conference paper at ICLR 2025
dataset. We want to learn model f parameterized by A θ C RP, that predicts 9 := f(x) for any x E X. For classification tasks, we have Y (.....,, M), where M is the number of classes, and we further assume that the model f can be separated into an encoder part h : X + Id and classification weights w E xd2xM, with d2 the embedding dimension, such that Vx E X, fxx = w h(x)) and (oftmax([())))
V = arg max
To learn our model, we optimize the weights of the model 0 in a stochastic manner, by repeating the minimization process of the empirical risk computed on batch of data Bg {(x,, y:) sampled from the training set, for & {I...., T iterations.
With Mixup (Zhang et al., 2018), at each iteration I, the empirical risk is computed on an augmented batch of data B, = {x,, ū)) 1== such that x == A₂x +1 -x)))xx(())) and ūن == Aay/ + (1-Xx))aa (x)) with X ~ Beta(a. a) and or E 6, a random permutation of n elements sampled uniformly. Thus, each input is mixed with another input randomly selected from the same batch, and X represents the strength of the interpolation between them. Besides simplicity, mixing elements within the batch significantly reduces both memory and computation costs.
In the following parts, we introduce a more general extension of this framework using warping functions, that spans different variants of Mixup, while preserving its efficiency. First, we show and illustrate how Mixup can lead to manifold mismatch and its impact on confidence scores.
3.1 MANIFOLD MISMATCH
We consider the general setup of manifold learning (Vural & Guillemot, 2018). We assume X H with H being a Hilbert space and consider classification problem with M > 2 classes, where samples of each class m E ............. are drawn from a probability measure Vm such that Vm has bounded support Mm C X. We further assume that the classes are separated, such that all samples belong to a single class and manifolds are disjoints. Then, v(x, y/) E DD y; E {I...., M and X; Vy. In this setting, show the following results, with proof in Appendix F:
Theorem 3.1. For any pair of manifold M,, Mj. there exists X X E M,,UMj, and X A2 E [0,
2 X2, such that
(i) AA E|A2, Aal x(A) = AxE + (1 - A)x, does not belong to the same manifold as X₁ and x.-
(ii) (x(A)) - x(A2)ll |A1 A2l |X x; 11
In other words, for any pair of class, can find pairs of points where their convex combination can fall in a region of the space (in the line segment) that belongs to neither of the original manifolds. These regions are delimited by the borders of the manifolds (x(1) and x(x2)) the line segment, and can be seen as regions of uncertainty with respect to the true underlying manifold. In practice, convex combinations of points that falls in such regions leads to manifold mismatch, they could either belong to a different class manifold or to no class manifold at all.
Now that we have shown the existence of combinations leading to manifold mismatch, we introduce a corollary of the theorem presented in Liu et al. (2023). We consider X, X' random variables associated to the inputs. and X = (1 - A)X + XX', for a fixed A E [0. 1]. The ground truth conditional distribution of the labels Y is expressed as vector-valued function g : - RM such The mixup-induced label
that P(Y = m|X X = x) = 9m (x:), for each dimension m E can then be assigned based on ground truth Y = arg max Im (X), or from the combination of conditional distribution Y arg max [(1 X)gm (X) + Agm (X').. With that, we can say that the mixup label is noisy when the two assignments disagree, i.e. when Y +Ý.
Theorem 3.2. For any fixed X, X' and X related by X (1 - X)X + XX' for a fixed X E [0.1]. and for samples (x. y), (x', s/) E D, with x = (1 -1- -)xx + Xx' the probability of assigning a noisy label is lower bounded by
P(K* +Vp x) min(A, (1 ())
Wi (Myy
R
where (:))..) is the Wasserstein metric, and R = x'ex ||x - x 11 is the radius of the space (see Appendix F.2).
This result tells us that the probability of assigning a noisy label is lower bounded by the distance between the original manifolds. Thus, to avoid noisy labels, we want to reduce the probability that
4

---

Published as a conference paper at ICLR 2025
Dataset
0.8
CIFARLO
P(UA3))) (
0.6
0.4-
903
0.2
00-
0.00
Nomalloed distano
11.3
1.0
1.5
2.0
Loker
Hight
EFM
Lowkr Highe
Vike
(a)
(b)
Figure 1: (a) Probability that predicted label of mixed samples corresponds to the label of either of the two points used for mixing, depending on the distance between the two points. (b) Performance (Accuracy in %, higher is better, bottom) and calibration (ECE, lower is betler, top) comparison with Resnet34 on CIFARIO (left) and CIFAR 100 (right) datasets. We compare results when mixing only elements with distance Higher (in green) than the median, and Lower (in orange) than the median of all pairwise distances within each batch.
= 0.5 when the original manifolds are far from each other. Furthermore, the distance between two points roughly informs about the distance between the manifolds.
From the above discussion, we assume that in practice, the higher the distance between two points, the less likely their convex combination will fall near the original manifolds, and thus, the more likely a model would assign a different label than the original labels of the two points. We verify this in the following experiments presented in Figure la. Using a Resnet18 (He al., 2016) trained with ERM respectively on CIFARIO or CIFAR 100 (Krizhevsky et al., 2009), we generate 10000 mixed samples X; from data of each test set. Each interpolation coefficient used to obtain X; is sampled from uniform distribution between 0 and 1, We then measure the frequency that the model assigns the same label to x, than the label of either of the points in the pair used to mix (3), or (o)) with respect the distance between the points in the pair. As can be seen, the frequency of assigning the same label decreases with the distance. This confirms that the uncertainty on the label the mixed samples directly depends in practice the distance between the points.
Then, we conduct an empirical analysis to compare accuracy and calibration metrics using the classic ERM, the original Mixup, and modified version of Mixup that selects pairs to mix according to given quantile of the overall pairwise distances within the batch. To have equivalent proportions of possible data to mix with (same diversity), distances lower than given quantile and higher than the quantile o' = 1 q. More specifically, we show in Figure 1b the results for q 0.5 (the median) of the overall pairwise distances within the batch, using Resnet34 (He et al., 2016) on CIFARIO and CIFAR 100 (Krizhevsky et al., 2009) datasets. Detailed values for this experiment with different quantiles q can be found in Appendix D, and implementation details in Section 4. First, we observe that Mixup improves upon ERM's s accuracy, but can degrade calibration depending on the dataset, which is consistent with findings from Wang et al. (2023). Then, when selecting pairs according to distance, sufficiently high proportion of data to mix is necessary to improve accuracy (q 2 0.5). Finally, mixing data with lower distances achieves better calibration opposed to mixing data with higher distances. These results show that there is a trade-off between adding diversity by increasing the proportion of elements to mix, and uncertainty by mixing elements far from each other. Furthermore, it shows that cannot restrict pairs to mix by selecting data solely based on distance, as it can degrade performance by reducing diversity of synthetic samples. To better control this trade-off with Mixup, we propose tailor interpolation coefficients to the training data, from the distance between the points to mix, such that all points and directions can be considered for mixing.
analyze results when mixing only pairs of points with
3.2 WARPED MIXUP
To dynamically change the interpolation depending on the similarity between points, we rely on warping functions wr, warp interpolation coefficients X at every iteration ( depending the
5

---

Published as a conference paper at ICLR 2025
2.0
1.0-
Density of (x,(
1.5
0.8
1.0
0.4
0.75
0.5
Dataset
CIFARIO
CIFARL00
1.0
1.5
Distance between x, and Xo(8)
Normalized
and Xaj)
(a)
(b)
Figure 2: (a) Density of interpolation coefficients wy after warping with the similarity kernel depending on the distance between pairs. (b) Probability that predicted label for mixed samples with SK mixup corresponds to the label of either of the two points used for mixing.
parameter These functions w, are bijective transformations from [0, 1] to [0, 1] defined such:
w. (xx) (x, with Tx u (1 u) du,
(2)
B(T,T)
where Ix, is the regularized incomplete beta function, which is the cumulative distribution function (CDF) of the Beta distribution, B(T,T) r) is a normalization constant and r E IR; is the warping parameter that governs the strength and direction of the warping. Our motivation behind such w. is to preserve the same type of distribution after warping, i.e., Beta distributions with symmetry around 0.5, through Inverse Transform Sampling (see Appendix for the proof):
Proposition 3.3. Let u( 1]). Then (A) Beta(T, f) any r 0.
One should note that according to Proposition 3.3, the initial interpolation parameter should follow uniform distribution for w. (A) follow Beta(T, r) distribution. Thus, in the remaining of the paper, we always draw according to uniform distribution (or equivalently, Beta(1, 1)), which has the additional benefit of removing a as hyperparameter. We thus extend the Mixup framework:
x; wr (At) + wi (X.)) J : + (1 w- (A.))
A ~ Be)
In addition to providing a general framework, using warping functions is more computationally efficient in practice, which discuss in details in Appendix G. In the following part, we present method to select the right depending on the data to mix, to dynamically change the distribution of the interpolation coefficients.
3.3 SIMILARITY KERNEL
Recall that our main goal is to apply stronger interpolation between similar points, and reduce interpolation otherwise. Since the behavior Beta distributions and w, are logarithmic with respect to therefore, the parameter should be exponentially correlated with the distance, with symmetric behavior around 1. To this end, we define a class of similarity kernels, based on normalized and centered Gaussian kernel, that outputs the correct warping parameter for the given pair of points. Given a batch of data x = {x, ) i= E Rddxn the index of the first element the mix E (1,..., n), along with the permutation E G,, to obtain the index of the second element, compute the following similarity kernel:
d (X,, Xo(i))
T(x, i, Tiax, Tsta) = -max exp
(4)
272
| % l1
with d, dx, xo()))
(5)
2j=1 |x, - Xx(5)

---

Published as a conference paper at ICLR 2025
Accuracy 0.967
Accuracy 0.367
Accuracy 0.967
1.0
1.0
0.5
6.0
60 2
-05
-0.
-10
-10
010
X;
X
(a) ERM
(b) Mixup
(c) Similarity Kemel Mixup (Ours)
Figure 3: Decision frontiers and data used during training (circles) and testing (stars) for (a) ERM,
(b) Mixup, and (c) our Similarity Kernel Mixup, on Moons toy dataset.
where d, is the squared L2 distance rescaled by the mean distance over the batch, and Tmax, Tstd respectively the amplitude and standard deviation (std) of the Gaussian, which are hyperparameters of the similarity kernel. The amplitude Tmax governs the strength of the interpolation in average, and Tad the extent of mixing. In practice, we found that these two parameters could be tuned separately, and that choosing good Taid more important than Tmax for calibration. This framework allows measuring similarity between points in any space that can represent them. More specifically, for classification, we use the L2 distance between embeddings, i.e., d,, (h(x,,, h(x((1), while for regression, we use the distance between labels, ie,,, dz (ys, Bo())), following Yao et al. (2022a).
Our motivation behind this kernel is to have distance is lower than average, to increase the mixing effect, and T < 1 otherwise, to reduce the mixing. Figure 2a illustrates the evolution of the density of warped interpolation coefficients wr (A), depending on the distance between the points to mix. Close distances (left part of the heatmap) induce strong interpolations, while far distances (right part of the heatmap) reduce interpolation. Using this similarity kernel to find the correct 7 to parameterize the Beta distribution defines our full Similarity Kernel (SK) Mixup framework. A detailed algorithm of the training procedure be found in Appendix E, and an in-depth discussion about warping and similarity measures in Appendix K.
when the two points mix are similar, i.e., the
In Figure 2b, we reproduce the experiments presented in Figure la, but using our SK Mixup to generate mixed samples, instead of a uniform distribution. We can see that governing interpolation by the distance reduces likelihood of manifold mismatch even when the samples are far away. Then, we also illustrate and compare behaviors of our method on the Moons toy problem in Figure 3. We observe that a model with Mixup (middle) can lead to worst confidence and performance compared to standard ERM (left), due to manifold mismatch. Since the moons are intertwined and non-convex, linear interpolations from the same moon can lie the other. Using our SK Mixup approach (right), we recover the accuracy and achieve more meaningful confidence scores.
4 EXPERIMENTS
4.1 PROTOCOLS
Image Classification We follow experimental settings from previous works (Liu et al., 2022a; Pinto et 2022, Wang et al., 2023; Noh et al., 2023) and evaluate our approach CIFAR-10. 10, CIFAR-100 (Krizhevsky et al., 2009), Tiny-Imagenet (Deng et al., 2009) and Imagenet (Russakovsky et al., 2015) datasets for In- In-Distribution (ID) performance and calibration. We evaluate models trained on CIFARIO and CIFAR 100 on CIFAR 10-C and CIFAR 00-C (Hendrycks & Dietterich, 2018) for covariate shift robustness, and models trained on ImageNet on Imagenet-R (Hendrycks et al., 2021a) and ImageNet-A (Hendrycks et al., 2021b) for Out-Of-Distribuute Resnet34, Resnet50, Resnet101 (He et al., 2016) and ViT (Dosovitskiy et al., 2021) architectures. We evaluate calibration using ECE and AECE (Naeini et al., 2015; Guo et al., 2017), negative log likelihood (NLL) (Hastic et al., 2009) and Brier score (Brier, 1950). and also after finding the optimal temperature on the validation set through Temperature Scaling (TS) (Guo et al., 2017). Results are reproduced and averaged over 4 different random runs, and we report standard deviation.
(OOD) robustness, using
Regression Here again, we follow settings of previous work on regression (Yao et al., 2022a). We evaluate performance on Airfoil (Kooperberg, 1997), Exchange- Rate and Electricity (Lai et al., 2018) datasets using Mean Averaged Percentage Error (MAPE), along with Uncertainty Calibration
7

---

Published as a conference paper at ICLR 2025
Table 1: Comparison of Performance (Accuracy in %) and calibration (ECE, Brier, NLL) with Resnet34 on CIFARIO and CIFARI00. Best in bold, second best underlined.
Methods
Text
Accurncy
BCB
CIFARIO
Brier (U)
NLL
Accuracy (f)
(CB
CIFARI00
Brier (J)
NLL (J)
0.27
73:47
0.17
7.92
1
95.97
0.27
12:30 0.92
8.54 0.64
25.68 + 1.53
78.11 + 0.57
11.92 002-+000 0.73
32.82
0.87
95.76
2.43
Mixup
0.5
95.71
0.26
6,85 + 1,42
7.46 0.54
21.52 + 1,70
77.14 11.14=#00 0.67
8.06 1.19
32.78
+-1 1.11
96.59
3.78
0.22
0:46
1
95.16 = 0.22
1.92 :: 0.06
7.51 0.33
16.80 +: 0.56
74,44 0.49
9.90 4: 0.65
36.94 0.40
106.38
1.47
Mixup IO
0.5
95.31 + 0.17
2.19 + 0.18
7:00++0:24
17.0-0.69
74,45 + 0.6
10:51 + 0.75
37.2++0.77
108.01
2.74
95.12 0.21
2.74 0.21
0.32 7.84
0.95 19.04
74.21
8.09
0.53 36.39
103.08
2.02
0.2
96.29 = 0.07
181 1: 0.24
6.48 0.13
16.14 :: 0.28
78.13
0.52
2.72 0.73
31.33
85,96
2.18
(Ouzz)
SK Mixup
0.8
0.4
0.6
96.00
96.42 0.24
96.36
0.09
0.41
4,89 0.56
3.04 +: 0.26
0.31
6.24 1 0.43
7.15
6.55
18.17 1 0.51
16.38 + 1.01
78.86
79.12
78.63
0.27
0.83
7.98
7.62 1: 0.82
30.68
30.87 4: 1.31
31.21 1 0.31
89.57 0.98
88.32
3.92
1.0
96.25
0.07
5,9 d: 0.8
6.89 4 0.25
19.38 1: 0.78
78.51
0.94
8.73 1.41
31.45
1.28
90.42 4.11
Table 2: Comparison of performance (Accuracy in %) and calibration (ECE and ECE after Tem- perature Scaling), with Resnet101 on CIFARIO, CIFAR 100 and finy-Imagene' datasets. t: Results reported from Noh et al. (2023). Best in bold, second best underlined, for reported results and ours separately.
Methods
Auc.
CIFARIC ECB
TS ECB
Azc.
CIFARTMI ECE
TS ESEC
Aec.
Tiny-Inagenet ECE (L)
ECE
3.06
3.40
ECP!
91.97
4.41
1.72
76.81
13.43
2.92
56.20
2.72
2.72
1.8"
94 18
3.35
1.51
76.91
7.99
4.38
65.52
3,11
2.51
2.18
FLSD"
93:26
3.92
0.93
76.61
3.29
2.04
64,02
1.85
1,85
CRL
95.04
3.74
1.12
72.60
7.29
3.32
65.87
3.57
1,60
CPC
77.50
13.52
66.44
3.93
3,95
MM
5.49
5.49
1.62
1.62
ERM
93,97 = 0.03
4.30
0:01
0,58 d: 0.01
74.27 #: 0.37
14.33 0.17
1.82
2: 0.04
65.7 a: 0.07
4.00 :: 0.22
1,62 :: 0.19
Mixup
94.78 0.13
2.20
1. 0.25
1.47 1 0.01
77.50 1 0.18
5.77
L. 2.81
2.58
1: 0.01
67.80
0.54
5.49 1: 1.11
1.93 1: 0.02
Manifold Mixup
94.75 0.03
2.13
+ 0.12
1.57 #: 0.05
77.26 + 1.35
5.34
= 0.04
3.07 + 0.22
68.57
0.04
6.56 + 0.57
2.11 + 0.01
RegMixup
0.01
0.00
0.08
0.06
0.08
69.76
0.06 13,85
0.84 0.62
34.47 0.17
0.30-
0.08
0.322
0:20
0.02
0.04
0.56
0.16
7,36
0.15
SK Mixup (Qurs)
95.04
0.07
2.20 1 0.12
0.47 0.01
78.20 1 0.66
2,23 0.39
1.11 0.06
67,60 1. 0.01
3/4 1. 0.24
1,06 J 1.001
SK Reghliuu (Chus)
96.02
0.04
2.36
+ 0.59
0.95 = 0.02
79.57 + 0.03
4.52 = 1.21
1.54 = 0.10
69.11 = 0.06
7.19 + 0:38
1.06 + 0.09
Error (UCE) (Laves et al., 2020) and Expected Normalized Calibration Error (ENCE) (Levi et al., 2022) for calibration. Results are reproduced and averaged over 10 different random runs. We also report standard deviation between the runs. A presentation of the different calibration metrics used can be found in Appendix C, along with detailed description of implementation settings and hyperparameters in Appendix H.
4.2 CLASSIFICATION
Effect of Taid We present in Table the performance and calibration results on CIFAR10 and CIFAR100 with a Resnet34, study the effect of varying Tatd We compare the results with Mixup (Zhang et al., 2018) and Mixup-10 (Wang et al., 2023), which mixes between Inputs Only while keeping one-hot labels, for varying values of a. These two versions of Mixup show the trade-off between the effect of confidence penalty (in Mixup) that can hurt calibration on the one hand, and trivial confidence promotion (in Mixup-IO) that can hurt accuracy on the other hand. By varying only Tad. therefore changing the extent of mixing, we can achieve a better and finer trade-off between these two effects, and improving both calibration and accuracy. Furthermore, it shows that our SK Mixup obtains better results than simply changing in Mixup and Mixup-IO. For other experiments, we first performed cross-vale validatio to select Fad- In general, we found that Tat = 0.25 worked for all datasets and used by default Taa 1 (unless stated otherwise). We refer the reader to Appendix J for more details on cross-vaz cooss-vaiddttio..
Comparison with state of the art In Table 2, present extensive comparison of results CIFARIO, CIFAR 100 and Tiny- ImageNet with a Resnet101. We compare accuracy and calibration results reported in the same settings for various approaches from Noh et al. (2023), including implicit
8

---

Published as a conference paper at ICLR 2025
Table Comparison of performance (Aceuracy %) and calibration (ECE, AECE) with Resnet101 in Covariate-Shiff on CIFARIO-C and CIFAR datasets, averaged over all corruptions and degrees of intensities. Best in bold, second best underlined.
CIFARIO-C ECE (4)
CIFAR IO0-C
Methods
Accuracy (1)
AECE (U)
Accuracy (1)
ECE ())
AECE ())
ERM
72.36 + 1.23
21.80 + 0.55
21.79 + 0.55
47.96 + 0.59
31.97 + 0.41
31.96 0.41
Mixup
77.46 + 1.09
14.25 +: 1.03
14.22
+ 1.01
53.18 + 0.61
17.73 4.07
17.73 4.00
Manifold Mixup
77.30 + 0.45
+.38
14.18 + 3.13
53.42
1.8
16.48 + 6.88
16.48 + 6.88
MIT-A
76.2 1: 0.31
0.31
15,46 1.0
15.44 0.98 0.98
53,36
1: 0.27
1.70
23.0 1 0.74
RankMixup
73.39 + 2.44
19.16
19.15 + 2.45 2.45
47.44
+ 2.36
30.99 + 1.14
30.99 + 1.14
RegMixup
80,92 +: 3.43
9.9 = 3.53
9.8 3.45
55.52 1:
16,95 + 0.37
16.95 + 0.37
SK Mixup (Ours)
79.31 = 0.62
6.87 = 0.94
6.89 + 0.94
55.04 # 1.28
9.90 = 0.12
9.9 0.12
SK RegMixup (Ours)
81.69 1: 2.9
5.88 1: 1,53
5.9 1 1,56
57.98 1: 0.32
6.89 1 1,38
6.9 1 1.37
Table Comparison In-Distribution (ID) performance (Accuracy in %) and calibration (ECE, AECE) on Imagenet dataset, and Out-of-distributie (OOD) performance and calibration on Imagenet- and Imagenet-A datasets, with Resnet50 and ViT-S/16. Best in bold, second best underlined.
Arch.
Methods
ImageNet
ImageNet.R R
ImageNet. A
Aoc. (f)
())
())
(t)
(J)
(
(1)
ECE (1)
ERM Mixup
76.06
4.12
4.05
22.31
24.77
24.77
0.71
43
43
76.59
1.8
1.77
24.49
19.2
19.2
0.89
37.92
37.92
Manifold Mixup
76.61
1.77
1.77
24.43
19.68
19.68
0.85
39.05
39.05
RN50
RankMixup
RogMixup
77.82
76.33
2.61
2.07
2.59
2.1
23.33
25.70
24.81
22.67
24.81
22.67
0.85
2.27
43.81
40.14
43.81
40.14
MIT-A
SK Mixup (Ours)
76.2
1.46
1.4
24.67
19.88
19.88
0.99
36.93
36.93
SK RegMixup (Ours)
77.1
1.78
1.77
25.92
18.44
18.44
2.63
34,93
34,93
ERM
69.34
9.03
9.03
15.46
35.93
35,93
1.83
45.03
45.03
Mixup
72.0
5.81
5.8
18.21
29.29
29.29
2.47
41.07
41.07
Manifold Mixup
72.04
6.99
6.95
18.93
30.27
30.27
2.15
43.47
43.47
VIT-S/16
RegMisup
RankMixup
69.99
74.44
9.30
7.36
9.30
7.34
21.01
15.77
32.64
37.37
32.64
37.37
1.77
3.8
46.82
44.4
46.82
44.4
MIT-A
72.81
5.69
5.69
17.6
31.07
31.07
41.8
41.8
SK Mixup (Ours)
71.83
3.89
3.9
18.37
28.56
28.56
2.28
39.33
39.33
SK RigMixxp (Ours)
75.11
2.0
1.98
22.06
26.23
26.23
4.44
38.25
38.25
methods such as LS (Müller al., 2019), FL. (Lin et 2017), FLSD (Mukhoti et al., 2020), MbL.S (Liu al., 2022a), and explicit methods, such as ECP (Pereyra ct al., 2017), MMCE (Kumar et al., 2018), CRL (Moon et al., 2020), CPC (Cheng & Vasconcelos, 2022). We also compare results with Mixup, Manifold Mixup (Verma et al., 2019), RegMixup (Pinto et al., 2022), RankMixup (Noh et al., 2023) and MIT-A (Wang et al., 2023) in the
settings multiple random seeds, using official codes and hyperparameters provided by the authors. We can see from these results that we achieve competitive accuracy and improves calibration compared other Calibration-driven Mixup methods. Note that our approach always improves accuracy and calibration over ERM. We also show that our framework can be combined with RegMixup further improve accuracy by about 1%. Then, we compare results in covariate shift settings in Table 3 against other calibration We observe that our SK Mixup achieves significantly better calibration results, about 3 points (of ECE and AECE) on CIFAR10-C and 7 points on CIFAR 100-C that RegMixup, and that our SK RegMixup achieves both better accuracy and calibration than other methods. Table 4, we also evaluate the scalability of our method on ImageNet dataset, both for ID and OOD performance with ResNet and VIT models. While we achieve slightly lower ID accuracy than other method with our SK Mixup, significantly improve ID calibration. Then, with SK RegMixup, improve calibration over original RegMixup, with similar ID accuracy, and we achieve both better OOD accuracy and calibration than other methods.
Mixup methods.

---

Published as a conference paper at ICLR 2025
Table 5: Performance (RMSE, MAPE) and calibration (UCE, ENCE) comparison on regression tasks.
Best in bold, second best underlined.
Exchange
Eleatricity
MAPE (1)
(CB
ENCG
MAPE (.)
UCB (.)
ENCE (1)
MAPE (1)
UCB
ENCE
ERM
L,720 0.211
107.6
1922
2.10%
0:7.
1924 0.287
0.82% 0.2.2
3.64% 0.0.4
15,263 =: 0.383
0.76% #
22.07%
1.27
Maxup
2,003 2: 2.2225
147.1
34.0
2.1258 0.6
1926 0.264
0.74%
0.22
3.52% 0.0.0
14,944 1: 0.386
1.62% 2: 0.07
24.29%
2.47
Mansfold Mixup
1,964 0.111
126.
= 15.8
2000%
0.64
2.006 0.346
0180%
0.22
3.82%
2.885
14872
4: 0.409
3.69%
2 0.05
24.53%
1.44
RegMisup
1,725 à 0.003
105.0
35.
1.90%
6 0.73
1.988 700
0. 8.90%
0:34
3.66%
0.75
14.790
0.W.
0.66%
6 0.17
23.888
2.255
C. Mixup
L,706 0.100
111.3
32.6
1,908
+: 0.75
1,893
0.222
0.78%
=: 0.20
3,00%
+: 0.64
15.083
= 0.535
0.69%
#: 0.11
23,5415
1.30
SK Mixup (Ours)
1609 3: 0.137
938 =: 25.5
185%
3 0.64
1814 2: 0.341
0.96%
+ 0.21
3.40%
0.56
14.699
a 0.191
0.61%
9,85
23.3%
2:06
Table Efficiency comparison between Mixup methods. We report the number of batch of data in memory at each iteration, the best epoch measured on validation data, time per epoch (in seconds) and total training time to reach the best epoch (in seconds), for Image classification on CIFARIO, CIFAR100 and -nngieeteeet datasets, with Resnet50.
Batch in momory
CIFARIO
CIFAR Time
Tiny-Imagenet
Method
Best Epoch
Time
Total Time
Best Epoch
Total Time
Best Epoch
Time
Total Time
Mixup
186
3162
181
17
3077
89
125
11125
RankMixup
4
147
11485
177
78
13806
80
540
43380
MIT-A
2
189
8694
167
46
7659
82
305
24908
RegMixup
2
173
5536
177
32
5664
94
240
22560
SK Mixup
1
183
28
5138
160
28
4480
69
189
13104
RegMixup
2
175
6825
39
6981
75
316
23700
4.3 REGRESSION
To demonstrate the flexibility of our framework regarding different tasks, we provide experiments on regression for tabular data and time series. Regression tasks have the advantage of having obvious meaningful distance between points, which is distance between labels. Therefore, following Yao al. (2022a), we directly measure the similarity between two points by the distance between their labels, ie., du (yis Vo())- This avoids the computation of distance on embeddings and makes our method as fast as the original Mixup. In Table 5, we compare our SK Mixup with Mixup (Zhang et al., 2018), Manifold Mixup (Verma et al., 2019), RegMixup (Pinto et al., 2022), and C-Mixup (Yao al., 2022a). We can see that our approach achieves competitive results with otatc-of-the-art C-Mixup, in both performance (MAPE) and calibration metrics.
4.4 EFFICIENCY COMPARISON
Our method achieves competitive results while being much more efficient than other state-of-the-art approaches. Indeed, as can be seen in Table 6, our SK Mixup is about 1.5x faster than MIT-A, about 3x faster than RankMixup while using single batch of data like Mixup. We present full comparison in Appendix N.
CONCLUSION
Motivated by calibration in both classification and regression tasks, we present Similarity Kernel Mixup, flexible framework take into account distance between data for linearly interpolation during training, based on a similarity kernel. The coefficients governing the interpolation are warped change their underlying distribution depending on the similarity between the points mix, such that similar data are mixed more strongly than less similar ones, preserving calibration by avoiding manifold mismatch, as we prove that likelihood of assigning noisy label when mixing increases with distance. This provides a more efficient data augmentation approach than Calibration-driven Mixup methods, both in terms of time and memory, improving both accuracy and calibration, even more aut-of-distribution. We illustrate through extensive experiments the effectiveness of the approach in classification as well as regression, spanning multiple neural network architectures including CNNs, ViTs, MLPs and RNNs. We also show that our proposed framework can be combined with RegMixup (Pinto et al., 2022) to further boost performance. Future works include applications to more complex tasks such as semantic segmentation, depth estimation, structured data.
10

---

Published as a conference paper at ICLR 2025
REPRODUCIBILITY STATEMENT
Throughout the paper, we made sure that all our experiments were fully reproducible, describing in details all datasets and architectures considered in Section 4.1, common evaluation metries for calibration in Appendix C, and all hyperparameters and training settings in Appendix H. We also describe how cross validation and the selection of hyperparameters are performed Appendix J.
ACKNOWLEDGEMENTS
This work was supported by the PEPR IA FOUNDRY project SANR-23-PEIA-0001 of the French National Research Agency (ANR). We also thank all anonymous reviewers for providing constructive comments and Andrei Bursuc for providing feedback on an early version of the manuseript.
REFERENCES
Arsenii Ashukha, Alexander Lyzhov, Dmitry Molchanov, and Dmitry Vetrov. Pitfalls of in-domain uncertainty estimation and ensembling in deep learning. In International Conference on Learning Representations, 2020. 3, 17
Raphaël Baena, Lucas Drumetz,, and Vincent Gripon. A local mixup to prevent manifold intrusion. In 30th European Signal Processing Conference, EUSIPCO 2022, Belgrade, Serbia, August 29 Sept. 2, 2022, pp. 1372-1376. IEFE, 2022. 2
Glenn W Brier. Verification of forecasts expressed in terms of probability. Monthly weather review, 78(1):1-3, 1950.
18
Olivier Chapelle, Jason Weston, Léon Bottou, and Vladimir Vapnik. Vicinal risk minimization. In Todd K. Leen, Thomas G. Dietterich, and Volker Tresp (eds.), Advances in Neural Information Processing Systems 13, pp. 416-424 426.422.44 MIT Press, 2000.
Nitesh V Chawla, Kevin W Bowyer, Lawrence O Hall, and W Philip Kegelmeyer. Smote: synthetic minority over-sampling technique. Journal of artificial intelligence research, 16:321-357, 2002. 1,
Jiacheng Cheng and Nuno Vasconcelos. Calibrating deep neural networks by pairwise constraints. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 13709-13718, 2022. 3,9
Muthu Chidambaram, Xiang Wang, Yuzheng Hu, Chenwei Wu, and Rong Ge. Towards understanding the data dependency of mixup-style training. arXiv preprint 10.07677, 2021. 2
Hsin- Ping Chou, Shih-Chieh Chang. Jia- Yu Pan, Wei Wei, and Da-Cheng Juan. Remix: Rebalanced mixup. In Adrien Bartoli and Andrea Fusiello (eds.), Computer Vision ECCV 2020 Workshops Glasgow, UK. August 23-28, 2020, Proceedings, Part VI, volume 12540 of Lecture Notes in Computer Science, pp. 1101 Springer, 2020. doi: 10. 10.1000//77-3 00.0000000----0
2, 24
Damien Dablain, Bartosz Krawczyk, and Nitesh V Chawla. Deepsmote: Fusing deep learning and smote for imbalanced data. IEEE Transactions on Neural Networks and Learning Systems, 2022.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. Ieee, 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x 16 words: Transformers for image recognition at scale. ICL.R 2021. 7
Benjamin Erichson, Soon Hoe Lim, Winnie Xu, Francisco Utrera, Ziang Cao, and Michael Mahoney. Noisymix: Boosting model robustness to common corruptions. In International Conference on Artificial Intelligence and Statistics, pp.
PMLR, 2024. 3
11

---

Published as a conference paper at ICLR 2025
Gianni Franchi, Nacim Belkhir, Mai Lan Ha, Yufei Hu, Andrei Bursuc, Volker Blanz, and Angela Yao. Robust semantic segmentation with superpixel-mix 32nd British Machine Vision Conference 2021, BMVC 2021, Online, November 22-25, 2021, pp. 158. BMVA Press, 2021. 1
Yarin Gal and Zoubin Ghahramani. Dropout Bayesian approximation: Representing model uncertainty in deep learning. In Proceedings of the 33rd International Conference on Machine Learning (ICML-16), 2016. 25
Alison Gibbs and Francis Edward Su. On choosing and bounding probability metrics, International statistical review, 70(3):419-435, 2002. 22
Kristjan Greenewald, Anming Gu, Mikhail Yurochkin, Justin Solomon, and Edward Chien. $kS- mixup regularization for deep learning via optimal transport. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. 3
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Weinberger. On calibration of modern neural networks. In International conference on machine learning. PMLR, 2017. 3, 7, 17, 18, 20
Hongyu Guo, Yongyi Mao, and Richong Zhang. Mixup as locally linear out-of- regulariza- tion. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pp. 3714-3722, 2019. 2
Thien M. Ha and Horst Bunke. Off-line, handwritten numeral recognition by perturbation method. IEEE Trans. Pattern Anal. Mach. Intell., 1965:535--33, 1997. doi:
Trevor Hastie, Robert Tibshirani, Jerome H Friedman, and Jerome H Friedman. The elements of statistical learning: data mining, inference, and prediction, volume Springer, 2009. 7, 18
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 770-778. IEEE Computer Society, 2016. doi: 10.11099VV...201..90. 7,
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness common corruptions and perturbations. In International Conference Learning Representations, 2018. 7
Dan Hendrycks, Norman Mu, Ekin Dogus Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshmi- narayanan. Augmix: A simple method to improve robustness and uncertainty under data shift. In International Conference Learning Representations, 2020. 2,
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang. Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of out-of-distribution generalization. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 1340-8339, 2021a. 7
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 15262-15271, 2021b.
Inwoo Hwang, Sangjun Lee, Yunhyeok Kwak, Seong Joon Oh, Damien Teney, Jin-Hwa Kim, and Byoung-Tak Zhang. Selecmix: Debiased learning by contradicting-pet sampling. Advances in Neural Information Processing Systems, 35:14345-14357, 2022. 1.
Hiroshi Inoue. Data augmentation by pairing samples for images classification. arXiv preprint arXiv:1801.00229, 2018. 1,2
Md Amirul Islam, Matthew Kowal, Konstantinos G Derpanis, and Neil DB Bruce Segmix: Co- occurrence driven mixup for semantic segmentation and adversarial robustness. International Journal of Computer Vision, 131(3):701-16, 2023.
12

---

Published as a conference paper at ICLR 2025
Xuan Kan, Zimu Li, Hejie Cui, Yue Yu, Ran Xu, Shaojun Yu, Zilong Zhang, Ying Guo, and Carl Yang. R-mixup: Riemannian mixup for biological networks. In Ambuj Singh, Yizhou Sun, Leman Akoglu, Dimitrios Gunopulos, Xifeng Yan, Ravi Kumar, Fatma Ozcan, and Jieping Ye (eds.), Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD 2023, Long Beach, CA, USA, August 6- 6-10, 2023, pp. 1073-1085. ACM, 2023. doi: 10.1145/3580305.35990 33999483. 1,2
Mhyn Kim, Wonho Choo, and Hyun Oh Song. Puzzle mix: Exploiting saliency and local statistics for optimal mixup. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine PMLR, 2020. 2
Learning Research, pp.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint
arXiv:1412.6980, 2014. 25
Charles Kooperberg. Statlib: archive for statistical software, datasets, and information. The American Statistician, 51(1):98, 1997. 7
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009. 7,7
Volodymyr Kuleshov, Nathan Fenner, and Stefano Ermon. Accurate uncertainties for deep learning using calibrated regression. In International conference on machine learning, pp. 2796-2804.
PMLR, 2018. 3
Aviral Kumar, Sunita Sarawagi, and Ujjwal Jain. Trainable calibration measures for neural networks from kernel mean embeddings. In International Conference on Machine Learning, pp. 2805-2814.
PMLR, 2018. 3,9
Guokun Lai, Wei- Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long-and short-term temporal patterns with deep neural networks. In The 41st international ACM SIGIR conference on research & development in information reirieval, pp. 95-104, 2018. 7, 25
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017.
Olivier Laurent, Adrien Lafage, Enzo Tartaglione, Geoffrey Daniel, Jean-mare Martinez, Andrei Bursuc, and Gianni Franchi. Packed ensembles for efficient uncertainty estimation. In The Eleventh International Conference on Learning Representations, 2022. 3
Max-Heinrich Laves, Sontje Ihler, Jacob F Fast, Lüder A Kahrs, and Tobias Ortmaier. Well-calibrated regression uncertainty in medical imaging with deep learning. In Medical Imaging with Deep Learning, pp. 393-412. PMLR, 2020. 3, 8, 18
Dan Levi, Liran Gispan, Niv Giladi, and Ethan Fetaya. Evaluating and calibrating uncertainty prediction in regression tasks. Sensors, 22(15):5540, 2022. 3, 8, 19
David A Levin and Yuval Peres. Markov chains and mixing times, volume 107. American Mathemat-
ical Soc., 2017. 21
Siyuan Li, Zedong Wang, Zicheng Liu, Di Wu, and Stan Z. Li. Openmixup: Open mixup toolbox and benchmark for visual representation learning. CoRR, (as/22299.4855. 2022. 2
Soon Hoe Lim, N. Benjamin Erichson, Francisco Utrera, Winnie Xu, and Michael W. Mahoney. Noisy feature mixup. In International Conference Learning Representations, 2022. 3
Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection, In Proceedings of the IEEE international conference on computer vision, pp. 2980-2988. 2017. 3.9
Bingyuan Liu, Ismail Ben Ayed, Adrian Galdran, and Jose Dolz. The devil is in the margin: Margin- based label smoothing for network calibration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 80-88, 2022a. 7,9
13

---

Published as a conference paper at ICLR 2025
Zicheng Liu, Siyuan Li, Di Wu, Zihan Liu, Zhiyuan Chen, Lirong Wu, and Stan Z. Li. Automix: Unveiling the power of mixup for stronger classifiers. In Shai Avidan, Gabriel J. Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner (eds.), Computer Vision ECCV 2022: 17th European Conference, Tel Aviy, Israel, October 23-27, 2022, Proceedings, Part XXIV, volume 13684 of Lecture Notes in Computer Science, pp. 441 458-458. Springer, 2022b. doi: 1,2
10.1007/978-3-000 --011708----311-005-----
Zixuan Liu, Ziqiao Wang, Hongyu Guo, and Yongyi Mao. Over-training with mixup may hurt generalization. In The Eleventh International Conference on Learning Representations, 2023. 2,
Linghui Meng, Jin Xu, Xu Tan, Jindong Wang, Tao Qin, and Bo Xu. Mixspeech: Data augmenta- tion for low-resource automatic speech recognition. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7008-7012. IEEE, 2021.
Jooyoung Moon, Jihyo Kim, Younghak Shin, and Sangheum Hwang. Confidence-aware learning for deep neural networks. In Proceedings of the 37th International Conference on Machine Learning, pp. 7034-00344 PMLR, 2020. 3,9
Jishnu Mukhoti, Viveka Kulharia, Amartya Sanyal, Stuart Golodetz, Philip Torr, and Puneet Dokania. Calibrating deep neural networks using focal loss. Advances in Neural Information Processing Systems, 33:15288- -15999, 2020. 3,9
Rafael Müller, Simon Kornblith, and Geoffrey E Hinton. When does label smoothing help? Advances in neural information processing systems, 32, 2019. 3,9
Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. Obtaining well calibrated proba- bilities using bayesian binning. In Proceedings of the AAAI conference on artificial intelligence, volume 29, 2015. 7
Jongyoun Noh, Hyekang Park, Junghyup Lee, and Bumsub Ham. Rankmixup: Ranking-based mixup training for network calibration. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1358-1368, 2023. 3, 7, 8, 9, 25, 27
Junsoo Oh and Chulhee Yun. Provable benefit of mixup for finding optimal decision boundaries. In International Conference on Machine Learning, pp. 26403-26450. PMLR, 2023. 30
Ragja Palakkadavath, Thanh Nguyen- genuenn-Tang, Sunil Gupta, and Svetha Venkatesh. Improving domain generalization with interpolation robustness. In NeurlPS 2022 Workshop on Distribution Shifts:
Connecting Methods and Applications, 2022. 1,3
Gabriel Pereyra, George Tucker, Jan Chorowski, Lukasz Kaiser, and Geoffrey Hinton. Regularizing neural networks by penalizing confident output distributions. arXiv preprint arXiv:1701 1XXi:1700100545 2017. 3, 9
Francesco Pinto, Harry Yang, Ser-Nam Lim, Philip Torr, and Puneet K. Dokania. Regmixup: Mixup as regularizer can surprisingly improve accuracy and out distribution robustness. In Advances in Neural Information Processing Systems, 2022. 1, 2, 3, 7, 9, 10, 27, 30
Alexandre Ramé, Rémy Sun, and Matthieu Cord. Mixmo: Mixing multiple inputs for multiple outputs via deep subnetworks, In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pp. 803-813. IEEE, 2021. doi: 10.1 N0.100//CCVV99922.2222 .00086. 2
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Scan Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Li Fei-Fei. Imagenet large scale visual recognition challenge. Int. J. Comput. Vis., (15(3):211-252) 2015. doi: 10.1007/S1 1263-116555 011-0011----1-11111-11
Patrice Y Simard, Yann A LeCun, John S Denker, and Bernard Victorri. Transformation invariance in pattern recognition-tangentt distance and tangent propagation. In Neural networks: tricks of the trade, pp. 274. Springer, 2002.
Hao Song, Tom Diethe, Meelis Kull, and Peter Flach. Distribution calibration for regression. In International Conference Machine Learning, pp. 5897-5906. PMLR, 2019. 3
14

---

Published as a conference paper at ICLR 2025
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, (5(1):1929-195) 2014. 25
Cecilia Summers and Michael J Dinneen. Improved mixed-example data augmentation. In 2019 IEEE winter conference on applications of computer vision (WACV), pp. 1262-1270. IEEE, 2019.
Ryo Takahashi, Takashi Matsubara, and Kuniaki Uchara. Data augmentation using random image cropping and patching for deep cnns. IEEE Transactions on Circuits and Systems for Video Technology, 30(9):2917-22)) 2931, 2019. 2
Damien Teney, Jindong Wang, and Ehsan Abbasnejad. Selective mixup helps with distribution shifts, but not (only) because of mixup. arXiv preprint arXiv:2305. /6817, 2023. 1,3
Sunil Thulasidasan, Gopinath Chennupati, Jeff A Bilmes, Tanmoy Bhattacharya, and Sarah Michalak. On mixup training: Improved calibration and predictive uncertainty for deep neural networks. Advances in Neural Information Processing Systems, 32, 2019. 2, 3,
Huan Tian, Bo Liu, Tianqing Zhu, Wanlei Zhou, and S Yu Philip. Cifair: Constructing continuous domains of invariant features for image fair classifications. Knowledge-Bon Baadee Systems, 268:110417,
2023. 3
Yuji Tokozume, Yoshitaka Ushiku, and Tatsuya Harada. Between-class learning for image classifi- cation. In Proceedings of the IEEE conference on computer vision and paltern recognition, pp. 5486-5494, 2018. 1,2
Vladimir Vapnik. Statistical learning theory. Wiley, 1998. ISBN 978-0-41--000000 1
Shashanka Venkataramanan, Ewa Kijak, Laurent Amsaleg, and Yannis Avrithis. AlignMixup: Improving representations by interpolating aligned features. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 19152- 19161. IEEE, 2022. ISBN 978-1-66546-946-3. doi: 10. 10.11099CVR552288.2200 202..01 01888. 1,2
Vikas Verma, Alex I Lamb, Christopher Beckham, Amir Najafi, Joannis Mitliagkas, David Lopez-Paz, and Yoshua Bengio. Manifold mixup: Better representations by interpolating hidden states. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.). Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 6438 6447. PMLR, 09-15 Jun 2019. 2, 9, 10, 25
Elif Vural and Christine Guillemot. A study of the classification of low-dimensional data with supervised manifold learning. Journal of Machine Learning Research, 18(157):1-55, 2018.4
Deng-Bao Wang, Lanqing Li, Peilin Zhao, Pheng-Ann Heng, and Min-Ling Zhang. On the pitfall of mixup for uncertainty calibration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7609-7618, 8,, 2023. 1, 2, 3, 5, 7, 8, 9, 17, 20, 24, 25
Jason Wang, Luis Perez, et al. The effectiveness of data augmentation in image classification using deep learning. Convolutional Neural Networks Vis. Recognit, 11(2017):188, 2017.
Yeming Wen, Ghassen Jerfel, Rafael Muller, Michael W Dusenberry, Jasper Snoek, Balaji Lak- shminarayanan, and Dustin Tran. Combining ensembles and data augmentation can harm your calibration. In International Conference on Learning Representations, 2021.
Ross Wightman, Hugo Touvron, and Herve Jegou. Resnet strikes back: improved training procedure in timm. In NewrIPS 2021 Workshop on ImageNet: Past, Present, and Future, 2021.
Larry Yaeger, Richard I Lyon, and Brandyn Webb. Effective training of neural network character classifier for word recognition. M.C. Mozer, M. Jordan, and T. Petsche (eds.), Advances in Neural Information Processing Systems, volume 9, MIT Press, 1996. 1
Huaxiu Yao, Yiping Wang, Linjun Zhang, James Y Zou, and Chelsea Finn. C-mixup: Improving generalization in regression. Advances in Neural Information Processing Systems, 35:3361-3376. 2022a. 2, 7, 10, 25, 28
15

---

Published as a conference paper at ICLR 2025
Huaxiu Yao, Yu Wang. Sai Li, Linjun Zhang, Weixin Liang, James Zou, and Chelsea Finn. Improving out-of-distribution robustness via selective augmentation. In International Conference on Machine Learning, pp. 25407-25437, PMLR, 2022b. 1,3
Sangdoo Yun, Dongyoon Han, Sanghyuk Chun, Seong Joon Oh, Youngjoon Yoo, and Junsuk Choe. Cutmix: Regularization strategy to train strong classifiers with localizable features. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul. Korea (South), October 27 November 2, 2019, pp. 6022-6031. IEEE, 2019. doi: 10. 10.100//CCV22210000001 1, 2, 30
Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations, 2018. 2, 4, 8, 10
Linjun Zhang, Zhun Deng, Kenji Kawaguchi, and James Zou. When and how mixup improves calibration. In International Conference on Machine Learning, pp. 26135-26160, PMLR, 2022.
Jianchao Zhu, Liangliang Shi, Junchi Yan, and Hongyuan Zha. Automix: Mixup networks for sample interpolation via cooperative barycenter learning. In Computer Visi 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part X 16, pp. 633 649. Springer, 2020. 2
16

---

Published as a conference paper at ICLR 2025
A BROADER IMPACT
This paper presents work whose goal is to improve the calibration of machine learning models. Calibration the process of improving the reliability of the confidence score associated with predictions. This is an important step towards the development of trustworthy models. Having better calibrated models can also help in the detection of biases, overfitting or out-of-distribution data, which can prevent models from being misused in practice.
B LIMITATIONS
The majority of theoretical analysis around mixup have been focused on generalization, while we are interested on calibration. Analyzing the impact on calibration is very difficult, as we also need take into account the effect of Temperature Scaling (TS) (Guo et al., 2017) on a validation set. Indeed, TS can change ordering of results (Ashukha et al., 2020), and, more importantly, Wang el. (2023) observed that calibration using Mixup can be worse than ERM after TS. This is observation that we also confirm our experiments (e.g. in Table 1). Thus, an analysis including temperature scaling would be required first for vanilla Mixup before analyzing our proposed method. This would be significant contribution on its own, and our future works include exploring this direction.
Our approach also introduces two additional hyperparameters. However, since in our we always draw initial parameters À from (1, 1, 1). a is not hyperparameter anymore. Then, Tmaz and Tstd be tuned separately, and, mentioned in Section 4.2, we found that impact calibration was mainly controlled by Tatd. Furthermore, one should note that we always used Tstd 0.25 in image classification experiments, showing our approach
sensitive hyperparameter choice between datasets. We discuss selection of hyperparameters with cross- valsdvvniiation in Appendix J.
17

---

Published as a conference paper at ICLR 2025
C INTRODUCTION TO CALIBRATION METRICS
As discussed in Section 2.2, calibration measures the difference between predictive confidence and actual probability. More formally, with ğ and y Y, respectively the model's prediction and target label, and p its predicted confidence, a perfectly calibrated model should satisfy P(ù = ylp = p) = for p E [0, 1].
We use several metrics for calibration in the paper, namely, ECE, AECE, Brier score and NLL for classification tasks, and UCE and ENCE for regression tasks. We formally introduce all of them here.
C.I METRICS FOR CLASSIFICATION TASKS
NLL The negalive log-likelihood (NLL) is common metric for model's prediction quality
(Hastic et al., 2009). It is equivalent to cross-entropy in multi-class classification. NLL is defined as:
NLL(x, = N Σ log(ply,lx)),
(6)
where p(y; /x. represents the confidence of the model in the output associated to x; for the target class y;.
Brier score The Brier score (Brier, 1950) for multi-class classification is defined as
N
Brier(x,) y)
((aaa)))) - B(i.)))
N
where assume that the target label y; is represented one-hot vector the e possible class, i.e. Y; E R. Brier score is the mean square error (MSE) between predicted confidence and target.
ECE Expected Calibration Error (ECE) is popular metric for calibration performance for clas- sification tasks in practice. It approximates the difference between accuracy and confidence in expectation by first grouping all the samples into M equally spaced bins {Bm with respect to their confidence scores, then taking weighted average of the difference between accuracy and confidence for each bin. Formally, ECE defined as (Guo et al., 2017):
Byl (aco(mm)) conf( Bm)),
BCE
Σ m=1
(8)
with acc(Bm.) B.m. Bm,
10:= the accuracy of bin Bm. and conf(Bm)
Bm.l EiEB,, (x,) the average confidence within bin B
AECE The Adaptive ECE (AECE) is computed similarly to ECE, with the difference that bin sizes are calculated to evenly distribute samples across the bins.
C.2 METRICS FOR REGRESSION TASKS
probabilistic regression model takes x X as input and outputs mean ply (x) and variance of (x) targeting the ground-truth y Y. The UCE and ENCE calibration metrics are both extension of ECE for regression tasks to evaluate variance calibration. They both apply binning scheme with M bins over the predicted variance.
Uncertainty Calibration Error (UCE) (Laves et al., 2020) measures the average of the absolute difference between mean squared error (MSE) and mean variance (MV) within each bin. It is formally defined by
UCE
18

---

Published as a conference paper at ICLR 2025
Bml MSS((Bm) MV(Bm)),
UCE
with MSE( Bm) = BBmm ExEB (My, (x,) yz) and MV(Bm.) = [Bml EieBm. og (xx)².
ENCE Expected Normalized Calibration Error (ENCE) (Levi al., 2022) measures the absolute normalized difference, between rool mean squared error (RMSE) and woot mean variance (RMV) within each bin. It is formally defined by
M Σ
RMSE Bm. - RMV(B.))
ENCE = M
(10)
RMV(Bm)
with RMSE(Bm) LiEB (My, (x:) 11)3 and RMV(Bm) V [mml Lien, oz (x:)²
19

---

Published as a conference paper at ICLR 2025
Table Performance (Accuracy in %) and calibration (ECE, Brier, NLL) before and after Tempera- ture Scaling (TS) (Guo et al., 2017) with Resnet34 when mixing only elements higher or lower than quantile q. Best in bold, second best underlined.
Quaatile
Accuracy
ECE
NLLG)
ECE
TS Boer
(L)
0.0/ Fligher (ERM)
1.27
0.2.2
104
1.04
0.33
0.6
Higher (Mixup)
95.97 0:27
0.64
Lower
95.70 0.24
1.58
7.12 0.36
179 1.01
0.99 0.3
7,08 0.37
16.1
Lower 0.25
95.73 0.18
2.42 a: 0.55
7.12 4: 0.25
19:54 1.1
1.74 1:4-0..3 0.45
7.07
0.26
19.39 #: 1.11
Lower
95.88 0.28
3.04 0.4
6.67 + 0.34
16.68
0.84
1.56 0.78
6.68
0.34
15.86 + +.7.1
C10
Lower
96.16 0.09
1.63
1 0.32
6.33 1 0.14
16.55 0.59
1.12 0.16
6.35
1 0.15
15.20 1044
Lower 0.9
96.31 0.08
3.71 0.34
6.20 0.24
16.65
0.41
1,10 0.05
6.14 0.11
15.16 1. 0.29
Higher 0.9
95.58 0..34
2.72 0.2
7.38 1 0.49
20.54 1.31
1,86 0.25
7,4.1 0.45
20.32 1.25
0.75
0:0.4
1.12
95.38
7.28
20.71
1.23
19.12
Higher Higher 0.1
0.49
18.73
1.34 -.14
17.06 0.99
96.28 0.03
4.38 0.16
6.15 + 0.05
17.18 0.28
1.13=0.11
0.04 6.14
15.24 : 0.37
Lower 0.0/ lligher (ERM)
73:47 1.59
13.0
0.75
39.56 2.19
161 7.92
2,54
0.15
36.17
2.05
100.82 6.93
Lower 1.0/ Higher 0.U (Mixup))
78.11 0.57
11.92
0.73
32.82
0.87
95.76
2.45
2,49
= 0.19
31.06 =: 0.09
87.94 +: 1998
Lower 0.1
75.40 0.553
4.73 0.32
35.87 0.52
108
1,12
3,48
0.24
35,92
105 87
Lower
1:4.:::00
0.17
32,93
0.64
95.43
1.78
4-1.5
52
90.75
1.53
30.64
Lower
79.24 0.7
30.11 1.05
K3 3.32
1.99
0.94 29.72
2.82
Higher 0.9
77.3 0.43
1.35 # 1.66
52
0.78
90.54 2.56
1:92=
32.0 0.59
58.69 # 1.67
Higher 0:75
77.8 1.05
1.77
1.14
31 16 1.23
90 16
4.16
2:20
0.24
31
1.18
16.16 3.86
Higher 0.5
28.74 0.43
4.32
0.96
30.47 0.54
86.96
1.87
2.52
0.22
30.37 1: 0.56
84.64 1: 1.6
Higher 0.25
78.51 0.47
3.35
1.19
30.13
65 2.6
0.26
30.42 0.59
$4.64 1 2.23
0.53
5.32
29.94
85.09
2.23 0.34
1.28
D EFFECT OF DISTANCE ON CALIBRATION
In Table 7, we show the exact results for the plot presented Section 3.1, along with results obtained for different quantiles One should compare results of "Lower q" with "Higher 1 1 g" to have equivalent numbers possible element to mix with (diversity). We repeat our observations here for ease of reading. First, we observe that Mixup improves upon ERM's accuracy, but can degrade calibration depending on the dataset, which is consistent with findings from Wang et al. (2023). Then, when selecting pairs according to distance, sufficiently high proportion of data to mix is necessary to preserve accuracy (q > 0.5). Finally, mixing data with lower distances achieves a better calibration opposed to mixing data with higher distances.
E DETAILED ALGORITHM
Algorithm Similarity Kernel Mixup training procedure
Input: Batch of data B = {(x, p similarity parameters (Tmax; Tsu)) model parameters at the current iteration 0g
+ Ø B
o 6n
(Sample random permutation]
for vi E {1....,,1) n do
A Beta(l,1)
T = T(x, Tiax Tsna)
(Compute warping parameters through Equations (4) and (5))
x; wi, (>i)x, + (1 Wr. (x:)
[Generate new data]
0. Wr. ) + (1 Wr, (x)))
[Generate new labels)
B + B U (x; Ti))
(Aggregate new batch)
end for
Compute and optimize loss over B Output: updated parameters of the model 0++1
We present a pseudocode of our Similarity Kernel Mixup procedure for single training iteration in Algorithm 1. The generation of new data is explained in the pseudocode sequential process for simplicity and ease of understanding, but the actual implementation is optimized to work in parallel on GPU through vectorized operations.
20

---

Published as a conference paper at ICLR 2025
F PROOFS
F.1 THEOREM 3.1
We give the full proof of Theorem 3.1 below.
F.1.1 PROOF OF THE FIRST PART
Proof. For (i), we will separate the cases in two depending on intersection of the convex hulls of the two manifolds M, M,, being empty or not.
Intersection not empty Let first suppose that M, n n., + 0 and take from M, N.M. Then, since the manifolds are disjoint, z belongs either to M,, Mj or none the two.
Suppose that z M,, Since we also have z MM there exists Xg, X E M, and do [0, 1], such that z λoxa + (1 Ao)x,, thanks to convexity. On the line segment [Xs, xt), since z E M; and manifolds are disjoint 3(A,, 22) E [0, 1 > X2 such that Aλ E|22, X(A) = Xxa + (1 X)x; Mj. Then, λo A1(
With the same reasoning, if E Mj. we can find X5. x; E M,, Finally, if z belongs neither in M, nor Mj. then we can find X2. x both either manifold since z E M, n.M, and obtain the same result.
Empty intersection Now, suppose that M,n,, = 0. Without loss of generality, we can consider X E .M; and X E M,. Then, AA [0, 1]. we define x(A) Ax + (1 A)x,. the linear convex combination of the two points weighted by A. Then, since the convex hulls of the manifolds are disjoints, there exists \1 E [0, 1], such that AA A1, E M₁ and AA < A1 x(A) & M, Symmetrically, there exists X2 E [0, 1]. such that AA S X(A) E MMj and AA A2, x(A) & Mj. The convex hulls the manifolds being disjoints, we have that A > A2.
F.1.2
PROOF OF THE SECOND PART
Proof. Result (ii) is obtained directly by rewriting:
[x(A,) - x(A2)ll |||1 (1 \1)x/ - (A2x + (1 - 12)x1)11x
(11)
= -||x| + A1 (xx - xi) - -X A2(xA xi)llx
(12)
1A1 >2|1|x{ X1/12
(13)
F.2 THEOREM 3.2
We start by defining the Total Variation and the Wasserstein metric, and necessary lemmas.
Definition F.1. (Total Variation) The total variation between two probability measures and on X
TV (V v) sup lu( v(E)),
(14)
BCX
where the supremum is taken over all measurable sets E C X. Lemma F.2. (Levin & Peres (2017), Proposition 4.2) Let μ and be two probability measures X. If X is countable, then
1 Σ Iu(x)= -x(x]]- TV (w, v)
(15)
XEX
Lemma F.3. (Coupling Inequality, Levin & Peres (2017), Proposition 4.7) Let i and be two probability measures on X. Then any coupling of random variables (X, Y) with respective marginals I and y satisfies
TV(,, < +Y).
(16)
21

---

Published as a conference paper at ICLR 2025
Definition F.4. (Wasserstein metric) The Wasserstein metric between two probability measures and // on
/
W W1( =
inf Tell(H.)) XxX
||x x'
[1zddm(xx
(17)
where the infimum is taken [(,,, v), defined the set of all probability measures X x X with respective marginals μl and v.
Lemma F.. (Gibbs & Su (2002), Theorem 4) Let and v be two probability measures on X. Then, the Wasserstein metric and the Total Variation distance satisfy the following relation,
WI(K, v) S. R. (V,
(18)
where R x'xx,''xx |X x'Ilxe is the radius of the space.
We prove Theorem 3.2:
Proof. By Lemma F3, we have
P(L* 2 (V((()))))))))))))
However, also have
Then, from Lemma F.2, we obtain
P((P())
Σ
P(Y m|X) - P =\
M
Σ
fm(X) - (X) + Xxm(('))
m=1
And symmetrically,
TV(P(118), 2
1
Σ
P(Y' = m\x)) - P(ý =m|8)1
1 M
Σ (fm(X') - ((1- - (X) (x'))
2 m=1
(1 (YTV( P(Y
which both gives our first inequality:
P( + Ýix)= 2 min(A, (1 - ()))
Now, we come back to our classification problem with M classes, where samples (x, y) x (I....,M .M are drawn from probability measures My respectively supported on the class man- ifolds M,. Then, (P(( x), x), P(Y' = x')) = TV TVM, My'). and using Lemma F.5 concludes the proof.
F.3 PROPOSITION 3.2
We give the proof of Proposition 3.3 below:
22

---

Published as a conference paper at ICLR 2025
Proof. Let 0 and F+ (T, then F, is the cumulative distribution function (CDF) of
Beta distribution with parameters (T, Vz E [0,1], we have:
((w+(A) - x) = (T,T 5x)
(19)
P(A (o(TT)
(20)
= LT,,T)
(21)
where Equation (20) is obtained since F, is continuous and monotonically increasing on [0, 1].
and Equation (21) because À follows uniform distribution on [0, 1]. It follows that (A) BTt
23

---

Published as a conference paper at ICLR 2025
Table Comparison of performance (Accuracy in %), calibration (ECE, AECE) after Temperature Scaling, and total training time (in min), with three different variants. All experiments are using ResNet34 on CIFARIO on a single A100 GPU and reproduced 4 different random seeds.
Method
Accuracy (1)
ECE (L)
AECE (4)
Total training time (in min) (L)
Warping
96.42 + 0.24
0.53 + 0.06
0.71 = 0.05
57.36 = 0.31
Warping w/ lookup
96.04 + 0.2
0.57 + 0.1
0.70 = 0.12
55.59 = 0.4
No warping
96.3 0.1
0.35 = 0.07
0.76 = 0.16
56.88 = 1.03
G BENEFITS OF WARPING
G.1 DISENTANGLING INPUTS AND TARGETS
Using such warping functions presents the advantage of being able to easily separate the mixing of inputs and targets, by defining different warping parameters 7()) and (()):
x; = wi (Ae)x; + (1 wiy (Ax))
X ~ 1)),
(22)
ū: == woo (Ae) + (1 w,vo (o)))
Disentangling imputs and targets can be interesting when working in the imbalanced setting (Chou et al., 2020). Notably, with 7(0) = 1, (()) == O, we recover the Mixup Input-Only (IO) variant (Wang al., 2023) where only inputs
mixed, used in the experiments in Table and with 7(0) x (() 1, the Mixup Target-Only (TO) variant (Wang et al., 2023), where only labels mixed. can also reveal interesting for structured prediction, where targets are structured objects such graph prediction depth estimation.
G.2 COMPUTATIONAL EFFICIENCY
On a more practical side, although the Beta CDF and its inverse have no closed form solutions for non-intege non-integer values of its parameters a and B, accurate approximations are implemented in many statistical software packages. Then, sampling from Beta distributions with different parameters for each pair points cannot be done directly on GPU. Coefficients sampled for each pair need to be sent to GPU at each batch, slowing the training because of CPU-GPU synchronizations. An efficient implementation is to define single torch. distri ions Bet a with parameters a = 3 = 1 (or Uni for)) GPU, and then compute linear approximation of the inverse Beta CDF from precomputed lookup tables, which is common when using inverse transform sampling. We present a comparison in Table in terms of performance (accuracy and calibration) and computation time, of the three different variants of implementation discussed when training a ResNet34 on CIFAR10 on single A 100 GPU (Tmaz Tstd 0.4). The three variants achieve comparable performance, and the time difference is in favour of using warping functions and lookup tables.
24

---

Published as a conference paper at ICLR 2025
H DETAILED EXPERIMENTAL SETTINGS
Image Classification On CIFARIO and CIFARIOO, we use SGD as the optimizer with a momentum of 0.9 and weight decay of 10-4, a batch size of 128, and the standard augmentations random crop, horizontal al f flip and no wormall limelliaatit ion. Models are trained for 200 epochs, with an initial learning rate of 0.1 divided by a factor 10 after 80 and 120 epochs. On Tiny- ImageNet, models are trained for 100 epochs using SGD with an initial learning rate of 0.1 divided by a factor of 10 after 40 and epochs, a momentum of 0.9 and weight decay of 10 10- We use batch size of 64 and the same standard augmentations. On Imagenet, ResNet-50 (RN50) models are trained for 100 epochs using SGD with an initial learning rate of 0.1, cosine annealing scheduler, a momentum of 0.9 and weight decay of 10-4. We use total batch of 256 split over 3 GPUs and standard augmentations. For ViT models, we train them for 300 epochs using A optimizer, an initial learning rate of 0.01 divided by factor 10 after 100 and 200 epochs, with total batch size of 128 split over GPUs. Regression Following Yao et al. (2022a), we train a three-layer fully connected network augmented with Dropout (Srivastava al., 2014) on Airfoil, and LST-Attn (Lai et al., 2018) on Exchange-Rate and Electricity. All models are trained for 100 epochs with the Adam optimizer (Kingma & Ba, 2014), with a batch size of 16 and learning rate of 0.01 on Airfoil, and batch size of 1 128 and learning rate of 0.001 on Exchange-Rac Rate and Electricity. To estimate variance for calibration, we rely on MC Dropout (Gal & Ghahramani, 2016) with a dropout of 0.2 and 50 samples.
Method-specific hyperparameters On classification datasets, we used Tmar = 1 and Tstd 0.25 unless stated otherwise. We used the hyperparameters provided by the authors to reproduce state- of-the-art methods, namely, a 1 and ÁA 0.5 for MIT-A (Wang et al., 2023), and = 0.1, a = 2.0 and a = 1.0 for CIFAR 10/100 and Tiny-ImageNet respectively, and Q = 4, for RankMixup (M-NDCG) (Noh et al., 2023). To compare with Mixup, we use a 0.2. the best performing one found in Thulasidasan et al. (2019), and the same value for Manifold Mixup (Verma et al., 2019). On Imagenet, we set Tmaz 0.1 and Tstd 0.25, and regularization term (in SK RegMixup), we set Taar = 10 and keep the same Tstd- On regression datasets, used (Tmmayy Tstd) = (0.0001, 0.5) Airfoil, (Tmaz: (Tta) (5, 1) on Exchange Rate, and (Tmaas Tatd) = (2, 0.2) on Electricity. We followed results from Yao et al. (2022a), and fixed 0.5 on Airfoil, a 1.5 on Exchange Rate and a = 2 on Electricity, for Mixup, Manifold Mixup and C-Mixup. Additionally, we searched for the best bandwidth parameter for C-Mixup with cross- volsdaaai,,,,on, and found 0.01 on Airfoil, 0.05 on Exchange Rate, and 0.5 on Electricity. Likewise, we searched for the best a for RegMixup, and found = 0.5 on Airfoil, 10 on Exchange-Rate and a = 10 on Electricity.
0.1. When using SK Mixup
25

---

Published as a conference paper at ICLR 2025
Accuracy 957
100
Accuraty = 0,800
100
Accuracy 0.967
1.00
0.75
0.55
0.75
0.50
0.00
1010
2000
X1
X1
ERM
(b) Mixup
(c) Similarity Kernel Mixup (Ours)
Figure 4: Decision frontiers and data used during training (circles) and testing (stars) for (a) ERM, (b) Mixup, and (c) our Similarity Kernel Mixup. on Circles toy dataset.
Table 9: Comparison of Performance (Accuracy %) and calibration (ECE, Brier, NLL) after Temperature Scaling (TS) with Resnet34 CIFARIO and CIFARIO0. Best in bold, second best underlined.
Methods
CIPARIO
CIFARI00
Accuracy (f)
'SS ECB
'SS Brier
TS (L)
Accurncy
TS BCB
TS Brier
TS NLL
0:11
17.50
73.47 1.59
+
10082
1
95.97 = 0.27
1,36 0.13
6,53 0.36
16.35 = 0.72
78.11 0.57
2.49::0.119
31.06 =: 0.69
87.94 :: 1.98
Mixup
0.5
98.71
0.26
1,33 0.08
703 0.46
17.47 + 1.18
77.14 + 0.67
2.7 0.36
32,01 + 0.93
91.22 3.05
0.1
95.37
0.22
1,13 + 0.11
7.37 0.36
17.43 = 0.79
76.01 + 0.62
2.24
33.41 +: 0.57
93.96
1
95.16 = 0.22
0.6 0.11
7.3
15.56 =: 0.67
74.44 1 0.49
2.02 0:144
35.25 :: 0.43
96.5
1.62
Mixup
0.5
98.31
0.58- 0.06
7.12-0.2
15,09
74.45
0.6
0.09
35.2 0.58
96.75
0.1
95.12 0.21
0.09
=
15.76 = 0.55
74.21 0.46
2.31
35.38 4: 0:48
98.24
1.81
0.2
96.29 0.07
1.35 + 0.14
6.37= 0.11
15.97 = 0.22
78.13 + 0.52
1,31 1:31::00 0.23
0.23
31.18 +: 0.72
85.38 + 1.92
Mixup
96.36
0.09
0.52 0.08
0.09
15.7 2: 0.29
78.63 1 0.27
0.83
0.300
30.41
0.30
81.97 :
(Ours)
0.8
96.00
0.41
0.56 + 0.05
6.63 0.58
16.64 : 1.27
79.12
0.52
1666 0.16
29.75 :: 0,57
82.82 1.62
96.25
0.12
6,31
16.14
78.51
0.94
+.09
1.04
85.06
1 ADDITIONAL RESULTS
1.1 VISUALIZATION
We provide another visualization in Figure on the Circles toy dataset. As interpolations of data from the external circle can fall into the inner one, training with Mixup lead to worst confidence and performance compared to standard ERM, but SK Mixup recover the accuracy while achieving more meaningful decision boundaries and confidence scores.
1.2 EFFECT OF Tsro
We present in Table results associated Table 1 in the main text, but after temperature scaling. We have similar observations than discussed in Section 4.2.
1.3
COMPARISON WITH STATE OF THE ART
We show in Table 10 calibration and predictive performance results associated to Table 2, using the AECE calibration metric, before and after temperature scaling.
26

---

Published as a conference paper at ICLR 2025
Table 10: Comparison of performance (Accuracy in %) and calibration (AECE and AECE after Temperature Scaling), with Resnet101 on CIFAR10, CIFARI00 and Results reported from Noh al. (2023). Best in bold, second best underlined, for reported results and ours separately.
datasets. f:
Methods
Aec. (1)
CIFARIU ABCE (.)
ABCE ())
Ace.
CIBARTOO ABCE
(EC
Acc. (T)
Tlay-Imogeset ABCE (4)
TS ABCE (1)
MMCE
94.99
3335
12.9
77,82
13,42
2,5
66.44
3.38
3.38
ECP
93.97
76.81
13.42
56.20
2.70
2.70
93.59
3.23
1.37
76.12
3,22
2.51
64.02
2.09
2.09
FLSD!
93.26
3,67
0.94
76.61
3.29
2,04
61.02
1811
1.1
CRL
95.04
3,73
2.03
77,60
7.14
3.31
63.87
3.56
1,52
CPC
77.45
ERM
90.97 0.03
4.30 0.01
0.63
0,01
14.27
0.37
14.32 0.17
1882 0.05
65.7 2: 0.07
3,00 0.24
1.57 1:0.0
Mixup
94.78 0.13
2.73
0.0.2
2.59 0,02
77.30
0.18
5.77 2.78
2.67
0.02
67. 30 0.54
5:45
1.15
1.68 1.001
Mandfld Mixup
94.75
0.03
2.62
0.02
2.61 0.05
77.76
1.35
5.32 + 0.04
3.16 0.17
68.57 0.04
6.55 0.58
2.06 0.1
RegMixup
95.89
0.00
1.55
1 0.02
1.30 0.06
76.85
0.08
6.70 0.09
2.67 0.08
69.76 0.67
10.19 0.06
1.0 0.03
KankMiuup
94.42 0.17
0.17
2.82
0.22
0.63 0.14
77.27
0.08
11.09 2: 0.22
2.04 0.19
65.40
0.01
13.85 9.60
1.44 0.06
MIT-A
05.27
0.01
2.37
0.02
1,59
0.01
77.33
0.56
9.54 + 0.48
0.05 2.32
68.03
0.09
7.84 0.10
1.57 0.55
SK Vixup (Our)
05.04
0.07
2.38
0.05
0.98
0.05
78
0:46
2.37 0.34
1,38 0.11
67.60
0.01
3,17 0.22
1.19 0.02
SK RegMixup (Durs)
96.02
0.04
2.40
0.50
0.03
79.57
0.03
1.34
69.11
0.06
7.17 0.42
1.06 1.100
Table 11: Comparison of Accuracy and ECE, before and after Temperature Scaling (TS), with cross- cross-validation when varying Tsd. Results are obtained with a Resnet50 and averaged over 4 splits.
Dataset
Tstd
Accuracy
ECE (L)
TS ECE (L)
0.25
95.18
2.87
0.97
0.4
95.23
3.31
1.42
0.5
95.37
2.7
2.08
0.6
95.45
3.15
1.99
0.75
95.57
3.44
2.05
CIFAR10
0.8
95.44
3.28
2.07
1
95.57
3.44
2.05
1.25
95.46
2.71
2.25
1.5
95.29
3.42
2.36
2
94.93
3.24
2.43
0.25
77.74
3.98
1.77
0.4
78.26
4.24
2.4
0.5
78.37
3.45
3.61
0.6
78.29
2.9
4.08
0.75
78
3.65
3.91
CIFARI00
0.8
78.03
3.33
4.27
1
77.35
4.82
3.4
1.25
77.31
3,98
3.88
1.5
77.92
4.19
4.06
J ON THE SELECTION AND SENSITIVITY TO HYPERPARAMETER
As discussed in Section 3.2, we always draw initial interpolation parameters À from Beta(1, removing
hyperparameter. Then, Tmar and Tatd be tuned separately, and, mentioned in Section 4.2, we found that impact on calibration was mainly controlled by Tsta- We selected the values giving the best trade-off between accuracy and calibration using cross-validation, with stratified sampling on a 90/10 split of the training set, similarly to Pinto et al. (2022), and average the results across different splits. We detail Table 11 cross-validation results (Accuracy and ECE) showing sensitivity to Tatd CIFAR 10 and CIFAR100 datasets, using a ResNet50. These results lead us using Tstd 0.25 for all image classification experiments, and show that our approach is not that sensitive to hyperparameter choice between datasets. For Imagenet, we set Tmar = 0.1 to follow commonly used value a = 0.1 in Mixup for this dataset.
27

---

Published as a conference paper at ICLR 2025
Performance (RMSE, MAPE) and calibration (UCE, ENCE) comparison for different Table 12:
warping functions on Airfoil dataset. Averaged over 10 random seeds.
Warping
RMSE (L)
MAPE ())
UCE ())
ENCE (U)
Sigmoid
2.892 + 0.343
1,733 + 0.229
117.95 = 45.26
0,018 + 0.008
Beta CDF
2.807 0.261
1.694 + 0.176
126.02 + 23.32
0.018 0.005
Inverse Beta CDF
2.707 + 0.199
1.609 + 0.137
93.79 + 25.46
0.019 + 0.008
Table 13: Performance (RMSE, MAPE) and calibration (UCE, ENCE) comparison for different similarity distance Airfoil dataset. Averaged over 10 random seeds.
Warping
RMSE (1)
MAPE (J)
UCE (J)
ENCE (J)
Input distance
2.848 + 0.355
1,706 0.215
105.23 26.32
0.018 + 0.008
Embedding distance
2.737 = 0.205
1.636 + 0.142
101.62 = 21.94
0.016 = 0.005
Label distance
2.707 + 0.199
1.609 0.137
93.79 25.46
0.019 + 0.008
K
ON THE CHOICE OF SIMILARITY KERNEL
K.1 ON WARPING FUNCTIONS
As discussed in Section 3.3, the choice of the similarity kernel highly dependent on the warping function, and more specifically to the correlation between the warping parameter and the shape of the warping function. Given our choice of using the inverse of the Beta CDF as Wr, and that its shape is logarithmically correlated to т, the choice of Gaussian kernel seems natural to have exponential correlation with the distance. Then, the normalization and centering avoid dealing with different range of embedding values between datasets and architectures when choosing correct value of Tstd. Regarding the choice of warping function, as mentioned in Section 3.2, any bijection with a sigmoidal shape could be considered. Besides the inverse of Beta CDF, we also tried the Beta CDF and the Sigmoid (A + 1+cA/r ). As they all have logarithmic correlations wrt T, we used our Gaussian similarity kernel for the three of them. We compare performance on Airfoil dataset after finding the best parameters in each case in Table 12. The inverse Beta CDF have both the advantage of better results, while preserving the underlying Beta distribution by inverse transform sampling, as we show in Proposition 3.3.
K.2
ON EMBEDDING SPACES AND DISTANCES
K.2.1
REGRESSION TASKS
As discussed in Section 3.3, in regression tasks, we considered either input, embedding or label distance:
Input distance: (x,, (x)))
Embedding distance: d,, (h(x;), h(xx(i)))
Label distance: (u.
Input and label distances both have the advantages of inducing almost no computational overhead, as opposed to embedding distance that requires an additional forward pass in the network. We report results comparing the three options on Airfoil dataset in Table 13. We found that label distance was the best performing one, which is in line with results from Yao et al. (2022a), and what we use in the experiments in Section 4.3.
K.2.2 CLASSIFCAATIO TASKS
In classification tasks, as we lack a meaningful distance between label, we restricted our choice between input and embedding distance. Even though computing distances directly in the input
28

---

Published as a conference paper at ICLR 2025
Table 14: Comparison of performance (Accuracy in %), calibration (ECE) after Temperature Scaling, for two different similarity distance. All experiments are using a ResNet34 on CIFARIO and CIFARION 100 datasets and averaged over different random seeds.
Dataset
Distance
7sad
Accuracy (f)
ECE (4)
0.2
95.82 + 0.19
0.41 + 0.13
0.4
96.10 + 0.17
0.46 + 0.06
Input distance
0.6
95.87 = 0.47
0.6 0.13
0.8
96.12 0.15
0.70 + 0.08
1.0
96.01 + 0.27
0.74 0.19
CIFARTO
0.2
96.29 + 0.07
1.35 0.14
0.4
96.42 + 0.24
0.53 + 0.06
Embedding distance
0.6
96.36 + 0.09
0.52 0.08
0.8
0.56 9..0
1.0
96.25
+ 0
0.2
78.43 0.52
1.51
0.4
1.35 0.35
Input distance
0.6
78.36
0.20
0.8
78.50 + 0.41
0.16
78.50
0.83
1.53 0.08
CIFAR100
0.2
78.13 0.52
1.31 + 0.23
0.4
78.86 + 0.83
1.42 + 0.19
Embedding distance
0.6
78.63 + 0.27
1.74 + 0.30
0.8
79.12
0.52
1.66 + 0.16
1.0
78.51 + 0.94
1.49 + 0.03
space faster than relying on embeddings, the latter achieves a better trade-off between accuracy and calibration improvements, which motivated us to use embedding distances as similarity in our experiments in Section 4.2, We compare in Table the two distances, for the different value of Tatd- using a Resnet34 on CIFAR10 and CIFARIO R100 datasets.
29

---

Published as a conference paper at ICLR 2025
Table 15: Comparison of performance (Accuracy in %) and calibration (ECE and ECE after Tempera- ture Scaling), using simple baseline with Resnet101 on CIFARIO, CIFARI00 and Tiny-Imagene
datasets.
CIFARIO ECE (L)
CIFARI00
Tiny-Imaeenet ECE (4)
Olow
Chigh
Acc. (t)
TS ECE (4)
Acc. (1)
ECE (4)
TS ECE ())
Acc. (t)
TS ECE (U)
5
0.1
95.55
1.32
0.69
79.35
5.13
1.55
66.46
1.85
1.52
0.5
6.68
76.95
67.15
1.37
0.1
93.43
2.92
1.12
75.90
1.15
1.12
67.65
1.59
1.18
10
0.5
95.15
5.71
0.59
76.63
6.48
1.65
67.83
6.49
1.16
Table 16: Comparison of performance (Accuracy in %) and calibration (ECE and ECE after Tempera- ture Scaling), using non-linear mixing with Resnet101 CIFARIO, CIFAR 100 and Tiny-Imagenet datasets.
CIFARIO ECE (4)
CIFARI00 ECE (1)
Tiny-Imagene ECE (Q)
Methods
Acc. (1)
TS ECE (U)
Acc. (f)
TS ECE (L)
Acc. (1)
TS ECE (.)
CutMix
95.89
2.14
0.84
78.58
9.49
2.99
67.79
6.25
1.62
SK CutMix
95.25
1.96
0.5
78.65
6.70
2.21
67.86
2.94
1.98
L COMPARISON WITH A SIMPLER BASELINE
By defining fixed Beta distributions depending on the distance, we can have a more efficient (in terms of computation time) implementation. However, we also expect worse results as it will lack granularity in the behavior wrt to the distance. We experimented with defining two separate Beta distributions, one with parameter Clow, for pairs with distance lower than average, and the other with parameter Ohigh. for pairs with distance higher than average. We present in Table 15, results with the same settings Table 2. We see that using lower (higg improves calibration, which confirms the high level idea of reducing interpolation for high-distance pairs. However, finding a good pair of parameters (Coww; (tighh is more dataset-dependent than relying on our similarity kernel approach.
M COMBINATION WITH NON-LINEAR MIXING
Combining our method (or other calibration-driven method) with non-linear mixup for images, like CutMix (Yun et al., 2019), can be done, since the changes are in different parts to each other. However, extensive experiments in Pinto et (2022) (in their Appendix H) have shown that combining their RegMixup with CutMix is not trivial as it does not always improve predictive performance and robustness. We also expect that introducing a non-linear mixing operation will not always lead the same improvements of calibration with our method, since theoretical groundings are always based on the linear mixing operation. Recent results from Oh & Yun (2023) has also shown that the distortion of the training loss induced by non-linear mixup methods, like CutMix, leads to different and less optimal decision boundaries than the loss obtained with linear mixup. We leave for future work the theoretical derivations of using the non-linear mixing operation with the aim to improve calibration and robustness. Nonetheless, we present in Table 16 first results with CutMix and SK CutMix, in which apply our similarity kernel to change dynamically the Beta distribution in CutMix, with the same settings as Table 2. We can see that SK CutMix can improve accuracy over CutMix. Although SK CutMix can also improve calibration over CutMix, since CutMix does not improve calibration over Mixup, SK CutMix does not lead to better calibration than SK Mixup.
30

---

Published as a conference paper at ICLR 2025
Table 17: Efficiency comparison between Mixup methods. We report the number batch of data in memory at each iteration, the best epoch measured on validation data, time per epoch (in seconds) and total training time to reach the best epoch (in seconds).
(a) Image classification on CIFARIO, CIFAR100 and Tiny-imagenet datasets, with Resnet50.
Batch
CIFARIO
CIFAR
uny-IImggnnt
Method
memory
Best Epech
Time
Total Time
Best Epoch
Time
Total Time
Best Epoch
Time
Total Time
Mixup
186
17
3162
181
17
3077
89
125
11125
CutMix
4224
24
3552
143
9867
RankMixup
147
78
11485
177
78
13806
80
540
43380
MIT-A
8694
7659
305
24908
RegMixup
173
5536
177
32
5664
94
240
22560
SK Mixup
183
5138
160
28
4480
69
189
13104
SK CutMix
5880
124
30
3720
185
13135
SK RegMixup
175
6825
179
39
6981
316
23700
(b) Time series regression Electricity dataset.
Methods
Batch in memory
Best Epoch
Time
Total Time
C-Mixup
87
46.51
4046
MegMixup
85
11.46
975
SK Mixup (Ours)
78
11.16
867
N FULL COMPUTATIONAL EFFICIENCY COMPARISON
We present in Table 17 the full comparison of efficiency with all baselines. As can be seen, SK Mixup is about 1.5x faster than MIT-A, about 3x faster than RankMixup, and about 4x faster than C-Mixup. Additionally, we use single batch of data for each iteration, while MIT-A requires training on twice the amount of data per batch, and four times the amount of data per batch for RankMixup. This adds significant memory constraints, which limits the maximum batch size possible in practice. Unlike C-Mixup, our approach does not rely on sampling rates calculated before training, which add of computational overhead and are difficutt obtain for large datasets.
31


Reviews:
[{"review_id": "04rP6Zp51P", "reviewer": "Reviewer_ZK3c", "summary": "This paper proposes a novel Mixup framework that employs a Similarity Kernel (SK) called SK Mixup to achieve a stronger interpolation between similar points while reducing interpolation otherwise. As a motivation of this study, the authors defined the concept of manifold mismatch, which can negatively impact the calibration of confidence in Mixup. They conducted experimental validation to assess the impact of this phenomenon on the distance between points to mix. Following the presentation of SK Mixup, the effectiveness of the proposed approach in alleviating the manifold mismatch was demonstrated through extensive experiments in classification and regression. Nevertheless, the extension to more complex tasks and theoretical analysis on the regularization effect of SK Mixup still need to be undertaken -- This is why the authors did not receive the highest score in *Contribution*.", "questions": "**Question 1.** Why did not SK RegMixup be applied to ViT in Table 4? Is there any potential challenges or limitations of applying SK RegMixup to ViT architectures?\n\n**Question 2.** [line 976-978] In the proof of the first part in Theorem 3.1., whatever the value of $\\lambda\\_{1}$ is, $\\lambda$ can be 0. When $\\lambda=0$, then $\\tilde{\\mathbf{x}}(\\lambda)=\\mathbf{x}\\_{l} \\in \\mathcal{M}\\_{j}$, not $\\mathcal{M}\\_{i}$, and so does symmetrical case. In my opinion, it would be $\\forall \\lambda \\geq \\lambda\\_{1},~\\tilde{\\mathbf{x}}(\\lambda)=\\lambda \\mathbf{x}\\_{k}+(1-\\lambda)\\mathbf{x}\\_{l} \\in \\mathcal{M}\\_{j}$. The authors are kindly requested to examine and fix them if there is an error.\n\n---\n\n**Things to improve the paper that did not impact the score:**\n\n* Figure 4 caption: there is no result about *Circles* toy datasets. The mention of it from the caption would be removed if it's not intended to be included\n\n* [line 442] The ECE and AECE of MIT-A in ImageNet-R would be in bold. The authors have to double-check their result highlighting in Table 4 to ensure consistency and accuracy.\n\n*  In Table 4, is there any reason why ECE and AECE are exactly same in OOD settings? The authors are kindly requested to explain why ECE and AECE are identical in OOD settings or, if this is unexpected, verify whether there might be an error in the reporting or calculation of these metrics.\n\n* The experimental results show that the SK RegMixup is an effective method. However, Table 6 does not provide the efficiency of the SK RegMixup in terms of computational cost. It is recommended to include computational cost metrics for SK RegMixup in Table 6, similar to what they've provided for other methods.\n\n* [line 879] Typo: cccross", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "* The proposed approach was well motivated by targeting the appropriate research problem of calibration-driven Mixup methods in classification and regression: Manifold mismatch. Furthermore, this problem was theoretically defined and proved with the supplement materials.\n\n* The proposed method was in a clear and well-organized manner, and the proof was provided in a manner that supported its conclusions.\n\n* The research problem was validated through experimental results; a comparison was made with the baseline.\n\n* Comprehensive experiments in classification and regression, with a particular focus on performance and computational cost, demonstrated the efficacy of the proposed approach.", "weaknesses": "There is no weakness in the paper that would justify its rejection.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wpRIjNkVSj", "reviewer": "Reviewer_XhfP", "summary": "This paper first demonstrates that distance between the data used in mixup can impact the likelihood of manifold mismatch (a phenomenon where mixed samples lie outside of the class manifolds of the original data used for mixing). It then proposes an efficient framework to mitigate the occurrence of manifold mismatch. The key idea is to dynamically change the distributions of mixing coefficients via a similarity kernel that measures the distance between the mixed samples. Empirical results are provided to demonstrate the effectiveness of the proposed method.", "questions": "- In Eq. (3), there is a -1 inside the exponential. Why not absorb it into $\\tau_{max}$? Are there any reasons why it is presented that way? \n- In the classification scenario where the data samples (but not the labels) are noisy and corrupted, thus blurring the true distance between the samples used in mixing, how would the proposed method perform?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Overall the paper is well written and is easy to follow\n- An innovative yet efficient method is proposed to improve mixup so that manifold mismatch likelihood is reduced, and better calibration and accuracies can be achieved\n- Adequate empirical results on both classification and regression tasks are provided to demonstrate the proposed method", "weaknesses": "- While Theorem 1 shows the existence of manifold mismatch, it does not tell us anything about the assumption (which is used in the paper) that the higher the distance between two points, the more likely their convex combination will fall outside of the original manifolds, and thus, the more likely a model would assign a different label than the original labels of the two points. Some theoretical results in establishing the validity of such assumption would further strengthen the paper. \n- There are some missing papers that could be worth mentioning in the related work. In particular, mixup related works such as [1,2,3]. \n- For the experiments, it is not clear how the proposed method would compare with the method proposed by [1,2,3], and also how would it perform for more complex datasets like ImageNet, ImageNet-C, ImageNet-R and ImageNet-P.\n\n---\n\n[1] Dan Hendrycks, Norman Mu, Ekin D. Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshminarayanan. AugMix: A simple data processing method to improve robustness and uncertainty. Proceedings of the ICLR, 2020\n\n[2] Soon Hoe Lim, N. Benjamin Erichson, Francisco Utrera, Winnie Xu, and Michael W. Mahoney. Noisy feature mixup. Proceedings of the ICLR, 2022.\n\n[3] Erichson, N. Benjamin, Soon Hoe Lim, Francisco Utrera, Winnie Xu, Ziang Cao, and Michael W. Mahoney. Noisymix: Boosting robustness by combining data augmentations, stability training, and noise injections. Proceedings of the AISTATS 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6cN5nDQV2D", "reviewer": "Reviewer_Fzdf", "summary": "The paper has come up with a concept called manifold mismatch, which is a phenomenon in Mixup that can harm the generalization and calibration performance of the trained models. The paper also show that such a manifold mismatch behavior is correlated to the distance between the data points being paired and mixed in Mixup. Then, by applying a similarity kernel, the paper proposes a variant training algorithm of Mixup that can dynamically adjust the mixing coefficient depending on the distance of the data points being mixed. Finally, the authors have empirically verified the effectiveness of the proposed algorithm on various models and tasks.", "questions": "1. Line 011 or 012 on page 1. \"Along with improved performance\", what performance, please make it specified\n2. Line 012 or 013 on page 1. \"improving calibration and predictive confidence\", from my understanding they are the same thing, no need to use \"and\" here in my opinion.\n3. Line 014. Infact in this line the paper has mentioned \"calibration of confidence\", what is calibration of \"confidence\", again, in my opinion it's just calibration.\n4. Line 019 to 020. Again, please specify what \"performance\" it is when mentioning \"improve performance\".\n5. Line 054. Here the paper mentions \"label noise\", can the authors provide a brief description of it like the way they have described manifold intrusion in previous text?\n6. Line 140. Again, please specify the term \"performance\".\n7. Is there and related research works about calibration-driven Mixup methods in regression tasks?\n8. Line 161. If possible it's much better to turn this title to the next page, it would look better.\n9. Line 162 to 169. The notation preliminaries in this paragraph feel a bit too simple and careless. For example, when defining a dataset, one may want to first define a data space and label space, indicating that these spaces are subspaces of some vector spaces of certain dimensions, and then indicate that the training dataset is of some size with the examples being drawn from some data distribution over the data space, etcs.. Putting those details in one single line feels careless. Also, what is $M$? Many would of course just simply take it as the number of classes in the classification problems, but it also needs to be specified in the paper. \n10. Line 162 to 169. There is also some notations that are inconsistent. The paper first indicates that the labels are $M$-dim vectors, then what is the input and output dimension of the encoder $h_\\phi$? The way it is presented make the model output $f_\\theta(\\text{x})$ feels like a scalar, while it should be a vector the same dimension as the labels since $\\hat{y}:=f_\\theta(\\text{x})$.\n11. Line 193 to 194. \"bounded support $\\mathcal{M}_m\\subset\\mathcal{H}$\". Is this also how the manifolds are defined in this paper?\n12. Line 194 to 195. To assume that classes are separated, it means all samples belonging to \"different classes manifolds\" should disjoint, is that correct?\n13. Line 208. Why would the mixed points' belonging \"to no vlass manifold at all\" be a bad thing? In my opinion, when using Mixup for training, especially when the data dimension is high, it's actually mostly the case that the mixed point fall into the \"void\" areas in the entire data space.\n14. Line 212. \"the higher the distance between two points, the more likely their convex combination will fall outside of the original manifolds\". Here is a counter example. Suppose we have two classes of points in $\\mathbb{R}^2$. Suppose their manifolds are two separated line segments on the horizontal axis. If we combine the rightest point from the left manifold and the leftest point from the right manifold, the mixed points will almost all fall into the middle area which is outside both the original manifolds. If we combine the leftest points from both manifold, the fact is, a much bigger proportion of mixed points would fall inside the left manifold. But, the distance between the points in the second case is much larger than that in the first case.\n15. Line 215. \"Using a Resnet18 trained ...\", is it trained using ERM or Mixup?\n16. How to choose $\\tau_{std}$ and $\\tau_{max}$?\n17. How does SK Mixup help improve calibration? The derivation of the algorithm only suggests that SK Mixup can help mitigate manifold mismatch, but is manifold mismatch the necessary reason, or even the real reason, of bad calibration?\n18. In the process of trading off between diversity and uncertainty, how would calibration and generalization behave? Will they behave like a up-side-down U curve such that there is a sweet spot?\n19. The page number of page 7, and the header \"under review\" line in page 8, there are some strange hyperlinks around the text.\n20. Theorem 3.1 (i). \"$\\lambda\\in]\\lambda_1,\\lambda_2[$\", it should be $[\\lambda_1,\\lambda_2]$ right?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "1. It's a novel idea to summarize several observed phenomenons or properties of Mixup into one unified terminology: manifold mismatch\n2. The correlationship between data points pair distance and the occurence of manifold mismatch is carefully derived.\n3. The proposed algorithm is meant to balance the diversity and the uncertainty of the virtual examples in Mixup, which is meaningful and helpful", "weaknesses": "1. Poor presentation\n2. Doesn't clarify the connection between manifold mismatch and calibration. \n3. The core claim that distance impacts manifold mismatch is too simple to be seen, and it's actually under-justified since there seems to be much more necessary conditions of manifold mismatch like data distribution, structure of learned features and manifolds, etc.. In fact, the idea of improving Mixup by forcing more mixtures between closer points is not novel, like k-mixup [1]\n4. The idea of dynamically adjust the mixing coeffecient is also not new, like AdaMixup [2].\n\n[1] https://arxiv.org/abs/2106.02933\n\n[2] https://arxiv.org/abs/1809.02499", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BAs9mK8Bp9", "reviewer": "Reviewer_ymX4", "summary": "This paper proposes to improve different mixup methods by adjusting the probability distribution to sample interpolation weights according to the distance between a given sample pair. Using the proposed method, sample pairs that are far from each other are less likely to be mixed. Empirical results on different tasks and data sets demonstrate the effectiveness of the proposed method.", "questions": "- Instead of using a similarity kernel introduced in this paper, a simple implementation is to divide sample pairs based on their distances (like the case in Figure 2) and use two different Beta distributions to sample the interpolation weights. How will this baseline perform compared to the proposed method? Such comparison can further verify the motivation and support the proposed method. \n- While the authors have mentioned that non-linear interpolation methods have several disadvantages (larger computational cost, limited application) in section 2.1, some empirical comparison on their performance for image data sets in Table 2-4 should still be necessary. In addition, the authors may also include some of these interpolation methods in Table 6 to verify that their computational cost are muchh larger compared to linear mixup methods. \n- Moreover, can the proposed method be combined with these non-linear interpolation methods as well? I suppose such combination will be straight-forward, as we only need to change the sampling distribution of interpolation weights for different sample pairs. Some discussion (possibly with some empirical results) will be welcome here. \n- I am a bit puzzled by the performance of different methods on ImageNet-A in Table 4. None of them has an accuracy higher than 3%, which seems worse than most methods reported in [1]. Is this due to some discrepancies in experimental setup (e.g., less training epochs)? If so, the authors may need to clarify that such discrepancies are not in favor of their proposed method. \n- Also, how are the baseline methods chosen in Table 6? From my perspective, I suppose we need to compare the time cost of SK Mixup against Mixup or some other baseline methods with rather good performance. Nevertheless, it seems either of the above cases applies, and some explanation may be needed here. \n\n## References\n[1] On Feature Normalization and Data Augmentation. CVPR 2021", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The proposed method is clearly introduced and easy to understand", "weaknesses": "- The motivation of proposed method can be strengthened\n- Some baseline methods seem missing, which can make the empirical comparison not supportive enough", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)