Tailoring Mixup to Data for Calibration

## Abstract

Abstract Among all data augmentation techniques proposed so far, linear interpolation of training samples, also called Mixup , has found to be effective for a large panel of applications.
Along with improved performance, Mixup is also a good technique for improving calibration and predictive uncertainty.
However, mixing data carelessly can lead to manifold intrusion, i.e., conflicts between the synthetic labels assigned and the true label distributions, which can deteriorate calibration.
In this work, we argue that the likelihood of manifold intrusion increases with the distance between data to mix.
To this end, we propose to dynamically change the underlying distributions of interpolation coefficients
depending on the similarity between samples to mix, and define a flexible framework to do so without losing in diversity. We provide extensive experiments for classification and regression tasks, showing that our proposed method improves performance
and calibration of models, while being much more efficient. The code for our work is available at https://github.com/qbouniot/sim_kernel_mixup .

## 1 Introduction

The *Vicinal Risk Minimization (VRM)* principle (Chapelle et al.,, ) improves on the well-known *Empirical Risk Minimization (ERM)* (Vapnik,, ) for training deep neural networks by drawing virtual samples from a vicinity around true training data. This data augmentation principle is known to improve the generalization ability of deep neural networks when the number of observed data is small compared to the task complexity. In practice, the method of choice to implement it relies on hand-crafted procedures to mimic natural perturbations (Yaeger et al.,, ; Ha and Bunke,, ; Simard et al.,, ).
However, one counterintuitive but effective and less application-specific approach for generating synthetic data is through interpolation, or mixing, of two or more training data.

The process of interpolating between data have been discussed multiple times before (Chawla et al.,, ; Wang et al.,, ; Inoue,, ; Tokozume et al.,, ), but *Mixup* (Zhang et al.,, ) represents the most popular implementation and continues to be studied in recent works (Pinto et al.,, ; ; Wang et al.,, ).
Ever since its introduction, it has been a widely studied data augmentation technique spanning applications to *image classification and generation* (Zhang et al.,, ), *semantic segmentation* (Franchi et al.,, ; Islam et al.,, ), *natural language processing* (Verma et al.,, ), *speech processing* (Meng et al.,, ), *time series and tabular regression* () or *geometric deep learning* (Kan et al.,, ), to that extent of being now an integral component of competitive state-of-the-art training settings (Wightman et al.,, ). The idea behind *Mixup* can be seen as an efficient approximation of *VRM*, by using a linear interpolation of data points selected from within the same batch to reduce computation overheads (Zhang et al.,, ).

Figure: (a) Vanilla Mixup
Refer to caption: x1.png

The process of Mixup as a data augmentation during training can be roughly separated in three phases: (i) selecting tuples (most often pairs) of points to mix together, (ii) sampling coefficients that will govern the interpolation to generate synthetic points, (iii) applying a specific interpolation procedure between the points weighted by the coefficients sampled.
Methods in the literature have mainly focused on the first and third phases, *i.e.*, the process of sampling points to mix through predefined criteria (Hwang et al.,, ; ; ; Palakkadavath et al.,, ; Teney et al.,, ) and on the interpolation itself, by applying sophisticated and application-specific functions (Yun et al.,, ; Franchi et al.,, ; Venkataramanan et al.,, ; Kan et al.,, ).
On the other hand, these *interpolation coefficients*, when they exist, are always sampled from the same distribution throughout training. Recent works have shown that mixing carelessly different points can result in incorrect labels and hurt generalization (Guo et al.,, ; ; Liu et al.,, ), while mixing similar points helps in diversity (Chawla et al.,, ; Dablain et al.,, ). Furthermore, several previous work have highlighted a trade-off between performance and calibration in Mixup (Thulasidasan et al.,, ; Pinto et al.,, ; Wang et al.,, ).

In this work, we aim to provide an *efficient* and *flexible* framework for taking similarity into account when interpolating points. Notably, we argue that similarity should influence the *interpolation coefficients* rather than the selection process of points to mix, to avoid losing in diversity. A high similarity should result in strong interpolation, while a low similarity should lead to almost no changes. Consequently, controlling the interpolation through the distance of the points to mix should improve calibration by reducing *manifold intrusion* and *label noise*.
Figure 1 illustrates the difference between taking into account similarity in the selection process and through the interpolation. Selecting only similar pairs to mix restricts the possible synthetic data generated (*Middle*). Governing interpolation coefficients through similarity of points avoid restricting possible directions of mixing (*Right*).
Furthermore, restricting the mixing operation by the distance between points forces the synthetic samples to be more likely in the vicinity of original samples, making the overall process closer to VRM.
Our contributions towards this goal are as follows:

- •
We show that taking into account similarity when mixing can alleviate label noise and improve calibration. Mixing only similar data leads to better calibration than including dissimilar ones.
- •
We present a flexible framework to dynamically change the distributions of interpolation coefficients. We apply a similarity kernel that takes into account the distance between points to select a parameter for the distribution tailored to each pair to mix. The underlying distribution of interpolation is warped to be stronger for similar points and weaker otherwise.
- •
We quantitatively ascertain the effectiveness of our Similarity Kernel Mixup with extensive experiments on multiple datasets, from image classification to regression tasks, and multiple deep neural network architectures. Our approach achieves a better trade-off between improving *accuracy* and *calibration*. Additionally, we highlight the efficiency of our method, obtaining competitive results with less computation per iteration, and reaching the best performance globally faster.

## 2 Related Work

### 2.1 Data augmentation based on mixing data

The idea of mixing two or more training data points to generate additional synthetic ones has been developed in various ways in the literature.
Non-linear interpolation .
Non-linear combinations are mainly studied for dealing with image data. Instead of a naive linear interpolation between two images, the augmentation process is done using more complex non-linear functions, such as cropping, patching and pasting images together (Takahashi et al.,, ; Summers and Dinneen,, ; Yun et al.,, ; Kim et al.,, ) or through subnetworks (Ramé et al.,, ; ; Venkataramanan et al.,, ). Not only are these non-linear operations focused on images, but they generally introduce a significant computational overhead compared to the simpler linear one (Zhu et al.,, ; Li et al.,, ). The recent *R-Mixup* (Kan et al.,, ), on the other hand, considers other Riemannian geodesics rather than the Euclidean straight line for graphs, but is also computationally expensive.
Linear interpolation .
Mixing samples online through linear interpolation represents the most efficient and general technique compared to the ones presented above (Zhang et al.,, ; Inoue,, ; Tokozume et al.,, ). Among these different approaches, combining data from the same batch also avoids additional samplings.
Several follow-up works extend Mixup from different perspectives. *Manifold Mixup* (Verma et al.,, ) interpolates data in the feature space, *k-Mixup* (Greenewald et al.,, ) extends the interpolation to use *k* points instead of a pair, *Remix* (Chou et al.,, ) separates the interpolation in the label space and the input space. Notably, *AdaMixUp* (Guo et al.,, ) learns to predict a mixing policy to apply to avoid manifold intrusion using two additional parallel models with intrusion losses. This leads to a more complex training and includes more parameters to learn. Furthermore, the predicted range is very narrow around 0.5, reducing diversity of the mixed samples, and the training includes non-mixed samples, making the approach similar to the recent *Regmixup* (Pinto et al.,, ). *Local Mixup* (Baena et al.,, ) is another approach interested on the manifold intrusion problem, by weighting the loss function with the distance between the points using K-nearest neighbors graphs. However, this results in completely discarding the interpolation with points that are far away, losing potential augmentation directions and, therefore, in diversity. Moreover, these two methods are focusing on improving generalization, while we are looking at the problem of manifold intrusion through the lens of improving *calibration*.
Selecting points . A recent family of methods applies an online linear combination on specifically selected pairs of examples (; ; Hwang et al.,, ; Palakkadavath et al.,, ; Teney et al.,, ), across classes () or across domains (; Palakkadavath et al.,, ; Tian et al.,, ). These methods achieve impressive results on distribution shift and out-of-distribution generalization (), but recent theoretical developments have shown that much of the improvements are linked to a resampling effect from the restrictions in the selection process, and are unrelated to the mixing operation (Teney et al.,, ). These selective criteria also induce high computational overhead. One related approach is *C-Mixup* (), that fits a Gaussian kernel on the labels distance between points in regression tasks. Then points to mix together are sampled from the full training set according to the learned Gaussian density. However, the Gaussian kernel is computed on all the data before training, which is difficult when there is a lot of data and no explicit distance between them.

### 2.2 Calibration in classification and regression

*Calibration* is a metric to quantify uncertainty, measuring the difference between a model’s confidence in its predictions and the actual probability of those predictions being correct. We refer the reader to Appendix C for a short review of calibration metrics

In classification .
Modern deep neural network for image classification are now known to be *overconfident* leading to *miscalibration* (Guo et al.,, ). One can rely on *temperature scaling* (Guo et al.,, ) to improve calibration *post-hoc*, or using different techniques during learning such as *ensembling* (Lakshminarayanan et al.,, ; Wen et al.,, ; Laurent et al.,, ), *explicit penalties* (Pereyra et al.,, ; Kumar et al.,, ; Moon et al.,, ; Cheng and Vasconcelos,, ), or *implicit* ones (Müller et al.,, ; Lin et al.,, ; Mukhoti et al.,, ; ), including notably through *mixup* (Thulasidasan et al.,, ; Pinto et al.,, ; Noh et al.,, ). One should note that calibration results should be compared after temperature scaling, since it can change the ordering of results (Ashukha et al.,, ).
Calibration-driven Mixup methods . The problem of the trade-off between performance and calibration with Mixup has been extensively studied in previous works (Thulasidasan et al.,, ; Zhang et al.,, ; Pinto et al.,, ; Wang et al.,, ). Notably, Wang et al., () observed that calibration using Mixup can be degraded after temperature scaling. Therefore, they proposed another improvement of mixup, *MIT* (Wang et al.,, ), by generating two sets of mixed samples and then deriving their correct label. *RegMixup* (Pinto et al.,, ) considers Mixup as a regularization term, using stronger interpolation on every pair. Finally, *RankMixup* (Noh et al.,, ) uses the interpolation coefficients from multiple mixed pairs as an additional supervisory signal for ranking of confidence. We propose a more efficient method to achieve a good trade-off between accuracy and calibration with Mixup.
In regression .
The problem of calibration in deep learning has also been studied for regression tasks (Kuleshov et al.,, ; Song et al.,, ; Laves et al.,, ; Levi et al.,, ), where it is more complex as we lack a simple measure of predictive confidence. In this case, regression models are usually evaluated under the variational inference framework with Monte Carlo (MC) Dropout (Gal and Ghahramani,, ) to quantify confidence.
In our work, we use a *similarity kernel* to mix more strongly similar data and avoid mixing less similar ones, leading to preserving label quality and confidence of the network.
As opposed to all other methods discussed above, we also show the flexibility of our approach by its effectiveness on both classification and regression tasks. We detail our framework, the similarity used and the intuition behind below.

## 3 Similarity Kernel Mixup

Figure: (a) ERM
Refer to caption: x4.png

Preliminary notations and background .
First, we define the notations and elaborate on the learning conditions that will be considered throughout the paper.
Let $\mathcal{D}=\{({\mathbf{x}}_{i},y_{i})\}_{i=1}^{N}=({\mathbf{X}},{\mathbf{y}})
\in{\mathbb{X}}^{N}\times{\mathbb{Y}}^{N}\subset\mathbb{R}^{d\times N}\times
\mathbb{R}^{N}$ be the training dataset. We want to learn a *model* $f_{\theta}$ parameterized by $\theta\in\Theta\subset\mathbb{R}^{p}$, that predicts $\hat{y}:=f_{\theta}({\mathbf{x}})$ for any ${\mathbf{x}}\in{\mathbb{X}}$. For classification tasks, we have ${\mathbb{Y}}\subset\mathbb{R}^{c}$, and we further assume that the model $f_{\theta}$ can be separated into an encoder part $h_{\varphi}$ and classification weights ${\mathbf{w}}\in\mathbb{R}^{c}$, such that $\forall{\mathbf{x}}\in{\mathbb{X}},f_{\theta}({\mathbf{x}})={\mathbf{w}}^{\top
}h_{\varphi}({\mathbf{x}})$.
To learn our model, we optimize the *weights* of the model $\theta$ in a stochastic manner, by repeating the minimization process of the empirical risk computed on *batch* of data $\mathcal{B}_{t}=\{({\mathbf{x}}_{i},y_{i})\}_{i=1}^{n}$ sampled from the training set, for $t\in\{1,\dots,T\}$ iterations.

With *Mixup* (Zhang et al.,, ), at each iteration $t$, the empirical risk is computed on augmented batch of data $\tilde{\mathcal{B}}_{t}=\{(\tilde{{\mathbf{x}}}_{i},\tilde{y}_{i})\}_{i=1}^{n}$, such that $\tilde{{\mathbf{x}}}_{i}:=\lambda_{t}{\mathbf{x}}_{i}+(1-\lambda_{t}){\mathbf{
x}}_{\sigma_{t}(i)}$ and $\tilde{y}_{i}:=\lambda_{t}y_{i}+(1-\lambda_{t})y_{\sigma_{t}(i)}$, with $\lambda_{t}\sim\texttt{Beta}(\alpha,\alpha)$ and $\sigma_{t}\in\mathfrak{S}_{n}$ a random permutation of $n$ elements sampled uniformly. Thus, each input is mixed with another input randomly selected from the same batch, and $\lambda_{t}$ represents the strength of the interpolation between them. Besides simplicity, mixing elements within the batch significantly reduces both memory and computation costs.

In the following parts, we introduce a more general extension of this framework using warping functions, that spans different variants of *Mixup*, while preserving its efficiency. First, we discuss and illustrate the *manifold intrusion problem* with *Mixup* and its impact.

### 3.1 Manifold Intrusion

The assumption underlying our work is that there exists a trade-off in data augmentation procedures between adding diversity and introducing uncertainty. In *mixup*, uncertainty can be introduced through manifold intrusion (Guo et al.,, ), *i.e*, conflicts between the synthetic labels of the mixed-up examples and the labels of original training data. More specifically, we argue that in *mixup* the likelihood of conflict in the synthetic label increases with the distance between the two points. As data live in manifolds of the representation space, the linear combination of two points far from each other can lie in a different manifold than the linear combination of the labels. The further away the points are, the more manifolds can exist in between.
We illustrate this problem on the *Moons* and *Circles* toy problems in Figure 2.
We observe that a model with Mixup *(middle)* can lead to worst confidence and performance compared to standard ERM *(left)*, due to manifold intrusion. Since the moons are intertwined and non-convex, linear interpolations from the same moon can lie in the other *(top)*. Likewise, interpolations of data from the external circle can fall into the inner one *(bottom)*. Using our Similarity Kernel Mixup approach (*right*) described in this paper, we recover the accuracy and achieve more meaningful confidence scores.

Then, we conduct an empirical analysis on real datasets that compares accuracy and calibration metrics using the classic *ERM*, the original *Mixup*, and *a modified version of Mixup* that selects pairs to mix according to a given quantile of the overall pairwise distances within the batch. To have equivalent proportions of possible data to mix with (*same diversity*), we analyze results when mixing only pairs of points with distances lower than a given quantile $q$, and higher than the quantile $q^{\prime}=1-q$.
More specifically, we show in Figure 3 the results for $q=0.75$ of the overall pairwise distances within the batch, using a Resnet34 (He et al.,, ) on CIFAR10 and CIFAR100 (Krizhevsky et al.,, ) datasets.
Detailed values for this experiment with different quantiles $q$ can be found in Appendix D, and implementation details in Section 4. First, we observe that Mixup improves upon ERM’s accuracy, but can degrade calibration depending on the dataset, which is consistent with findings from Wang et al., (). Then, when selecting pairs according to distance, a sufficiently high proportion of data to mix is necessary to preserve accuracy ($q>0.5$). Finally, mixing data with lower distances achieves a better calibration as opposed to mixing data with higher distances.

These results show that there is a trade-off between adding diversity by increasing the proportion of elements to mix, and uncertainty by mixing elements far from each other. Furthermore, it shows that we cannot restrict pairs to mix by selecting data solely based on distance, as it can degrade performance by reducing diversity of synthetic samples.
To better control this trade-off with Mixup, we propose to tailor *interpolation coefficients* to the training data, from the distance between the points to mix. We detail our framework in the next part.

Figure: Figure 3: Performance (Accuracy in %, *higher is better*, bottom) and calibration (ECE, *lower is better*, top) *after Temperature Scaling (TS)* Guo et al., () comparison with Resnet34 on CIFAR10 (left) and CIFAR100 (right) datasets. We compare results when mixing only elements with distance Higher (in *green*) than the 0.25 quantile, and Lower (in *orange*) than the 0.75 quantile of all pairwise distances within each batch.
Refer to caption: x10.png

### 3.2 Warped Mixup

To dynamically change the interpolation depending on the similarity between points, we rely on *warping functions* $\omega_{\tau}$, to *warp* interpolation coefficients $\lambda_{t}$ at every iteration $t$ depending on the parameter $\tau$. These functions $\omega_{\tau}$ are *bijective transformations* from $[0,1]$ to $[0,1]$ defined as such:

$$ $\omega_{\tau}(\lambda_{t})=I^{-1}_{\lambda_{t}}(\tau,\tau)\quad\text{with} \quad I_{\lambda_{t}}=\int_{0}^{\lambda_{t}}\frac{u^{\tau-1}(1-u)^{\tau-1}}{B( \tau,\tau)}du,$ (1) $$

where $I_{\lambda_{t}}$ is the *regularized incomplete beta function*, which is the *cumulative distribution function (CDF)* of the Beta distribution, $B(\tau,\tau)$ is a normalization constant and $\tau\in\mathbb{R}^{*}_{+}$ is the *warping parameter* that governs the strength and direction of the warping. Similar warping has been used in the *Bayesian Optimization* literature (Snoek et al.,, ), however many other suitable bijection with sigmoidal shape could be considered in our case.
Our motivation behind such $\omega_{\tau}$ is to preserve the same type of distribution after warping, *i.e.*, Beta distributions with symmetry around $0.5$, through *Inverse Transform Sampling* (see Appendix F for the proof):

###### Proposition 3.1 .

Let $\lambda\sim\mathcal{U}([0,1])$. Then $\omega_{\tau}(\lambda)\sim\texttt{Beta}(\tau,\tau)$ for any $\tau>0$.

One should note that according to Proposition 3.1, the initial interpolation parameter $\lambda$ should follow a *uniform distribution* for $\omega_{\tau}(\lambda)$ to follow a $\texttt{Beta}(\tau,\tau)$ distribution. Thus, in the remaining of the paper, we always draw $\lambda$ according to a uniform distribution (or equivalently, a $\texttt{Beta}(1,1)$), which has the additional benefit of *removing $\alpha$ as a hyperparameter*.
Figure 4(a) illustrates the shape of $\omega_{\tau}$ and their behavior with respect to $\tau$. These functions have a symmetric behavior around $\tau=1$ (*in blue*), for which warped outputs remain unchanged. When $\tau>1$ (*in red and purple*) they are pulled towards the center ($0.5$), and when $\tau<1$ (*in orange and green*), they are pushed towards the extremes (0 0 and $1$).
We thus extend the Mixup framework with warping:

$$ $\lambda_{t}\sim\texttt{Beta}(1,1),\qquad\begin{aligned} \tilde{{\mathbf{x}}}_{ i}&:=\omega_{\tau}(\lambda_{t}){\mathbf{x}}_{i}+(1-\omega_{\tau}(\lambda_{t})) {\mathbf{x}}_{\sigma_{t}(i)}\\ \tilde{y}_{i}&:=\omega_{\tau}(\lambda_{t})y_{i}+(1-\omega_{\tau}(\lambda_{t})) y_{\sigma_{t}(i)}.\end{aligned}$ (2) $$

In addition to providing a general framework, using warping functions proves to be more computationally efficient, which we discuss in details in Appendix G. In the following part, we present our method to select the right $\tau$ depending on the data to mix, to dynamically change the distribution of the interpolation coefficients.

###### Proposition 3.1 .

### 3.3 Similarity Kernel

Figure: (a)
Refer to caption: x11.png

Recall that our main goal is to apply stronger interpolation between similar points, and reduce interpolation otherwise. Since the behavior of Beta distributions and $\omega_{\tau}$ are logarithmic with respect to $\tau$, therefore, the parameter $\tau$ should be exponentially correlated with the distance, with a symmetric behavior around 1.
To this end, we define a class of *similarity kernels*, based on a normalized and centered Gaussian kernel, that outputs the correct warping parameter for the given pair of points. Given a batch of data ${\mathbf{x}}=\{{\mathbf{x}}_{i}\}_{i=1}^{n}\in\mathbb{R}^{d\times n}$, the index of the first element in the mix $i\in\{1,\dots,n\}$, along with the permutation $\sigma\in\mathfrak{S}_{n}$ to obtain the index of the second element, we compute the following *similarity kernel*:

$$ $\displaystyle\tau({\mathbf{x}},i,\sigma;\tau_{\text{max}},\tau_{\text{std}})$ $\displaystyle=\tau_{\text{max}}\exp\left(-\frac{\bar{d}_{n}({\mathbf{x}}_{i},{ \mathbf{x}}_{\sigma(i)})-1}{2\tau_{\text{std}}^{2}}\right),$ (3) $\displaystyle\text{with }\quad\bar{d}_{n}({\mathbf{x}}_{i},{\mathbf{x}}_{ \sigma(i)})$ $\displaystyle=\frac{\|{\mathbf{x}}_{i}-{\mathbf{x}}_{\sigma(i)}\|_{2}^{2}}{ \frac{1}{n}\sum_{j=1}^{n}\|{\mathbf{x}}_{j}-{\mathbf{x}}_{\sigma(j)}\|_{2}^{2}},$ (4) $$

where $\bar{d}_{n}$ is the squared $L_{2}$ distance rescaled by the mean distance over the batch, and $\tau_{\text{max}},\tau_{\text{std}}$ are respectively the *amplitude* and *standard deviation (std)* of the Gaussian, which are hyperparameters of the similarity kernel. The amplitude $\tau_{\text{max}}$ governs the strength of the interpolation in average, and $\tau_{\text{std}}$ the extent of mixing. In practice, we found that these two parameters could be tuned separately, and that choosing a good $\tau_{\text{std}}$ is more important than $\tau_{\text{max}}$.
The framework presented allows to measure similarity between points in any space that can represent them. More specifically, for classification tasks, we use the $L_{2}$ distance between embeddings, *i.e.*, $\bar{d}_{n}(h_{\varphi}({\mathbf{x}}_{i}),h_{\varphi}({\mathbf{x}}_{\sigma(i)})$, while for regression tasks, we use the distance between labels, *i.e.*, $\bar{d}_{n}(y_{i},y_{\sigma(i)})$, following .

Our motivation behind this kernel is to have $\tau>1$ when the two points to mix are similar, *i.e.*, the distance is lower than average, to increase the mixing effect, and $\tau<1$ otherwise, to reduce the mixing.
Figure 4(b) illustrates the evolution of the density of *warped* interpolation coefficients $\omega_{\tau}(\lambda)$, depending on the distance between the points to mix. Close distances *(left part of the heatmap)* induce strong interpolations, while far distances *(right part of the heatmap)* reduce interpolation. Using this similarity kernel to find the correct $\tau$ to parameterize the *Beta distribution* defines our full *Similarity Kernel (SK) Mixup* framework. A detailed algorithm of the training procedure can be found in Appendix E, and an in-depth discussion about warping and similarity measures in Appendix J.

## 4 Experiments

### 4.1 Protocols

Image Classification . We follow experimental settings from previous works (; Wang et al.,, ; Noh et al.,, ) and evaluate our approach on CIFAR-10, CIFAR-100 (Krizhevsky et al.,, ), Tiny-Imagenet (Deng et al.,, ) and Imagenet (Russakovsky et al.,, ) datasets, using Resnet34, Resnet50 and Resnet101 architectures (He et al.,, ). We evaluate calibration using ECE and AECE (Naeini et al.,, ; Guo et al.,, ), negative log likelihood (NLL) (Hastie et al.,, ) and Brier score (Brier,, ), after finding the optimal temperature on the validation set through Temperature Scaling (Guo et al.,, ). Results are reproduced and averaged over 4 different random runs, and we report standard deviation between the runs.
Regression . Here again, we follow settings of previous work on regression (). We evaluate performance on Airfoil (Kooperberg,, ), Exchange-Rate and Electricity (Lai et al.,, ) datasets using Mean Averaged Percentage Error (MAPE), along with Uncertainty Calibration Error (UCE) (Laves et al.,, ) and Expected Normalized Calibration Error (ENCE) (Levi et al.,, ) for calibration. Results are reproduced and averaged over 10 different random runs. We also report standard deviation between the runs.

A presentation of the different calibration metrics used can be found in Appendix C, along with a detailed description of implementation settings and hyperparameters in Appendix H.

**Table 1: Comparison of Performance (Accuracy in %) and calibration (ECE, Brier, NLL) *after Temperature Scaling* with Resnet34 on CIFAR10 and CIFAR100. Best in bold, second best underlined.**
| Methods | $\alpha$ | $\tau_{\text{std}}$ | CIFAR10 | CIFAR100 |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Accuracy ($\uparrow$) | ECE ($\downarrow$) | Brier ($\downarrow$) | NLL ($\downarrow$) | Accuracy ($\uparrow$) | ECE ($\downarrow$) | Brier ($\downarrow$) | NLL ($\downarrow$) |  |  |  |
| ERM | – | – | 94.69 $\pm$ 0.27 | 0.82 $\pm$ 0.11 | 8.07 $\pm$ 0.31 | 17.50 $\pm$ 0.61 | 73.47 $\pm$ 1.59 | 2.54 $\pm$ 0.15 | 36.47 $\pm$ 2.05 | 100.82 $\pm$ 6.93 |
| Mixup | 1 | – | 95.97 $\pm$ 0.27 | 1.36 $\pm$ 0.13 | 6.53 $\pm$ 0.36 | 16.35 $\pm$ 0.72 | 78.11 $\pm$ 0.57 | 2.49 $\pm$ 0.19 | 31.06 $\pm$ 0.69 | 87.94 $\pm$ 1.98 |
| 0.5 | – | 95.71 $\pm$ 0.26 | 1.33 $\pm$ 0.08 | 7.03 $\pm$ 0.46 | 17.47 $\pm$ 1.18 | 77.14 $\pm$ 0.67 | 2.7 $\pm$ 0.36 | 32.01 $\pm$ 0.93 | 91.22 $\pm$ 3.05 |  |
| 0.1 | – | 95.37 $\pm$ 0.22 | 1.13 $\pm$ 0.11 | 7.37 $\pm$ 0.36 | 17.43 $\pm$ 0.79 | 76.01 $\pm$ 0.62 | 2.54 $\pm$ 0.24 | 33.41 $\pm$ 0.57 | 93.96 $\pm$ 1.76 |  |
| Mixup IO | 1 | – | 95.16 $\pm$ 0.22 | 0.6 $\pm$ 0.11 | 7.3 $\pm$ 0.33 | 15.56 $\pm$ 0.67 | 74.44 $\pm$ 0.49 | 2.02 $\pm$ 0.14 | 35.25 $\pm$ 0.43 | 96.5 $\pm$ 1.62 |
| 0.5 | – | 95.31 $\pm$ 0.17 | 0.58 $\pm$ 0.06 | 7.12 $\pm$ 0.21 | 15.09 $\pm$ 0.45 | 74.45 $\pm$ 0.6 | 1.94 $\pm$ 0.09 | 35.2 $\pm$ 0.58 | 96.75 $\pm$ 1.89 |  |
| 0.1 | – | 95.12 $\pm$ 0.21 | 0.7 $\pm$ 0.09 | 7.38 $\pm$ 0.27 | 15.76 $\pm$ 0.55 | 74.21 $\pm$ 0.46 | 2.39 $\pm$ 0.11 | 35.38 $\pm$ 0.48 | 98.24 $\pm$ 1.81 |  |
| SK Mixup *(Ours)* | 1 | 0.2 | 96.29 $\pm$ 0.07 | 1.35 $\pm$ 0.14 | 6.37 $\pm$ 0.11 | 15.97 $\pm$ 0.22 | 78.13 $\pm$ 0.52 | 1.31 $\pm$ 0.23 | 31.18 $\pm$ 0.72 | 85.38 $\pm$ 1.92 |
| 0.4 | 96.42 $\pm$ 0.24 | 0.53 $\pm$ 0.06 | 6.02 $\pm$ 0.41 | 15.28 $\pm$ 0.92 | 78.86 $\pm$ 0.83 | 1.42 $\pm$ 0.19 | 30.1 $\pm$ 1.17 | 84.22 $\pm$ 3.35 |  |  |
| 0.6 | 96.36 $\pm$ 0.09 | 0.52 $\pm$ 0.08 | 6.12 $\pm$ 0.09 | 15.7 $\pm$ 0.29 | 78.63 $\pm$ 0.27 | 1.74 $\pm$ 0.30 | 30.41 $\pm$ 0.30 | 84.97 $\pm$ 1.12 |  |  |
| 0.8 | 96.00 $\pm$ 0.41 | 0.56 $\pm$ 0.05 | 6.63 $\pm$ 0.58 | 16.64 $\pm$ 1.27 | 79.12 $\pm$ 0.52 | 1.66 $\pm$ 0.16 | 29.75 $\pm$ 0.57 | 82.82 $\pm$ 1.62 |  |  |
| 1.0 | 96.25 $\pm$ 0.07 | 0.55 $\pm$ 0.12 | 6.31 $\pm$ 0.18 | 16.14 $\pm$ 0.55 | 78.51 $\pm$ 0.94 | 1.49 $\pm$ 0.03 | 30.46 $\pm$ 1.04 | 85.06 $\pm$ 3.18 |  |  |

**Table 2: Comparison of performance (Accuracy in %) and calibration (ECE, AECE) *after Temperature Scaling*, with Resnet101 on CIFAR10, CIFAR100 and Tiny-Imagenet datasets. $\dagger$: Results reported from Noh et al., (). Best in bold, second best underlined, *for reported results and ours separately*.**
| Methods | CIFAR10 | CIFAR100 | Tiny-Imagenet |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Acc. ($\uparrow$) | ECE ($\downarrow$) | AECE ($\downarrow$) | Acc. ($\uparrow$) | ECE ($\downarrow$) | AECE ($\downarrow$) | Acc. ($\uparrow$) | ECE ($\downarrow$) | AECE ($\downarrow$) |  |
| ERM^† | 94.46 | 0.92 | 0.85 | 77.48 | 2.63 | 2.66 | 66.04 | 3.50 | 3.52 |
| MMCE^† | 94.99 | 1.15 | 12.9 | 77.82 | 3.06 | 2.80 | 66.44 | 3.40 | 3.38 |
| ECP^† | 93.97 | 1.72 | 1.70 | 76.81 | 2.92 | 3.04 | 66.20 | 2.72 | 2.70 |
| LS^† | 94.18 | 1.51 | 3.10 | 76.91 | 4.38 | 4.55 | 65.52 | 2.51 | 2.72 |
| FL^† | 93.59 | 1.12 | 1.37 | 76.12 | 2.58 | 2.51 | 64.02 | 2.18 | 2.09 |
| Mixup^† | 95.50 | 1.18 | 2.33 | 78.74 | 3.69 | 3.65 | 66.41 | 1.97 | 1.95 |
| FLSD^† | 93.26 | 0.93 | 0.94 | 76.61 | 2.04 | 1.78 | 64.02 | 1.85 | 1.81 |
| CRL^† | 95.04 | 1.12 | 2.03 | 77.60 | 3.32 | 3.31 | 65.87 | 1.60 | 1.52 |
| CPC^† | 95.36 | 1.52 | 2.37 | 77.50 | 2.96 | 3.23 | 66.44 | 3.93 | 3.74 |
| MbLS^† | 95.13 | 1.38 | 3.25 | 77.45 | 5.49 | 6.52 | 65.81 | 1.62 | 1.68 |
| RegMixup^† | 95.03 | 0.92 | 0.94 | 76.93 | 1.36 | 1.92 | 63.26 | 1.86 | 1.68 |
| RankMixup^† | 94.25 | 0.65 | 0.56 | 76.46 | 1.10 | 1.40 | 64.89 | 1.57 | 1.94 |
| ERM | 93.97 $\pm$ 0.03 | 0.58 $\pm$ 0.01 | 0.63 $\pm$ 0.01 | 77.99 $\pm$ 0.31 | 2.16 $\pm$ 0.25 | 2.10 $\pm$ 0.27 | 65.7 $\pm$ 0.07 | 1.62 $\pm$ 0.19 | 1.57 $\pm$ 0.2 |
| Mixup | 94.78 $\pm$ 0.13 | 1.47 $\pm$ 0.01 | 2.59 $\pm$ 0.02 | 77.50 $\pm$ 0.18 | 2.58 $\pm$ 0.01 | 2.67 $\pm$ 0.20 | 67.80 $\pm$ 0.54 | 1.93 $\pm$ 0.02 | 1.68 $\pm$ 0.01 |
| Manifold Mixup | 94.75 $\pm$ 0.03 | 1.57 $\pm$ 0.05 | 2.61 $\pm$ 0.03 | 77.76 $\pm$ 1.35 | 3.07 $\pm$ 0.22 | 3.16 $\pm$ 0.17 | 68.57 $\pm$ 0.04 | 2.11 $\pm$ 0.01 | 2.06 $\pm$ 0.01 |
| RankMixup | 94.42 $\pm$ 0.17 | 0.70 $\pm$ 0.11 | 0.63 $\pm$ 0.14 | 77.27 $\pm$ 0.08 | 2.12 $\pm$ 0.20 | 2.04 $\pm$ 0.19 | 65.40 $\pm$ 0.01 | 1.48 $\pm$ 0.02 | 1.44 $\pm$ 0.06 |
| MIT-A | 95.27 $\pm$ 0.01 | 0.93 $\pm$ 0.01 | 1.59 $\pm$ 0.13 | 77.23 $\pm$ 0.56 | 2.39 $\pm$ 0.03 | 2.32 $\pm$ 0.05 | 68.03 $\pm$ 0.09 | 1.52 $\pm$ 0.15 | 1.57 $\pm$ 0.05 |
| SK Mixup *(Ours)* | 95.04 $\pm$ 0.07 | 0.47 $\pm$ 0.01 | 0.98 $\pm$ 0.05 | 78.20 $\pm$ 0.46 | 1.11 $\pm$ 0.06 | 1.38 $\pm$ 0.11 | 67.60 $\pm$ 0.01 | 1.06 $\pm$ 0.01 | 1.19 $\pm$ 0.02 |

**Table 3: Comparison of performance (Accuracy in %) and calibration (ECE, AECE), with Resnet50 on Imagenet dataset. Best in bold, second best underlined.**
| Methods | Acc. ($\uparrow$) | ECE ($\downarrow$) | AECE ($\downarrow$) |
| --- | --- | --- | --- |
| ERM | 76.06 $\pm$ 0.35 | 4.12 $\pm$ 0.22 | 4.05 $\pm$ 0.20 |
| Mixup | 76.59 $\pm$ 0.5 | 1.8 $\pm$ 0.21 | 1.77 $\pm$ 0.19 |
| Manifold Mixup | 76.61 $\pm$ 0.27 | 1.77 $\pm$ 0.15 | 1.77 $\pm$ 0.18 |
| RankMixup | 76.33 $\pm$ 0.06 | 2.07 $\pm$ 0.16 | 2.1 $\pm$ 0.14 |
| MIT-A | 76.7 $\pm$ 0.29 | 1.8 $\pm$ 0.08 | 1.74 $\pm$ 0.1 |
| Similarity Kernel Mixup *(Ours)* | 76.2 $\pm$ 0.26 | 1.46 $\pm$ 0.09 | 1.4 $\pm$ 0.07 |

**Table 4: Performance (RMSE, MAPE) and calibration (UCE, ENCE) comparison on regression tasks. Best in bold, second best underlined.**
| Methods | Airfoil | Exchange Rate | Electricity |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| MAPE ($\downarrow$) | UCE ($\downarrow$) | ENCE ($\downarrow$) | MAPE ($\downarrow$) | UCE ($\downarrow$) | ENCE ($\downarrow$) | MAPE ($\downarrow$) | UCE ($\downarrow$) | ENCE ($\downarrow$) |  |
| ERM | 1.720 $\pm$ 0.219 | 107.6 $\pm$ 19.2 | 2.10% $\pm$ 0.78 | 1.924 $\pm$ 0.287 | 0.82% $\pm$ 0.28 | 3.64% $\pm$ 0.74 | 15.263 $\pm$ 0.383 | 0.76% $\pm$ 0.08 | 22.07% $\pm$ 1.27 |
| Mixup | 2.003 $\pm$ 0.126 | 147.1 $\pm$ 34.0 | 2.12% $\pm$ 0.63 | 1.926 $\pm$ 0.284 | 0.74% $\pm$ 0.22 | 3.52% $\pm$ 0.59 | 14.944 $\pm$ 0.386 | 0.62% $\pm$ 0.07 | 24.28% $\pm$ 2.47 |
| Manifold Mixup | 1.964 $\pm$ 0.111 | 126.0 $\pm$ 15.8 | 2.06% $\pm$ 0.64 | 2.006 $\pm$ 0.346 | 0.86% $\pm$ 0.29 | 3.82% $\pm$ 0.85 | 14.872 $\pm$ 0.409 | 0.69% $\pm$ 0.05 | 24.53% $\pm$ 1.44 |
| RegMixup | 1.725 $\pm$ 0.092 | 105.0 $\pm$ 23.1 | 1.92% $\pm$ 0.73 | 1.918 $\pm$ 0.290 | 0.80% $\pm$ 0.24 | 3.66% $\pm$ 0.73 | 14.790 $\pm$ 0.395 | 0.66% $\pm$ 0.12 | 23.68% $\pm$ 2.53 |
| C-Mixup | 1.706 $\pm$ 0.104 | 111.2 $\pm$ 32.6 | 1.90% $\pm$ 0.75 | 1.893 $\pm$ 0.222 | 0.78% $\pm$ 0.20 | 3.60% $\pm$ 0.64 | 15.085 $\pm$ 0.533 | 0.69% $\pm$ 0.11 | 23.54% $\pm$ 1.90 |
| SK Mixup (Ours) | 1.609 $\pm$ 0.137 | 93.8 $\pm$ 25.5 | 1.85% $\pm$ 0.84 | 1.814 $\pm$ 0.241 | 0.76% $\pm$ 0.21 | 3.40% $\pm$ 0.56 | 14.649 $\pm$ 0.191 | 0.61% $\pm$ 0.05 | 23.93% $\pm$ 2.05 |

**Table 5: Efficiency comparison between Mixup methods. We report the number of batch of data in memory at each iteration, the best epoch measured on validation data, time per epoch (in seconds) and total training time to reach the best epoch (in seconds).**
| Method | Batch in memory | CIFAR10 | CIFAR100 | Tiny-Imagenet |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Best Epoch | Time | Total Time | Best Epoch | Time | Total Time | Best Epoch | Time | Total Time |  |  |
| RankMixup | 4 | 147 | 78 | 11485 | 177 | 78 | 13806 | 80 | 540 | 43380 |
| MIT-A | 2 | 189 | 46 | 8694 | 167 | 46 | 7659 | 82 | 305 | 24908 |
| SK Mixup | 1 | 183 | 28 | 5138 | 160 | 28 | 4480 | 69 | 189 | 13104 |

### 4.2 Classification

Throughout our experiments in classification, we use by default $\tau_{\text{max}}=1$ (unless stated otherwise), since we found that *the impact on calibration was mainly controlled by $\tau_{\text{std}}$*, as discussed below.

Effect of $\tau_{\text{std}}$. We present in Table 1 the performance and calibration results on CIFAR10 and CIFAR100 with a Resnet34, to study the effect of varying $\tau_{\text{std}}$. We compare the results with *Mixup* (Zhang et al.,, ) and *Mixup-IO* (Wang et al.,, ), which mixes between Inputs Only while keeping one-hot labels, for varying values of $\alpha$. These two versions of Mixup show the trade-off between the effect of confidence penalty (in Mixup) that can hurt calibration on the one hand, and trivial confidence promotion (in Mixup-IO) that can hurt accuracy on the other hand. By varying only $\tau_{\text{std}}$, therefore changing the *extent* of mixing, we can achieve a better and finer trade-off between these two effects, and *improving both calibration and accuracy*.
Furthermore, it shows that our Similarity Kernel Mixup method obtains better results than simply changing $\alpha$ in Mixup and Mixup-IO.
For other experiments, we first performed cross-validation to select $\tau_{\text{std}}$. In general, we found that $\tau_{\text{std}}=0.25$ worked for all datasets. We refer the reader to Appendix I for more details on cross-validation.

Comparison with state of the art . In Table 2, we present an extensive comparison of results on CIFAR10, CIFAR100 and Tiny-ImageNet with a Resnet101, and on Imagenet for Resnet50 in Table 3. We compare performance and calibration results reported *in the same settings* for various approaches from Noh et al., (), including *implicit methods* such as LS (Müller et al.,, ), FL (Lin et al.,, ), FLSD (Mukhoti et al.,, ), MbLS (), Mixup (Thulasidasan et al.,, ) and RegMixup (Pinto et al.,, ), and *explicit methods*, such as ECP (Pereyra et al.,, ), MMCE (Kumar et al.,, ), CRL (Moon et al.,, ), CPC (Cheng and Vasconcelos,, ). We have also reproduced results of Mixup, Manifold Mixup (Verma et al.,, ), RankMixup (Noh et al.,, ) and MIT-A (Wang et al.,, ) in the same settings on multiple random seeds, using official codes and hyperparameters provided by the authors. We can see from these results that we achieve competitive performance for accuracy and improves calibration compared to other *Calibration-driven Mixup methods*. Notably, our approach always improves accuracy over ERM.

### 4.3 Regression

To demonstrate the flexibility of our framework regarding different tasks, we provide experiments on regression for tabular data and time series.
Regression tasks have the advantage of having an obvious meaningful distance between points, which is distance between labels. Therefore, following , we directly measure the similarity between two points by the *distance between their labels*, *i.e.*, $\bar{d}_{n}(y_{i},y_{\sigma(i)})$. This avoids the computation of distance on embeddings and makes our method *as fast as the original Mixup*.
In Table 4, we compare our Kernel Warping Mixup with Mixup (Zhang et al.,, ), Manifold Mixup (Verma et al.,, ), RegMixup (Pinto et al.,, ), and C-Mixup (). We can see that our approach achieves competitive results with state-of-the-art C-Mixup, in both performance (MAPE) and calibration metrics.

### 4.4 Efficiency Comparison

Our method achieves competitive results while being *much more efficient* than other state-of-the-art approaches. Indeed, as can be seen in Table 5, our Similarity Kernel Mixup is about $1.5\times$ *faster* than MIT-A, about $3\times$ faster than RankMixup, and about $4\times$ faster than C-Mixup.

Additionally, we use *a single batch of data* for each iteration, while MIT-A requires training on *twice* the amount of data per batch, and *four times* the amount of data per batch for RankMixup. This adds significant memory constraints which limits the maximum batch size possible in practice. Unlike C-Mixup, our approach does not rely on *sampling rates* calculated before training, which add a lot of computational overhead and are difficult to obtain for large datasets.

## 5 Conclusion

Motivated by calibration in both classification and regression tasks, we present *Similarity Kernel Mixup*, a flexible framework for linearly interpolating data during training, based on a *similarity kernel*.
The coefficients governing the interpolation are warped to change their underlying distribution depending on the similarity between the points to mix, such that similar data are mixed more strongly than less similar ones, preserving calibration by avoiding *manifold intrusion and label noise*.
This provides a *more efficient* data augmentation approach than *Calibration-driven Mixup methods*, both in terms of time and memory, with a better trade-off between performance and calibration improvement.
We show through extensive experiments the effectiveness of the approach in classification as well as in regression.
It is also worth noting that the proposed framework could be extended to other Mixup variants such
as CutMix (Yun et al.,, ) or Manifold Mixup (Verma et al.,, ).
Future works include applications to more complex tasks such as semantic segmentation or depth estimation.

## Appendix A Broader Impact

This paper presents work whose goal is to improve the calibration of machine learning models. Calibration is the process of improving the reliability of the confidence score associated with predictions. This is an important step towards the development of trustworthy models. Having better calibrated models can also help in the detection of biases, overfitting or out-of-distribution data, which can prevent models from being misused in practice.

## Appendix B Limitations

The majority of theoretical analysis around mixup have been focused on generalization, while we are interested on calibration. Analyzing the impact on calibration is very difficult, as we also need to take into account the effect of Temperature Scaling (TS) Guo et al., () on a validation set. Indeed, TS can change ordering of results Ashukha et al., (), and, more importantly, Wang et al., () observed that calibration using Mixup can be worse than ERM after TS. This is an observation that we also confirm in our experiments (e.g. in Table 1). Thus, an analysis including temperature scaling would be required first for vanilla Mixup before analyzing our proposed method. This would be a significant contribution on its own, and our future works include to explore this direction.

Our approach also introduces two additional hyperparameters. However, since in our case we always draw initial parameters $\lambda$ from $\texttt{Beta}(1,1)$, $\alpha$ is not a hyperparameter anymore. Then, $\tau_{max}$ and $\tau_{std}$ can be tuned separately, and, as mentioned in Section 4.2, we found that impact on calibration was mainly controlled by $\tau_{std}$. Furthermore, one should note that we always used $\tau_{std}=0.25$ in image classification experiments, showing our approach is not sensitive to hyperparameter choice between datasets. We discuss selection of hyperparameters with cross-validation in Appendix I.

## Appendix C Introduction to Calibration Metrics

As discussed in Section 2.2, calibration measures the difference between predictive confidence and actual probability. More formally, with $\hat{y}$ and $y\in{\mathbb{Y}}$, respectively the model’s prediction and target label, and $\hat{p}$ its predicted confidence, a perfectly calibrated model should satisfy $P(\hat{y}=y|\hat{p}=p)=p$, for $p\in[0,1]$.

We use several metrics for calibration in the paper, namely, ECE, AECE, Brier score and NLL for classification tasks, and UCE and ENCE for regression tasks. We formally introduce all of them here.

### C.1 Metrics for classification tasks

##### NLL

The *negative log-likelihood* (NLL) is a common metric for a model’s prediction quality (Hastie et al.,, ). It is equivalent to cross-entropy in multi-class classification. NLL is defined as:

$$ $\text{NLL}({\mathbf{x}},{\mathbf{y}})=-\frac{1}{N}\sum_{i=1}^{N}\log(\hat{p}({ \mathbf{y}}_{i}|{\mathbf{x}}_{i})),$ (5) $$

where $\hat{p}({\mathbf{y}}_{i}|{\mathbf{x}}_{i})$ represents the confidence of the model in the output associated to ${\mathbf{x}}_{i}$ for the target class ${\mathbf{y}}_{i}$.

##### Brier score

The Brier score (Brier,, ) for multi-class classification is defined as

$$ $\text{Brier}({\mathbf{x}},{\mathbf{y}})=-\frac{1}{N}\sum_{i=1}^{N}\sum_{j=1}^{ c}(\hat{p}(y_{(i,j)}|{\mathbf{x}}_{i})-y_{(i,j)})^{2},$ (6) $$

where we assume that the target label ${\mathbf{y}}_{i}$ is represented as a one-hot vector over the $c$ possible class, *i.e.*, ${\mathbf{y}}_{i}\in\mathbb{R}^{c}$. Brier score is the mean square error (MSE) between predicted confidence and target.

##### ECE

Expected Calibration Error (ECE) is a popular metric for calibration performance for classification tasks in practice. It approximates the difference between accuracy and confidence in expectation by first grouping all the samples into $M$ equally spaced bins $\{B_{m}\}_{m=1}^{M}$ with respect to their confidence scores, then taking a weighted average of the difference between accuracy and confidence for each bin. Formally, ECE is defined as (Guo et al.,, ):

$$ $\text{ECE}:=\sum_{m=1}^{M}\frac{|B_{m}|}{N}|\text{acc}(B_{m})-\text{conf}(B_{m })|,$ (7) $$

with $\text{acc}(B_{m})=\frac{1}{|B_{m}|}\sum_{i\in B_{m}}\mathds{1}_{\hat{y}_{i}=y_
{i}}$ the accuracy of bin $B_{m}$, and $\text{conf}(B_{m})=\frac{1}{|B_{m}|}\sum_{i\in B_{m}}\hat{p}({\mathbf{y}}_{i}|
{\mathbf{x}}_{i})$ the average confidence within bin $B_{m}$.

##### AECE

The Adaptive ECE (AECE) is computed similarly to ECE, with the difference that bin sizes are calculated to evenly distribute samples cccross the bins.

### C.2 Metrics for regression tasks

A probabilistic regression model takes ${\mathbf{x}}\in{\mathbb{X}}$ as input and outputs a mean $\mu_{y}({\mathbf{x}})$ and a variance $\sigma^{2}_{y}({\mathbf{x}})$ targeting the ground-truth $y\in{\mathbb{Y}}$. The UCE and ENCE calibration metrics are both extension of ECE for regression tasks to evaluate *variance calibration*. They both apply a binning scheme with $M$ bins over the predicted variance.

##### UCE

Uncertainty Calibration Error (UCE) (Laves et al.,, ) measures the average of the absolute difference between *mean squared error (MSE)* and *mean variance (MV)* within each bin. It is formally defined by

$$ $\text{UCE}:=\sum_{m=1}^{M}\frac{|B_{m}|}{N}|\text{MSE}(B_{m})-\text{MV}(B_{m})|,$ (8) $$

with $\text{MSE}(B_{m})=\frac{1}{|B_{m}|}\sum_{i\in B_{m}}(\mu_{y_{i}}({\mathbf{x}}_
{i})-y_{i})^{2}$ and $\text{MV}(B_{m})=\frac{1}{|B_{m}|}\sum_{i\in B_{m}}\sigma^{2}_{y_{i}}({\mathbf
{x}}_{i})^{2}$.

##### ENCE

Expected Normalized Calibration Error (ENCE) (Levi et al.,, ) measures the absolute *normalized* difference, between *root mean squared error (RMSE)* and *root mean variance (RMV)* within each bin. It is formally defined by

$$ $\text{ENCE}:=\frac{1}{M}\sum_{m=1}^{M}\frac{|\text{RMSE}(B_{m})-\text{RMV}(B_{ m})|}{\text{RMV}(B_{m})},$ (9) $$

with $\text{RMSE}(B_{m})=\sqrt{\frac{1}{|B_{m}|}\sum_{i\in B_{m}}(\mu_{y_{i}}({
\mathbf{x}}_{i})-y_{i})^{2}}$ and $\text{RMV}(B_{m})=\sqrt{\frac{1}{|B_{m}|}\sum_{i\in B_{m}}\sigma^{2}_{y_{i}}({
\mathbf{x}}_{i})^{2}}$.

## Appendix D Effect of distance on calibration

**Table 6: Performance (Accuracy in %) and calibration (ECE, Brier, NLL) *before and after Temperature Scaling (TS)* Guo et al., () with Resnet34 when mixing only elements higher or lower than a quantile $q$. Best in bold, second best underlined.**
| Dataset | Quantile of Distance | Accuracy ($\uparrow$) | ECE ($\downarrow$) | Brier ($\downarrow$) | NLL ($\downarrow$) | TS ECE ($\downarrow$) | TS Brier ($\downarrow$) | TS NLL ($\downarrow$) |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| C10 | Lower 0.0 / Higher 1.0 (ERM) | 94.69 $\pm$ 0.27 | 3.65 $\pm$ 0.22 | 8.90 $\pm$ 0.4 | 24.57 $\pm$ 1.04 | 0.82 $\pm$ 0.11 | 8.07 $\pm$ 0.31 | 17.50 $\pm$ 0.61 |
| Lower 1.0 / Higher 0.0 (Mixup) | 95.97 $\pm$ 0.27 | 12.3 $\pm$ 0.92 | 8.54 $\pm$ 0.64 | 25.68 $\pm$ 1.53 | 1.36 $\pm$ 0.13 | 6.53 $\pm$ 0.36 | 16.35 $\pm$ 0.72 |  |
| Lower 0.1 | 95.70 $\pm$ 0.24 | 1.58 $\pm$ 0.3 | 7.12 $\pm$ 0.36 | 17.9 $\pm$ 1.01 | 0.99 $\pm$ 0.3 | 7.08 $\pm$ 0.37 | 16.1 $\pm$ 0.85 |  |
| Lower 0.25 | 95.73 $\pm$ 0.18 | 2.42 $\pm$ 0.55 | 7.12 $\pm$ 0.25 | 19.54 $\pm$ 1.1 | 1.74 $\pm$ 0.45 | 7.07 $\pm$ 0.26 | 19.39 $\pm$ 1.11 |  |
| Lower 0.5 | 95.88 $\pm$ 0.28 | 3.04 $\pm$ 0.4 | 6.67 $\pm$ 0.34 | 16.68 $\pm$ 0.84 | 1.56 $\pm$ 0.28 | 6.68 $\pm$ 0.34 | 15.86 $\pm$ 0.71 |  |
| Lower 0.75 | 96.16 $\pm$ 0.09 | 3.63 $\pm$ 0.32 | 6.33 $\pm$ 0.14 | 16.55 $\pm$ 0.59 | 1.12 $\pm$ 0.16 | 6.35 $\pm$ 0.15 | 15.20 $\pm$ 0.44 |  |
| Lower 0.9 | 96.31 $\pm$ 0.08 | 3.71 $\pm$ 0.34 | 6.20 $\pm$ 0.24 | 16.65 $\pm$ 0.41 | 1.10 $\pm$ 0.05 | 6.14 $\pm$ 0.11 | 15.16 $\pm$ 0.29 |  |
| Higher 0.9 | 95.58 $\pm$ 0.34 | 2.72 $\pm$ 0.2 | 7.39 $\pm$ 0.49 | 20.54 $\pm$ 1.31 | 1.86 $\pm$ 0.25 | 7.4 $\pm$ 0.48 | 20.32 $\pm$ 1.25 |  |
| Higher 0.75 | 95.91 $\pm$ 0.14 | 3.68 $\pm$ 0.19 | 6.86 $\pm$ 0.21 | 20.7 $\pm$ 0.88 | 1.85 $\pm$ 0.17 | 6.84 $\pm$ 0.22 | 20.06 $\pm$ 1.12 |  |
| Higher 0.5 | 95.58 $\pm$ 0.28 | 4.55 $\pm$ 0.63 | 7.28 $\pm$ 0.38 | 20.71 $\pm$ 1.14 | 1.67 $\pm$ 0.13 | 7.23 $\pm$ 0.37 | 19.12 $\pm$ 0.74 |  |
| Higher 0.25 | 95.98 $\pm$ 0.3 | 4.28 $\pm$ 0.32 | 6.66 $\pm$ 0.49 | 18.73 $\pm$ 0.97 | 1.24 $\pm$ 0.18 | 6.65 $\pm$ 0.51 | 17.06 $\pm$ 0.99 |  |
| Higher 0.1 | 96.28 $\pm$ 0.03 | 4.38 $\pm$ 0.16 | 6.15 $\pm$ 0.05 | 17.18 $\pm$ 0.28 | 1.13 $\pm$ 0.11 | 6.14 $\pm$ 0.04 | 15.24 $\pm$ 0.37 |  |
| C100 | Lower 0.0 / Higher 1.0 (ERM) | 73.47 $\pm$ 1.59 | 13.0 $\pm$ 0.75 | 39.56 $\pm$ 2.19 | 121.06 $\pm$ 7.92 | 2.54 $\pm$ 0.15 | 36.47 $\pm$ 2.05 | 100.82 $\pm$ 6.93 |
| Lower 1.0 / Higher 0.0 (Mixup) | 78.11 $\pm$ 0.57 | 11.92 $\pm$ 0.73 | 32.82 $\pm$ 0.87 | 95.76 $\pm$ 2.43 | 2.49 $\pm$ 0.19 | 31.06 $\pm$ 0.69 | 87.94 $\pm$ 1.98 |  |
| Lower 0.1 | 75.40 $\pm$ 0.53 | 4.72 $\pm$ 0.32 | 35.87 $\pm$ 0.52 | 108.38 $\pm$ 1.12 | 3.48 $\pm$ 0.24 | 35.92 $\pm$ 0.5 | 105.87 $\pm$ 1.41 |  |
| Lower 0.25 | 77.14 $\pm$ 0.51 | 2.56 $\pm$ 0.17 | 32.93 $\pm$ 0.64 | 95.47 $\pm$ 1.75 | 2.54 $\pm$ 0.22 | 32.95 $\pm$ 0.62 | 95.42 $\pm$ 1.78 |  |
| Lower 0.5 | 77.66 $\pm$ 0.15 | 3.40 $\pm$ 1.17 | 32.10 $\pm$ 0.41 | 90.75 $\pm$ 2.02 | 1.85 $\pm$ 0.43 | 31.94 $\pm$ 0.28 | 89.97 $\pm$ 1.53 |  |
| Lower 0.75 | 78.43 $\pm$ 0.62 | 4.38 $\pm$ 1.58 | 30.85 $\pm$ 0.65 | 86.83 $\pm$ 1.81 | 1.95 $\pm$ 0.6 | 30.64 $\pm$ 0.7 | 85.06 $\pm$ 1.89 |  |
| Lower 0.9 | 79.24 $\pm$ 0.7 | 6.02 $\pm$ 1.02 | 30.11 $\pm$ 1.05 | 85.53 $\pm$ 3.32 | 1.99 $\pm$ 0.03 | 29.72 $\pm$ 0.94 | 82.54 $\pm$ 2.82 |  |
| Higher 0.9 | 77.3 $\pm$ 0.43 | 3.55 $\pm$ 1.66 | 32.19 $\pm$ 0.78 | 90.54 $\pm$ 2.56 | 1.92 $\pm$ 0.22 | 32.0 $\pm$ 0.59 | 88.69 $\pm$ 1.67 |  |
| Higher 0.75 | 77.8 $\pm$ 1.05 | 3.77 $\pm$ 1.14 | 31.56 $\pm$ 1.23 | 90.16 $\pm$ 4.16 | 2.29 $\pm$ 0.24 | 31.48 $\pm$ 1.18 | 88.16 $\pm$ 3.86 |  |
| Higher 0.5 | 78.74 $\pm$ 0.43 | 4.32 $\pm$ 0.96 | 30.47 $\pm$ 0.54 | 86.96 $\pm$ 1.87 | 2.52 $\pm$ 0.22 | 30.37 $\pm$ 0.56 | 84.64 $\pm$ 1.63 |  |
| Higher 0.25 | 78.51 $\pm$ 0.47 | 3.35 $\pm$ 1.19 | 30.13 $\pm$ 0.6 | 85.49 $\pm$ 2.6 | 2.34 $\pm$ 0.26 | 30.42 $\pm$ 0.59 | 84.64 $\pm$ 2.23 |  |
| Higher 0.1 | 79.14 $\pm$ 0.53 | 5.32 $\pm$ 2.15 | 29.94 $\pm$ 0.76 | 85.09 $\pm$ 2.53 | 2.23 $\pm$ 0.34 | 29.62 $\pm$ 0.51 | 82.22 $\pm$ 1.28 |  |

In Table 6, we show the exact results for the plot presented in Section 3.1, along with results obtained for different quantiles $q$. One should compare results of "Lower $q$" with "Higher $1-q$" to have equivalent numbers of possible element to mix with (*diversity*). We repeat our observations here for ease of reading. First, we observe that Mixup improves upon ERM’s accuracy, but can degrade calibration depending on the dataset, which is consistent with findings from Wang et al., (). Then, when selecting pairs according to distance, a sufficiently high proportion of data to mix is necessary to preserve accuracy ($q>0.5$). Finally, mixing data with *lower* distances achieves a better calibration as opposed to mixing data with *higher* distances.

## Appendix E Detailed Algorithm

Figure: Algorithm 1 Similarity Kernel Mixup training procedure

We present a pseudocode of our *Similarity Kernel Mixup* procedure for a single training iteration in Algorithm 1. The generation of new data is explained in the pseudocode as a sequential process for simplicity and ease of understanding, but the actual implementation is optimized to work in parallel on GPU through vectorized operations.

## Appendix F Proof of Proposition

We give the proof of Proposition 3.1 below:

###### Proof.

Let $\tau>0$ and $F_{\tau}(x)=I_{x}(\tau,\tau)$, then $F_{\tau}$ is the *cumulative distribution function (CDF)* of a Beta distribution with parameters $(\tau,\tau)$.
$\forall x\in[0,1]$, we have:

$$ $\displaystyle P(\omega_{\tau}(\lambda)\leq x)$ $\displaystyle=P(I^{-1}_{\lambda}(\tau,\tau)\leq x)$ (10) $\displaystyle=P(\lambda\leq I_{x}(\tau,\tau))$ (11) $\displaystyle=I_{x}(\tau,\tau),$ (12) $$

where Equation 11 is obtained since $F_{\tau}$ is continuous and monotonically increasing on $[0,1]$, and Equation 12 because $\lambda$ follows a uniform distribution on $[0,1]$. It follows that $\omega_{\tau}(\lambda)\sim\texttt{Beta}(\tau,\tau)$.
∎

###### Proof.

## Appendix G Benefits of Warping

### G.1 Disentangling Inputs and Targets

Using such warping functions presents the advantage of being able to easily separate the mixing of *inputs* and *targets*, by defining different *warping parameters* $\tau^{(i)}$ and $\tau^{(t)}$:

$$ $\lambda_{t}\sim\texttt{Beta}(1,1),\qquad\begin{aligned} \tilde{{\mathbf{x}}}_{ i}&:=\omega_{\tau^{(i)}}(\lambda_{t}){\mathbf{x}}_{i}+(1-\omega_{\tau^{(i)}}( \lambda_{t})){\mathbf{x}}_{\sigma_{t}(i)}\\ \tilde{y}_{i}&:=\omega_{\tau^{(t)}}(\lambda_{t})y_{i}+(1-\omega_{\tau^{(t)}}( \lambda_{t}))y_{\sigma_{t}(i)}.\end{aligned}$ (13) $$

Disentangling *inputs* and *targets* can be interesting when working in the imbalanced setting (Chou et al.,, ). Notably, with $\tau^{(i)}=1,\tau^{(t)}\approx 0$, we recover the Mixup Input-Only (IO) variant (Wang et al.,, ) where only inputs are mixed, used in the experiments in Table 1, and with $\tau^{(i)}\approx 0,\tau^{(t)}=1$, the Mixup Target-Only (TO) variant (Wang et al.,, ), where only labels are mixed. It can also reveal interesting for *structured prediction*, where targets are structured objects such as *graph prediction* or *depth estimation*.

### G.2 Computational efficiency

On a more practical side, although the Beta CDF and its inverse have no closed form solutions for non-integer values of its parameters $\alpha$ and $\beta$, accurate approximations are implemented in many statistical software packages.
Then, sampling from Beta distributions with different parameters for each pair of points cannot be done directly on GPU. Coefficients sampled for each pair need to be sent to GPU at each batch, slowing the training because of CPU-GPU synchronizations. An efficient implementation is to define a single torch.distributions.Beta with parameters $\alpha=\beta=1$ (or Uniform) on GPU, and then compute a linear approximation of the inverse Beta CDF from *precomputed lookup tables*, which is common when using inverse transform sampling. We present a comparison in Table 7, in terms of performance (accuracy and calibration) and computation time, of the three different variants of implementation discussed when training a ResNet34 on CIFAR10 on a single A100 GPU ($\tau_{max}=1,\tau_{std}=0.4$). The three variants achieve comparable performance, and the time difference is in favour of *using warping functions and lookup tables*.

**Table 7: Comparison of performance (Accuracy in %), calibration (ECE, AECE) after Temperature Scaling, and total training time (in min), with three different variants. All experiments are using a ResNet34 on CIFAR10 on a single A100 GPU and reproduced on 4 different random seeds.**
| Method | Accuracy ($\uparrow$) | ECE ($\downarrow$) | AECE ($\downarrow$) | Total training time (in min) ($\downarrow$) |
| --- | --- | --- | --- | --- |
| Warping | 96.42 $\pm$ 0.24 | 0.53 $\pm$ 0.06 | 0.71 $\pm$ 0.05 | 57.36 $\pm$ 0.31 |
| Warping w/ lookup | 96.04 $\pm$ 0.2 | 0.57 $\pm$ 0.1 | 0.70 $\pm$ 0.12 | 55.59 $\pm$ 0.4 |
| No warping | 96.3 $\pm$ 0.1 | 0.55 $\pm$ 0.07 | 0.76 $\pm$ 0.16 | 56.88 $\pm$ 1.03 |

## Appendix H Detailed experimental settings

Image Classification . For all our experiments, we follow the protocols of Noh et al., (). On CIFAR10 and CIFAR100, we use SGD as the optimizer with a momentum of $0.9$ and weight decay of $10^{-4}$, a batch size of $128$, and the standard augmentations random crop, horizontal flip and normalization. Models are trained for 200 epochs, with an initial learning rate of $0.1$ divided by a factor $10$ after $80$ and $120$ epochs. On Tiny-ImageNet, models are trained for 100 epochs using SGD with an initial learning rate of 0.1 divided by a factor of 10 after 40 and 60 epochs, a momentum of 0.9 and weight decay of $10^{-4}$. We use a batch size of 64 and the same standard augmentations. On Imagenet, models are trained for 100 epochs using SGD with an initial learning rate of 0.1, a cosine annealing scheduler, a momentum of 0.9 and weight decay of $10^{-4}$. We use a total batch of 256 split over 3 GPUs and standard augmentations.
Regression . Following , we train a three-layer fully connected network augmented with Dropout Srivastava et al., () on Airfoil, and LST-Attn Lai et al., () on Exchange-Rate and Electricity. All models are trained for 100 epochs with the Adam optimizer Kingma and Ba, (), with a batch size of 16 and learning rate of $0.01$ on Airfoil, and a batch size of $128$ and learning rate of $0.001$ on Exchange-Rate and Electricity. To estimate variance for calibration, we rely on MC Dropout Gal and Ghahramani, () with a dropout of $0.2$ and $50$ samples.
Method-specific hyperparameters . On *classification datasets*, we used $\tau_{max}=1$ and $\tau_{std}=0.25$ unless stated otherwise. We used the hyperparameters provided by the authors to reproduce state-of-the-art methods, namely, $\alpha=1$ and $\Delta\lambda>0.5$ for MIT-A (Wang et al.,, ), and $w=0.1$, $\alpha=2.0$ and $\alpha=1.0$ for CIFAR10/100 and Tiny-ImageNet respectively, and $Q=4$, for RankMixup (M-NDCG) (Noh et al.,, ). To compare with Mixup, we use $\alpha=0.2$, the best performing one found in Thulasidasan et al., (), and the same value for Manifold Mixup (Verma et al.,, ). On Imagenet, we set $\tau_{max}=0.1$ and $\tau_{std}=0.25$, and $\alpha=0.1$.
On *regression datasets*, we used $(\tau_{max},\tau_{std})=(0.0001,0.5)$ on Airfoil, $(\tau_{max},\tau_{std})=(5,1)$ on Exchange Rate, and $(\tau_{max},\tau_{std})=(2,0.2)$ on Electricity. We followed results from , and fixed $\alpha=0.5$ on Airfoil, $\alpha=1.5$ on Exchange Rate and $\alpha=2$ on Electricity, for Mixup, Manifold Mixup and C-Mixup. Additionally, we searched for the best *bandwidth* parameter for C-Mixup with cross-validation, and found $0.01$ on Airfoil, $0.05$ on Exchange Rate, and $0.5$ on Electricity. Likewise, we searched for the best $\alpha$ for RegMixup, and found $\alpha=0.5$ on Airfoil, $\alpha=10$ on Exchange-Rate and $\alpha=10$ on Electricity.

## Appendix I On the selection and sensitivity to hyperparameters

As discussed in Section 3.2, we always draw initial interpolation parameters $\lambda$ from Beta(1,1), removing $\alpha$ as a hyperparameter. Then, $\tau_{max}$ and $\tau_{std}$ can be tuned separately, and, as mentioned in Section 4.2, we found that impact on calibration was mainly controlled by $\tau_{std}$. We selected the values giving the best trade-off between accuracy and calibration using cross-validation, with a stratified sampling on a 90/10 split of the training set, similarly to Pinto et al., (), and average the results across 4 different splits.
We detail in Table 8 cross-validation results (Accuracy and ECE) showing sensitivity to $\tau_{std}$ on CIFAR10 and CIFAR100 datasets, using a ResNet50. These results lead us to using $\tau_{std}=0.25$ for all image classification experiments, and show that our approach is not that sensitive to hyperparameter choice between datasets. For Imagenet, we set $\tau_{max}=0.1$ to follow the commonly used value of $\alpha=0.1$ in Mixup for this dataset.

**Table 8: Comparison of Accuracy and ECE, before and after Temperature Scaling (TS), with cross-validation when varying $\tau_{\text{std}}$. Results are obtained with a Resnet50 and averaged over 4 splits.**
| Dataset | $\tau_{\text{std}}$ | Accuracy | ECE ($\downarrow$) | TS ECE ($\downarrow$) |
| --- | --- | --- | --- | --- |
| CIFAR10 | 0.25 | 95.18 | 2.87 | 0.97 |
| 0.4 | 95.23 | 3.31 | 1.42 |  |
| 0.5 | 95.37 | 2.7 | 2.08 |  |
| 0.6 | 95.45 | 3.15 | 1.99 |  |
| 0.75 | 95.57 | 3.44 | 2.05 |  |
| 0.8 | 95.44 | 3.28 | 2.07 |  |
| 1 | 95.57 | 3.44 | 2.05 |  |
| 1.25 | 95.46 | 2.71 | 2.25 |  |
| 1.5 | 95.29 | 3.42 | 2.36 |  |
| 2 | 94.93 | 3.24 | 2.43 |  |
| CIFAR100 | 0.25 | 77.74 | 3.98 | 1.77 |
| 0.4 | 78.26 | 4.24 | 2.4 |  |
| 0.5 | 78.37 | 3.45 | 3.61 |  |
| 0.6 | 78.29 | 2.9 | 4.08 |  |
| 0.75 | 78 | 3.65 | 3.91 |  |
| 0.8 | 78.03 | 3.33 | 4.27 |  |
| 1 | 77.35 | 4.82 | 3.4 |  |
| 1.25 | 77.31 | 3.98 | 3.88 |  |
| 1.5 | 77.92 | 4.19 | 4.06 |  |

## Appendix J On the choice of similarity kernel

### J.1 On Warping functions

As discussed in Section 3.3, the choice of the similarity kernel is highly dependent on the warping function, and more specifically to the correlation between the warping parameter and the shape of the warping function. Given our choice of using the inverse of the Beta CDF as $\omega_{\tau}$, and that its shape is logarithmically correlated to $\tau$, the choice of a Gaussian kernel seems natural to have an exponential correlation with the distance. Then, the normalization and centering avoid dealing with different range of embedding values between datasets and architectures when choosing a correct value of $\tau_{std}$. Regarding the choice of warping function, as mentioned in Section 3.2, any bijection with a sigmoidal shape could be considered. Besides the inverse of Beta CDF, we also tried the Beta CDF and the Sigmoid ($\lambda\mapsto\frac{1}{1+e^{\lambda/\tau}}$). As they all have logarithmic correlations wrt $\tau$, we used our Gaussian similarity kernel for the three of them. We compare performance on Airfoil dataset after finding the best parameters in each case in Table 9. The inverse of Beta CDF have both the advantage of better results, while preserving the underlying Beta distribution by inverse transform sampling, as we show in Proposition 3.1.

**Table 9: Performance (RMSE, MAPE) and calibration (UCE, ENCE) comparison for different warping functions on Airfoil dataset. Averaged over 10 random seeds.**
| Warping | RMSE ($\downarrow$) | MAPE ($\downarrow$) | UCE ($\downarrow$) | ENCE ($\downarrow$) |
| --- | --- | --- | --- | --- |
| Sigmoid | 2.892 $\pm$ 0.343 | 1.733 $\pm$ 0.229 | 117.95 $\pm$ 45.26 | 0.018 $\pm$ 0.008 |
| Beta CDF | 2.807 $\pm$ 0.261 | 1.694 $\pm$ 0.176 | 126.02 $\pm$ 23.32 | 0.018 $\pm$ 0.005 |
| Inverse Beta CDF | 2.707 $\pm$ 0.199 | 1.609 $\pm$ 0.137 | 93.79 $\pm$ 25.46 | 0.019 $\pm$ 0.008 |

### J.2 On embedding spaces and distances

**Table 10: Performance (RMSE, MAPE) and calibration (UCE, ENCE) comparison for different similarity distance on Airfoil dataset. Averaged over 10 random seeds.**
| Warping | RMSE ($\downarrow$) | MAPE ($\downarrow$) | UCE ($\downarrow$) | ENCE ($\downarrow$) |
| --- | --- | --- | --- | --- |
| Input distance | 2.848 $\pm$ 0.355 | 1.706 $\pm$ 0.215 | 105.23 $\pm$ 26.32 | 0.018 $\pm$ 0.008 |
| Embedding distance | 2.737 $\pm$ 0.205 | 1.636 $\pm$ 0.142 | 101.62 $\pm$ 21.94 | 0.016 $\pm$ 0.005 |
| Label distance | 2.707 $\pm$ 0.199 | 1.609 $\pm$ 0.137 | 93.79 $\pm$ 25.46 | 0.019 $\pm$ 0.008 |

**Table 11: Comparison of performance (Accuracy in %), calibration (ECE) after Temperature Scaling, for two different similarity distance. All experiments are using a ResNet34 on CIFAR10 and CIFAR100 datasets and averaged over 4 different random seeds.**
| Dataset | Distance | $\tau_{\text{std}}$ | Accuracy ($\uparrow$) | ECE ($\downarrow$) |
| --- | --- | --- | --- | --- |
| CIFAR10 | Input distance | 0.2 | 95.82 $\pm$ 0.19 | 0.41 $\pm$ 0.13 |
| 0.4 | 96.10 $\pm$ 0.17 | 0.46 $\pm$ 0.06 |  |  |
| 0.6 | 95.87 $\pm$ 0.47 | 0.6 $\pm$ 0.13 |  |  |
| 0.8 | 96.12 $\pm$ 0.15 | 0.70 $\pm$ 0.08 |  |  |
| 1.0 | 96.01 $\pm$ 0.27 | 0.74 $\pm$ 0.19 |  |  |
| Embedding distance | 0.2 | 96.29 $\pm$ 0.07 | 1.35 $\pm$ 0.14 |  |
| 0.4 | 96.42 $\pm$ 0.24 | 0.53 $\pm$ 0.06 |  |  |
| 0.6 | 96.36 $\pm$ 0.09 | 0.52 $\pm$ 0.08 |  |  |
| 0.8 | 96.0 $\pm$ 0.41 | 0.56 $\pm$ 0.05 |  |  |
| 1.0 | 96.25 $\pm$ 0.07 | 0.55 $\pm$ 0.12 |  |  |
| CIFAR100 | Input distance | 0.2 | 78.43 $\pm$ 0.52 | 1.51 $\pm$ 0.10 |
| 0.4 | 78.24 $\pm$ 0.35 | 1.35 $\pm$ 0.22 |  |  |
| 0.6 | 78.36 $\pm$ 0.97 | 1.5 $\pm$ 0.20 |  |  |
| 0.8 | 78.50 $\pm$ 0.41 | 1.5 $\pm$ 0.16 |  |  |
| 1.0 | 78.50 $\pm$ 0.83 | 1.53 $\pm$ 0.08 |  |  |
| Embedding distance | 0.2 | 78.13 $\pm$ 0.52 | 1.31 $\pm$ 0.23 |  |
| 0.4 | 78.86 $\pm$ 0.83 | 1.42 $\pm$ 0.19 |  |  |
| 0.6 | 78.63 $\pm$ 0.27 | 1.74 $\pm$ 0.30 |  |  |
| 0.8 | 79.12 $\pm$ 0.52 | 1.66 $\pm$ 0.16 |  |  |
| 1.0 | 78.51 $\pm$ 0.94 | 1.49 $\pm$ 0.03 |  |  |

#### J.2.1 Regression tasks

As discussed in Section 3.3, in regression tasks, we considered either input, embedding or label distance:

- •
Input distance: $\bar{d}_{n}({\mathbf{x}}_{i},{\mathbf{x}}_{\sigma(i)})$,
- •
Embedding distance: $\bar{d}_{n}(h_{\varphi}({\mathbf{x}}_{i}),h_{\varphi}({\mathbf{x}}_{\sigma(i)})$,
- •
Label distance: $\bar{d}_{n}(y_{i},y_{\sigma(i)})$.

Input and label distances both have the advantages of inducing almost no computational overhead, as opposed to embedding distance that requires an additional forward pass in the network. We report results comparing the three options on Airfoil dataset in Table 10. We found that label distance was the best performing one, which is in line with results from , and what we use in the experiments in Section 4.3.

#### J.2.2 Classification tasks

In classification tasks, as we lack a meaningful distance between label, we restricted our choice between input and embedding distance. Even though computing distances directly in the input space is faster than relying on embeddings, the latter achieves a better trade-off between accuracy and calibration improvements, which motivated us to use embedding distances as similarity in our experiments in Section 4.2. We compare in Table 11 the two distances, for the different value of $\tau_{std}$, using a Resnet34 on CIFAR10 and CIFAR100 datasets.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 8
presentation: 4
questions: **Question 1.** Why did not SK RegMixup be applied to ViT in Table 4? Is there any potential challenges or limitations of applying SK RegMixup to ViT architectures?

**Question 2.** [line 976-978] In the proof of the first part in Theorem 3.1., whatever the value of $\lambda\_{1}$ is, $\lambda$ can be 0. When $\lambda=0$, then $\tilde{\mathbf{x}}(\lambda)=\mathbf{x}\_{l} \in \mathcal{M}\_{j}$, not $\mathcal{M}\_{i}$, and so does symmetrical case. In my opinion, it would be $\forall \lambda \geq \lambda\_{1},~\tilde{\mathbf{x}}(\lambda)=\lambda \mathbf{x}\_{k}+(1-\lambda)\mathbf{x}\_{l} \in \mathcal{M}\_{j}$. The authors are kindly requested to examine and fix them if there is an error.

---

**Things to improve the paper that did not impact the score:**

* Figure 4 caption: there is no result about *Circles* toy datasets. The mention of it from the caption would be removed if it's not intended to be included

* [line 442] The ECE and AECE of MIT-A in ImageNet-R would be in bold. The authors have to double-check their result highlighting in Table 4 to ensure consistency and accuracy.

*  In Table 4, is there any reason why ECE and AECE are exactly same in OOD settings? The authors are kindly requested to explain why ECE and AECE are identical in OOD settings or, if this is unexpected, verify whether there might be an error in the reporting or calculation of these metrics.

* The experimental results show that the SK RegMixup is an effective method. However, Table 6 does not provide the efficiency of the SK RegMixup in terms of computational cost. It is recommended to include computational cost metrics for SK RegMixup in Table 6, similar to what they've provided for other methods.

* [line 879] Typo: cccross
soundness: 4
strengths: * The proposed approach was well motivated by targeting the appropriate research problem of calibration-driven Mixup methods in classification and regression: Manifold mismatch. Furthermore, this problem was theoretically defined and proved with the supplement materials.

* The proposed method was in a clear and well-organized manner, and the proof was provided in a manner that supported its conclusions.

* The research problem was validated through experimental results; a comparison was made with the baseline.

* Comprehensive experiments in classification and regression, with a particular focus on performance and computational cost, demonstrated the efficacy of the proposed approach.
summary: This paper proposes a novel Mixup framework that employs a Similarity Kernel (SK) called SK Mixup to achieve a stronger interpolation between similar points while reducing interpolation otherwise. As a motivation of this study, the authors defined the concept of manifold mismatch, which can negatively impact the calibration of confidence in Mixup. They conducted experimental validation to assess the impact of this phenomenon on the distance between points to mix. Following the presentation of SK Mixup, the effectiveness of the proposed approach in alleviating the manifold mismatch was demonstrated through extensive experiments in classification and regression. Nevertheless, the extension to more complex tasks and theoretical analysis on the regularization effect of SK Mixup still need to be undertaken -- This is why the authors did not receive the highest score in *Contribution*.
weaknesses: There is no weakness in the paper that would justify its rejection.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: - In Eq. (3), there is a -1 inside the exponential. Why not absorb it into $\tau_{max}$? Are there any reasons why it is presented that way? 
- In the classification scenario where the data samples (but not the labels) are noisy and corrupted, thus blurring the true distance between the samples used in mixing, how would the proposed method perform?
soundness: 3
strengths: - Overall the paper is well written and is easy to follow
- An innovative yet efficient method is proposed to improve mixup so that manifold mismatch likelihood is reduced, and better calibration and accuracies can be achieved
- Adequate empirical results on both classification and regression tasks are provided to demonstrate the proposed method
summary: This paper first demonstrates that distance between the data used in mixup can impact the likelihood of manifold mismatch (a phenomenon where mixed samples lie outside of the class manifolds of the original data used for mixing). It then proposes an efficient framework to mitigate the occurrence of manifold mismatch. The key idea is to dynamically change the distributions of mixing coefficients via a similarity kernel that measures the distance between the mixed samples. Empirical results are provided to demonstrate the effectiveness of the proposed method.
weaknesses: - While Theorem 1 shows the existence of manifold mismatch, it does not tell us anything about the assumption (which is used in the paper) that the higher the distance between two points, the more likely their convex combination will fall outside of the original manifolds, and thus, the more likely a model would assign a different label than the original labels of the two points. Some theoretical results in establishing the validity of such assumption would further strengthen the paper. 
- There are some missing papers that could be worth mentioning in the related work. In particular, mixup related works such as [1,2,3]. 
- For the experiments, it is not clear how the proposed method would compare with the method proposed by [1,2,3], and also how would it perform for more complex datasets like ImageNet, ImageNet-C, ImageNet-R and ImageNet-P.

---

[1] Dan Hendrycks, Norman Mu, Ekin D. Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshminarayanan. AugMix: A simple data processing method to improve robustness and uncertainty. Proceedings of the ICLR, 2020

[2] Soon Hoe Lim, N. Benjamin Erichson, Francisco Utrera, Winnie Xu, and Michael W. Mahoney. Noisy feature mixup. Proceedings of the ICLR, 2022.

[3] Erichson, N. Benjamin, Soon Hoe Lim, Francisco Utrera, Winnie Xu, Ziang Cao, and Michael W. Mahoney. Noisymix: Boosting robustness by combining data augmentations, stability training, and noise injections. Proceedings of the AISTATS 2024.

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 1
questions: 1. Line 011 or 012 on page 1. "Along with improved performance", what performance, please make it specified
2. Line 012 or 013 on page 1. "improving calibration and predictive confidence", from my understanding they are the same thing, no need to use "and" here in my opinion.
3. Line 014. Infact in this line the paper has mentioned "calibration of confidence", what is calibration of "confidence", again, in my opinion it's just calibration.
4. Line 019 to 020. Again, please specify what "performance" it is when mentioning "improve performance".
5. Line 054. Here the paper mentions "label noise", can the authors provide a brief description of it like the way they have described manifold intrusion in previous text?
6. Line 140. Again, please specify the term "performance".
7. Is there and related research works about calibration-driven Mixup methods in regression tasks?
8. Line 161. If possible it's much better to turn this title to the next page, it would look better.
9. Line 162 to 169. The notation preliminaries in this paragraph feel a bit too simple and careless. For example, when defining a dataset, one may want to first define a data space and label space, indicating that these spaces are subspaces of some vector spaces of certain dimensions, and then indicate that the training dataset is of some size with the examples being drawn from some data distribution over the data space, etcs.. Putting those details in one single line feels careless. Also, what is $M$? Many would of course just simply take it as the number of classes in the classification problems, but it also needs to be specified in the paper. 
10. Line 162 to 169. There is also some notations that are inconsistent. The paper first indicates that the labels are $M$-dim vectors, then what is the input and output dimension of the encoder $h_\phi$? The way it is presented make the model output $f_\theta(\text{x})$ feels like a scalar, while it should be a vector the same dimension as the labels since $\hat{y}:=f_\theta(\text{x})$.
11. Line 193 to 194. "bounded support $\mathcal{M}_m\subset\mathcal{H}$". Is this also how the manifolds are defined in this paper?
12. Line 194 to 195. To assume that classes are separated, it means all samples belonging to "different classes manifolds" should disjoint, is that correct?
13. Line 208. Why would the mixed points' belonging "to no vlass manifold at all" be a bad thing? In my opinion, when using Mixup for training, especially when the data dimension is high, it's actually mostly the case that the mixed point fall into the "void" areas in the entire data space.
14. Line 212. "the higher the distance between two points, the more likely their convex combination will fall outside of the original manifolds". Here is a counter example. Suppose we have two classes of points in $\mathbb{R}^2$. Suppose their manifolds are two separated line segments on the horizontal axis. If we combine the rightest point from the left manifold and the leftest point from the right manifold, the mixed points will almost all fall into the middle area which is outside both the original manifolds. If we combine the leftest points from both manifold, the fact is, a much bigger proportion of mixed points would fall inside the left manifold. But, the distance between the points in the second case is much larger than that in the first case.
15. Line 215. "Using a Resnet18 trained ...", is it trained using ERM or Mixup?
16. How to choose $\tau_{std}$ and $\tau_{max}$?
17. How does SK Mixup help improve calibration? The derivation of the algorithm only suggests that SK Mixup can help mitigate manifold mismatch, but is manifold mismatch the necessary reason, or even the real reason, of bad calibration?
18. In the process of trading off between diversity and uncertainty, how would calibration and generalization behave? Will they behave like a up-side-down U curve such that there is a sweet spot?
19. The page number of page 7, and the header "under review" line in page 8, there are some strange hyperlinks around the text.
20. Theorem 3.1 (i). "$\lambda\in]\lambda_1,\lambda_2[$", it should be $[\lambda_1,\lambda_2]$ right?
soundness: 2
strengths: 1. It's a novel idea to summarize several observed phenomenons or properties of Mixup into one unified terminology: manifold mismatch
2. The correlationship between data points pair distance and the occurence of manifold mismatch is carefully derived.
3. The proposed algorithm is meant to balance the diversity and the uncertainty of the virtual examples in Mixup, which is meaningful and helpful
summary: The paper has come up with a concept called manifold mismatch, which is a phenomenon in Mixup that can harm the generalization and calibration performance of the trained models. The paper also show that such a manifold mismatch behavior is correlated to the distance between the data points being paired and mixed in Mixup. Then, by applying a similarity kernel, the paper proposes a variant training algorithm of Mixup that can dynamically adjust the mixing coefficient depending on the distance of the data points being mixed. Finally, the authors have empirically verified the effectiveness of the proposed algorithm on various models and tasks.
weaknesses: 1. Poor presentation
2. Doesn't clarify the connection between manifold mismatch and calibration. 
3. The core claim that distance impacts manifold mismatch is too simple to be seen, and it's actually under-justified since there seems to be much more necessary conditions of manifold mismatch like data distribution, structure of learned features and manifolds, etc.. In fact, the idea of improving Mixup by forcing more mixtures between closer points is not novel, like k-mixup [1]
4. The idea of dynamically adjust the mixing coeffecient is also not new, like AdaMixup [2].

[1] https://arxiv.org/abs/2106.02933

[2] https://arxiv.org/abs/1809.02499

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: - Instead of using a similarity kernel introduced in this paper, a simple implementation is to divide sample pairs based on their distances (like the case in Figure 2) and use two different Beta distributions to sample the interpolation weights. How will this baseline perform compared to the proposed method? Such comparison can further verify the motivation and support the proposed method. 
- While the authors have mentioned that non-linear interpolation methods have several disadvantages (larger computational cost, limited application) in section 2.1, some empirical comparison on their performance for image data sets in Table 2-4 should still be necessary. In addition, the authors may also include some of these interpolation methods in Table 6 to verify that their computational cost are muchh larger compared to linear mixup methods. 
- Moreover, can the proposed method be combined with these non-linear interpolation methods as well? I suppose such combination will be straight-forward, as we only need to change the sampling distribution of interpolation weights for different sample pairs. Some discussion (possibly with some empirical results) will be welcome here. 
- I am a bit puzzled by the performance of different methods on ImageNet-A in Table 4. None of them has an accuracy higher than 3%, which seems worse than most methods reported in [1]. Is this due to some discrepancies in experimental setup (e.g., less training epochs)? If so, the authors may need to clarify that such discrepancies are not in favor of their proposed method. 
- Also, how are the baseline methods chosen in Table 6? From my perspective, I suppose we need to compare the time cost of SK Mixup against Mixup or some other baseline methods with rather good performance. Nevertheless, it seems either of the above cases applies, and some explanation may be needed here. 

## References
[1] On Feature Normalization and Data Augmentation. CVPR 2021
soundness: 2
strengths: The proposed method is clearly introduced and easy to understand
summary: This paper proposes to improve different mixup methods by adjusting the probability distribution to sample interpolation weights according to the distance between a given sample pair. Using the proposed method, sample pairs that are far from each other are less likely to be mixed. Empirical results on different tasks and data sets demonstrate the effectiveness of the proposed method.
weaknesses: - The motivation of proposed method can be strengthened
- Some baseline methods seem missing, which can make the empirical comparison not supportive enough

Meta-review: This paper introduces a new Mixup method that is able of improve both accuracy and calibration. Specifically, the authors propose a Similarity Kernel to parameterize the distribution of interpolation coefficients, making Mixup more data-centric. The idea is both novel and significant. Extensive experiments demonstrate that this approach improves accuracy and calibration across a variety of tasks, including classification and regression.

The reviewers are positive, noting that the method is well motivated, theoretically justified, and supported by strong experimental validation. Major concerns were addressed during the rebuttal phase, and the paper is clear and enjoyable to read. One possible extension is to consider a non-linear or latent space similarity kernel, for instance, employing a pre-trained autoencoder to compute similarity in latent space.

I recommend to accept this submission.

Decision: Accept (Poster)