Using Mixup as a Regularizer Can Surprisingly Improve Accuracy & Out-of-Distribution Robustness
Authors: Francesco Pinto
Harry Yang
Ser-Nam Lim
Philip Torr
Puneet K. Dokania
Venue: NeurIPS 2022 Accept (2022)
Keywords: mixup
reliability
distribution shift
robustness
out-of-distribution detection

Abstract: We show that the effectiveness of the well celebrated Mixup can be further improved if instead of using it as the sole learning objective, it is utilized as an additional regularizer to the standard cross-entropy loss. This simple change not only improves accuracy but also significantly improves the quality of the predictive uncertainty estimation of Mixup in most cases under various forms of covariate shifts and out-of-distribution detection experiments. In fact, we observe that Mixup otherwise yields much degraded performance on detecting out-of-distribution samples possibly, as we show empirically, due to its tendency to learn models exhibiting high-entropy throughout; making it difficult to differentiate in-distribution samples from out-of-distribution ones. 
To show the efficacy of our approach (RegMixup), we provide thorough analyses and experiments on vision datasets (ImageNet & CIFAR-10/100) and compare it with a suite of recent approaches for reliable uncertainty estimation. 

Mixup as a Regularizer Can Surprisingly Improve Accuracy & Out-of-Distribution Robustness
RegMixup:
Francesco Pinto" University of Oxford francesco pintoceng ox ac. uk
Harry Yang Meta Al
Ser-Nam Lim Meta Al
Philip H.S. Torr University of Oxford
Puneet K. Dokania
University of Oxford & Five Al Ltd. puneet. dokeanafeeeai....ie... ai
Abstract
We show that the effectiveness of the well celebrated Mixup [Zhnng el al. 2018) be further improved if instead of using it as the sole learning objective, it is utilized as an additional regularizer to the standard cross-entropy loss. This simple change not only improves accuracy but also significantly improves the quality of the predictive uncertainty estimation of Mixup in most cases under various forms of covariate shifts and nut-of-distribution detection experiments. In fact, we observe that Mixup otherwise yields much degraded performance on detecting out-of- of-diitrrittton samples possibly,
show empirically, due its tendency learn models exhibiting high-entropy throughout; making it difficult to differentiate in-distribution samples from oul-of-distribution ones. To show the efficacy of our approach (RegMixug) P). we provide thorough analyses and experiments on vision datasets (ImageNet & CIFAR-IO/100 and compare with a suite of recent approaches for reliable uncertainty estimation.
Introduction
In real-world machine learning applications one is interested in obtaining models that reliably process novel inputs. However, though deep learning models have enabled breakthroughs in multiple fields, they known be unreliable when exposed samples obtained from distribution that different from the training distribution. Usually, the larger the extent of this difference between the train and the test distributions, the more unreliable these models are. This observation has led to growing interest in developing deep learning models that can provide reliable predictions even when exposed to unseen situations (Liu et al. 2020a b. Wen et al., 202T Lakshminarayanan al.) 2017]. Most of these approaches either use expensive ensembles, propose non-trivial modifications to the neural network architectures in order obtain reliable models. most cases, they trade in-distribution performance (accuracy) to perform reliably on: (1) out-of-din out-of-disttibutionn (OOD) samples; and (2) covariate shift (CS) Caionero-Candell et al. 2009 samples.
Towards developing practically useful and reliable models, we investigate the well known Mixup [Zhang al.) 2018 it is extremely popular in improving both model's accuracy and its robustness [Wen et al. 1 Hendrycks et al.) 2020b). It has already been observed that Mixup can help in retaining good accuracy when the test inputs are affected by superficial variations that do not affect the target label (i.e., they undergo CS) (Hendrycks et al. 20206). However, we find
FP continued spending time on this project during his internship at Five Al from June 2021 to March 2022. Code available at: attps://ggt thtbb cow.:/raanceecooitt//tteei..l..e
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

that Mixup's reliability degrades significantly when exposed to completely unseen samples with potentially different labels than the ones was shown during training (OOD). This is undesirable as in such situations we would like our models to reliably reject these inputs instead of making wrong predictions on them. We observe that the primary reason for such poor OOD performance of Mixup is due to its tendency to provide high predictive entropy for almost all the samples it receives. Therefore, it becomes difficult for the model to differentiate in-distribution samples from out-of-distribtiion ones. We would like to highlight that our observation is in contrast to the prior work (Thulasidasan et al. 2019 which suggests that Mixup provides reliable uncertainty estimates for OOD data as well.
We propose a simple yet effective fix to the aforementioned issue with Mixup: we suggest to train the model on an approximate data-distribe distraiiiiibtiinn that an explicit mixture of both the empirical and the Mixup vicinal approximations to the data data-distribution. We call this approach RegMixup. In nutshell, simply combines the Empirical Risk Minimization (ERM) [Vapnik] 1991] and the Vicinal Risk Minimization (VRM) [Chapelle et al. 2000] objectives together. Implementation wise, along with the Mixup objective on the interpolated sample, requires adding an additional cross-entropy loss on one of the the clean samples. We provide proper justifications behind this proposal and show that such simple modification significantly improves the performance of Mixup on a variety of experiments.
We would like to highlight that one of the core strengths of our approach is its simplicity. As opposed to the recently proposed techniques to improve uncertainty estimation like SNGP [Liu et al. 2020a] and DUQ Ivan Amersfoort et al. [020]. it does not require any modifications to the architecture and extremely simple to implement. It also does not trade accuracy in order to improve uncertainty estimates, and is a single deterministic model, hence, extremely efficient compared to the highly competitive Deep Ensembles (DE) [Lakshminarayanan et al. 2017]. Summary of our contributions:
We provide a simple modification to Mixup that significantly improves its in-distribution, covariate shift, and out-of-distribution performances.
Through extensive experiments using ImageNet- IK, CIFARIO/100 and their various CS counterparts along with multiple OOD datasets we show that, overall, MegMixup outper- forms recent state-of- ofte--of-the-rr angle-model approaches. In most cases, it outperforms the extremely competitive and expensive DE as well.
2 RegMixup: Mixup as a regularizer
Preliminary on ERM and VRM The principle of risk minimization [Vapnik] 1991] is to estimate function f e F. that, for given loss function f(..-),, minimizes the expected risk over the data- distribution P(x.y) y). The risk to be optimized is defined as R(f) = [()(), y yx) Since the distribution P(x,y) y) is unknown, crude yet widely used approximation is to first obtain training dataset D = y;)] }=) sampled from the distribution P and then obtain f by minimizing the 1/n Di=1 {x(xx) yz). This is equivalent to approximating the entire dita-distribution space by a finite n number of Delta distributions positioned at each (x;, y.), written P. (x. y) = 1/n Σi_ &x, (x)8y. (y). This approximation to the risk minimization objective is widely known as the Empirical Risk Minimization (ERM) [Vapnik] 1991].
empirical risk defined as (f)
ERM has been successfully used in plenty of real-world applications and undoubtedly has provided efficient and accurate solutions to many learning problems. However, it straightforward to notice that the quality of such ERM solutions would rely on how closely Pe mimics the true distribution P, and also on the capacity of the function class F. In situations where the function class is extremely rich with high capacity (for example, deep neural networks), and hence prone undesirable behaviours such as overfitting and memorization, a good approximation to P is generally needed to enforce suitable inductive biases in the model. To this end, for fixed training dataset, instead of delta distribution one could potentially fit a richer distribution in the vicinity of each input-output pair to estimate more informed risk computed in a region around each sample. This is precisely the principle behind Vicinal Risk Minimization ( (VRM) (Chapelle et al. 2000]. The approximate distribution in this case be written P, y) 1/n >,= Px,y, y),), where Px,y; (x, y) denotes the user-defined vicinal distribution around the i-th sample") Therefore, the vicinal risk boils down to
Note, the original VRM paper uses Ps, dy. (y) which simply is special case of this notation.

---

Ro( = 1 72
(I)
When the integral in the above summation is intractable, Monte Carlo estimate with m samples can be used as follows:
((I()). (x,y)
{{{()))33); (x,,3,) (x,,)
Several approaches in deep learning can be seen as special instance of VRM. For exam- ple, training neural network with multiple augmentations is special case where the aug- mented inputs are the samples from the unknown vicinal distribution. widely used applica- tion of VRM is the procedure to obtain a robust base classifier to design certifiable classifiers. For example, if Px.,y, (x,y) = Px (x)oy, (y) and Px, (x) is Gaussian distribution centered at Xs, Eq. 2 can be computed by taking the average loss over gaussian per- turbed inputs x while keeping the target labels same. Min-
imizing such a risk would lead to a classifier that is robust to additive noise bounded within an & ball. This is exactly the procedure that has been widely adopted in the random- ized smoothing literature in order to obtain base neural network for which a certifiable smooth classifier can be obtained [Lecuyer et al. 2019 Cohen et al., 2019 4 Be- low we discuss another highly effective use case of VRM called Mixup which is the main focus of this work.
33
05-
04
Figure 1: Beta(a, a) pdf for varying a.
Mixup Built on the fundamentals of VRM, the vicinal distribution defined in Mixup Zhang et al. 2018 is as follows:
Pr..,. =x (x), 65, (y))),
=
where À Beta(a, a:) E 10, 1J. a E 00). X; Ax; + (1 A)x; and y. Ay: + (1 A)y Note that the vicinal distribution here not only depends on yz) but also on another input-output pair Yi) drawn from the same training dataset. For fixed a (parameter of the Beta distribution, refer Figurel 1. implementing Mixup would require taking multiple Monte Carlo sample: for each datapoint (refer Eq. (2)) which can be computationally prohibitive. Therefore, in practice, only one sample (m 1) per Beta distribution per pair of samples from batch is considered at a time. Although this procedure might look like crude approximation to the original objective, it has resulted in highly promising results in variety of applications and is very well accepted in the research community. Without undermining the remarkable success of such a simple approach, below we highlight two of its behavioural characteristics that, as we show, might limit its effectiveness:
Small cross-validated a < 1: The shape of the vicinal distribution depends on the hyper- parameter of the Beta distribution, therefore, the values of decides the strength of the convex interpolation factor X. Since high values of a would encourage À Rs 0.5 resulting in interpolated x that is perceptually different from x (inducing a mismatch between train and test distributions). the cross-validated value of a for Mixup in most cases turns out be very small (a 0.2) in order to provide good generalization. Such small values of a leads to sharp peaks at 0 and (refer Figure D]. Therefore, effectively, Mixup ends up slightly perturbing a clean sample in the direction of another sample even though the vicinal distribution has the potential to explore a much larger interpolation space. High-entropy behaviour: As mentioned earlier, m is very unlikely that the interpolation factor À E {0, 1} even for small values of a. Thus, the model never gets exposed to uninterpolated clean samples during training and hence, it always learns to predict interpolated (or smoothed) labels y for every input. Just like DNNs with cross cross-entropy loss are overconfident because of their high capacity and target
is used in practice, therefore,
BBased on our understanding, the current literature do not mention this procedure as an instance VRM. We do not investigate how depends on
3

---

(1)
(xg: Xi]
CEipus 8.).9%)
000 'n 0,97
0200 "o 0.00
Mixup
A
(6..80) 6
96p "o oTayy
CE(m((/xx) (E
A
0,200
0,500 n. 47pm
RegMixup À - Beta(10, 10)
80)
0,Glgs
0,70p
0.30g
Figure 2: Mixup vs RegMixup practice. Illustration of how the cross-validated a affects the shape of the Beta distribution in both cases. Red regions represent 80% of the probability mass. Mixup typically samples z or 1, while for RegMixup À z 0.5. In the first case, one of the two interpolating images dominates the interpolated one; in the latter, a wide variety of images containing features from both the images in the pair are obtained.
Delta distribution [Guo et al. 2017 DNNs with Mixup turns out to be relatively less confident because the network retains its high capacity but observes only target smoothed labels. This underconfident behaviour results in high-entrops for both on-distribution and dustof-distribution samples. This is undesirable as will not allow predictive uncertainty to reliably differentiate in nn-distribution samples from out-of tit-of-dies of-distribution ones.
We validate the consequences of the above limitations of Mixup with the following simple experiment. In Figure we provide heat-maps to show how the entropy of the predictive distribution varies when interpolating samples belonging different classes. The heat-map is created as follows. First, we train a WideResNNe28-10 (WRN) ZZgoruykoo and Komodakis (0116) using the CIFAR-10 10 (C10) dataset. Then, we randomly choose IK pairs of samples {x,, x,} from the dataset such that y: y For each pair, via convex combination, we synthesize 20 samples xs using equally spaced As between 0 to 1. The heat-map then created using all the 20K samples. The intensity of each (X, H) bin in the heat-map indicates the number of samples in that bin. Note, DNN (trained
with vanilla cross-entropy loss) shows low entropy (thus yielding overconfidence) irrespective of where the interpo- lated sample lies. However, Mixup shows higher entropy for almost all Äs (thus yielding lower confidence on distribution samples as well). The impact of this behaviour shown in Table [ Though Mixup provides improved accuracy compared to DNN for in-distribution (IND) and covariate shift experiments, the high entropy behaviour makes it much worse than DNN when exposed to out-of- distribution detection task. For example, when SVHN used as the OOD dataset, the performance of Mixup drops by nearly 8.47% compared to DNN. This clearly shows that the distributions of the per-sample entropies the in-distribution and -fitt--diisribution sets are harder to separate. However, there is a clear improvement of nearly 5% for covariate shift experiments, implying that Mixup augmentations do improve robustness in this aspect. Note that, in the context of calibration, Mixup's 's confidence reducing behaviour (equivalent providing higher predictive entropy) was also observed by [Wen et al. 2021
RCC
1000
45
80
05
10
Figure 3: Heatmaps of the entropy profiles as the interpolation factor À [0, 1] between samples of two classes varies. Left (DNN), Middle (Mixup). Right (RegMixup). Note, RegMixup in- duces high-entropy barrier separating in- distribution & out-distribution samples.
Note, it is highly likely that * y; even if we do not impose this constraint as the problems under consideration have many classes.

---

We now provide very simple modification to Mixup that not only alleviates its aforementioned limitation on detecting OOD samples, but also significantly improves its performance on in-distribution and covariate-shiff samples. We propose to use the following approximation to the data-distribution
RegMixup
1 72 P(x,)) y)
n2 Σ (7 (x)66, + (1 (1 (x,,))))))))))))
where Px.,y, (x.y) is the mixup vicinal distribution and Y E [0, 1] is the mixture weight. The above approximation is simply an explicit assemble of ERM and VRM based approximations to data-distribution. Though, in theory, VRM subsumes ERM, we argue that because of the crude approximations needed to prevent increasing the training cost, it does not fully realize its potential. Therefore, explicitly combining them might result in a practically more expressive model. We provide extensive experimental evidence to support this hypothesis. Implementation wise, for each sample x; in a batch, another sample x, is drawn randomly from the same batch to obtain interpolated x; and yr. and then the following loss is minimized
CE(pn(S) ys) + CE(po($) V:),
where CE(...) denotes the standard cross-entropy loss, the hyperparameter E R2o, and p(.) the softmax output of a neural network parameterized by 0. Note, dividing Eq. (3) by (1 11) is exactly the same as using a data-distribution with ς = 1/149. In practice, we observe the model's s performance to not vary much with 7) and using 1/ (equivalent y 0.5) to provide highly effective results. for Refer to [ for an overview of the RegMixup training procedure.
What does this simple modification bring the model? (1) VRM now subsumes ERM because the vicinal distribution can perfectly represent the ERM training distribution. This is because of the fact that irrespective of the values of m and o, the model will always be exposed the clean training samples as well. (2) The interpolation factor À can potentially explore a much wider space as the presence of clean samples might help in controlling the performance drop due to the train/test distribution shift. Therefore, if a < 1 was actually the most effective solution, the cross validation would automatically find it.
Practical implications of such simple modification on the behaviour and the performance of the model
Large cross-validated a >> 1: As anticipated. the model is now able to explore strong interpolations because of the additional cross-entropy term over the unperturbed training data. Interestingly, the cross-validated a that we obtained in fact is very high (a E {10, 20), see Appendix [B.3] for cross- volss-valiottion details) leading to X = 0.5. Therefore, as opposed to the standard Mixup, RegMixup prefers having strong diverse interpolations during training. Refer Figure for visualizations.
Maximizing a soft proxy to entropy (knowingly unknowing the unknown): It is straight- forward to notice that a value of À Rs 0.5 would lead to x that a heavy mix of two samples (mimicking OOD samples, refer Fig. 2 and the corresponding target label vector would have almost equal masses corresponding to the labels of the interpolating samples. There- fore, minimizing CE(pe (y) (x), y4) in this case is equivalent to maximizing soft proxy to entropy defined over the label support of y; and y, (note, exact entropy maximization would encourage equal probability masses of 0.5 for these labels). We find this observation intrigu- ing as RegMixup naturally obtains a cross-vale cross-validated a that leads to a maximum likelihood solution having high-entropy heavily interpolated samples. This is extremely desirable property as it allows models to reliably differentiate between in and out distribution samples. Thus, Mixup automatically acts as regularizer in this case.
The entropy heat-map in Figure clearly shows that as opposed to DNN and Mixup, the entropy for RegMixup is very low for À close to either 0 or 1, however, it increases and remains high for all other intermediate interpo- lation factors, practically creating an entropy barrier.
Cov, Shift C1B-C Accuracy (1)
OOD Detection
CIO (Test)
C100 AL RLROOCT (1)
SVHN AUROC (t)
T-ImageNet
Models
Accuracy
AUROC (1)
DNN
96 14
76.6
88.61
96
86.44
Mixup
93.01
81.63
83.17
87
84
RegMixup (Our)
97.46
83.13
89,63
96.72
90.19
Table 1: In-distribution, covariate shift, and out-of- distribution detection results for WRN trained CIFAR-10 (C10). C10-C is the corrupted version of C10.
Also, as shown in Table MixMixup provides improvements in both accu-
5

---

racy (in uin-distribttin and covariate-shift) and that Mixup's s performance on OOD (T-ImageNet) dropped by 2.4% compared DNN whereas Reg- Mixup performed 3.75% better than DNN. Thus, in this case, it effectively improved the performance of Mixup by nearly 6.15%.
robustness. An interesting observation
3 Related Works
Covariate-Shift Robustness It is well known that neural networks performance can severely degrade when exposed to covariate shift (Taori et al.) 2016 Recht et al., 2019 [aan et al. 2019). In image classification, for instance, covariate shift could occur due to changes in environmental conditions [Hendrycks and Dietterich 2019] and the image capturing process, data sampling (Recht
ctt aT. 2019: 2018 Lu et al.). stylised representations of the objects (e.g. in paintings or drawings) [Wang al.) [2199] Hendrycks et al. 2021] etc. In the field of duttt--iiitiiuutiin generalization, many techniques assume the availability of sets of samples presenting such variations at training time (Shi ct al., 2021 Yao et al. [2022 Wang et al., 2021], other techniques suggest leveraging augmentations to mimic a set of variatonns that would allow the model to learn features that generalize
better (Hendrycks al. 2020b a or self-supervision (Hendrycks et al. 2019a Shen et al. 2022)
Out-Or-Distribution Detection While some authors suggest introducing an OOD reject option [Geifmman and EI- Yaniv [019]. the most recent literature tends to focus on leveraging a threshold classifier on an uncertainty measure computed on top of the predictive distribution of a network. For the latter approach, it therefore convenient for a model to produce high uncertainty predictions when facing OOD samples, and low uncertainty predictions when fed in-distribution samples (Liu jet al., 2020a] Within this family of approaches, the most popular ones are the expensive Deep Ensembles (DE) [Lakshminarayanan et al. 2017]. Note, the computational cost of DE scales linearly with the number of members, and also recent works have observed a few limitations of naively using ensembles directly [Wen et al. [02] Ashukha et al. 2020). There are several efficient variants of DE where the ensemble members are obtained either by processing multiple inputs and multiple outputs at the same time [Havasi et al. 2021]. J. adding member-specific components to the network [Wen) ict al., 2020). or by finding optima in distinct basins of low loss (Huang et al. 2017a). Significant attention has also been paid to Bayesian techniques where either stochastic gradient descent is being modified to perform approximate inference [Chen et al.) 2014] Zhang et al.; 2020 Durmus et al. 2016) an approximation to Gaussian Processes is being utilized (specifically, we focus SNGP [Liu et al. 2020a)). ). or Bayesian Logistic Regression is being used during inference via Laplace approximation (specifically, we focus on KFAC-LLLA (Kristiadi et al. (2020). Techniques utilizing distance functions in the feature space have also been suggested [van Amersfoort et al. 2020
Model calibration Modern NNs have been shown to be miscalibrated [Guo al. 2017 i.e. they exhibit mismatch between the model's confidence and its accuracy. It has been observed that temperature scaling [Guo et al. [0017] or replacing the typical cross-entropy loss [Chung et al., 202T Thulasidasan et al.) 2019 Mukhoti et al.] 2020 can be highly effective to reduce this mismatch.
Also, ensemble learning has been observed to help in reducing miscalibration [Lakshminarayanan et al. 2017 Wen et al. [021 Rahaman and Thiery 2020]. However, it has been shown that good calibration on in-distribution data does not necessarly imply good calibration performance under covariate-shifi (Ovadia et al.) 2019].
4 Experiments
Datasets and Network Architectures We employ the widely used WideResNNe28-100 10 (WRN) Zagoruyko and Komodakis 2016) and ResNet50 (RN50) [He et al. 2016) architectures. We train them on CIFAR- 1O (C10) and CIFAR- 100 (C100) datasets. We employ RN50 to perform experiments on ImageNet- K (Dngg et al. 2009 dataset. We report the average of all the metrics computed on seeds. For further details about the code base and the hyperparameters, refer to Appendix B
For Covariate Shift (CS) experiments on models trained on C10 and C100, we resort to the widely used CIFAR10-C (C10-C) and CIFAR 100- (C100-C) datasets, corrupted versions of C10 and C100 [Hendrycks and Dietterich] 2019). These datasets are made by applying 15 synthetically generated but realistic corruptions degrees of intensity on the test sets of C10 and C100,

---

respectively. For C1, we also use the CIFAR-10.1 (C10.1) |Recht et al. 2018 and CIFAR- 10.2 (C10.2) (Lu al.] datasets designed to test the generalization of CIFAR- 10 classifiers to natural covariate shifts. To the best of our knowledge there is no such analogous dataset for CIFAR- 100. For ImageNet-1 1 K experiments, we use the widely considered ImageNet-A (A) [Hendryeks et al. 2019b]. ImageNet-R (R) [Hendrycks et al. (02T]. ImageNetv2 (V2) [Recht et 22.91, and ImageNet-
Sketch (SK) [Wang et al. 2019] datasets for covariate shift experiments. We report the calibration experiments both in-distribution and covariate shifted inputs in Appendix [
For Out-of-Distributtio (OOD) detection, following SNGP [Liu et al. 2020a], we use C100 and SVHN Netzer et al. [201T) as OOD datasets for models trained on CTO. Similarly, for models trained on C100, we use C10 and SVHN as OOD datasets, Additionally, we also consider the ImageNet (T-ImageNet) dataset (Le and Yang 2013) as OOD set in both the cases. For models trained on ImageNet- IK we use ImageNer-O (O) (Hendryeks et al. 2019b as the OOD dataset.
Methods considered for comparisons Besides the natural comparison of our method with Mixup (Zhang et al., 2018 and networks trained via ERM on the standard cross-entropy loss (which we will refer to as DNN), we consider several other methods from the OOD detection and CS literature. For models trained C10 and C100, we consider:
DNN-SN and DNN-SRN: taking inspiration from (Liu al.) 2020a| we consider DNN models trained with Spectral Normalization (SN) [Miyato et al. 2018a] and Stable Rank Normalization (SRN) [Sanyal al. 2020] control the bi bipsipsciit constant of the networks, which has been shown to affect the generalization properties of neural networks.
SNGP: Spectrally Normalized Gaussian Process [Liu et al. [020aa]
DUQ: Deterministic Uncertainty Quantification (van Amersfoort et al., 2020
KFAC-LLLAA KFAC-Laplace Last Layer Approximation [Kristiadi al. 2020 makes a model Bayesian at test time by taking the Laplace approximation of the last layer [Ritter et al. 2018). We provide a simple outline of this approach in Appendix AugMix [Hendrycks et al. 2020b] A data augmentation technique that applies randomized augmentations to an input while enforcing them to be consistent during training. With the recommended hyperparameters in the paper, it is almost 4x slower than DNN during training while having the same inference requirements.
DE: Deep Ensembles [Lakshminarayanan et al. 2017] with 5 members, requiring 5x more compute than most single-model approaches such as DNN.
We would like to mention that, compared to vanilla DNN, our approach (RegMixup) is almost 1.5x slower and Mixup is about 1.2x slower during training. while having the same inference requirements. Due to the high compute requirements, for ImageNet-IK we consider DNN, Mixup and the two other strongest baselines: AugMix and DE. We also cross-validate the hyperparameters on a 10% split of the test set, which is removed at test time.
Few missing experiments: Below we provide extensive experiments for proper benchmarking. The number of datasets and architectures we use lead to many combinations, a few of which we were not able to produce good results for (even after extensive hyperparameter search), hence, some of the entries in the tables are missing. For example, we could not make DUQ work on C100 as it exhibited unstable behaviour. We could not produce promising results for SNGP RN50 CIFAR experiments using their official implementation. Similarly for Aug RN50 experiments on C10 and C100. We chose not to report these suboptimal numbers. Further details can be found in Appendix B
Table entries: Bold represents the best among all the single-model approaches, and underlined represents the best among all including the expensive Deep Ensembles.
Note, we do not consider methodologies requiring access an extermal dataset during training (either for CS or OOD) ) as not only this would be an unfair comparisor"] but we believe assuming such knowledge is against the goal of this work, which is to develop models robust unknown scenarios.
"since all the methods we consider do not leverage this information. RegMixup only relies on in dn-distribution training data, just like other approaches, and makes no assumption the type of CS nor the OOD inputs.
7

---

4.1 RegMixup Improves Accuracy on In-distribution and Covariate-shift Samples
Small-scale (CIFAR) Experiments on In-distribution Data In T we compare the accuracy of various approaches on the in-distribution test sets of C10 and C100, respectively.
Clearly,
IND Accuracy
WRN28-10
RN50
RegMixup outperforms Mixup in all these experiments. In fact, RegMixup is the best performing one among all the single-model approaches.
C10 (Test)
C100 (Test)
C10 (Test Accuracy (1)
C100 (Test)
Methods
Accuracy (1)
DNN
96.14
81.58
95.19
79.19
Mixup
97.01
82.60
96.05
80.1
RegMixup (our)
97.46
83,25
96.71
81.52
DNN-SN
95.20
79:27
These improvements are non-trivial. For instance, WRN trained on C100, it outperforms DNN and Mixup by 1.67% and 0.65%, respectively. It out- performs SNGP with a significant mar- gin of 4.05%.
DNN-SRN
96.22
81.38
95.39
78.96
SNGP
95,98
79.20
DUO
94.7
KFAC-LLLA
95.21
79:41
Augois
96.40
81.10
DE (5x)
96.75
83.85
96.23
82.09
Table 2: Accuracies (%) on IND samples for mod- els trained on C10 and C100
Note how the single-model approaches specif- ically designed to provide reliable predictive uncertainty estimations (for example, SNGP, DUQ, KFAC-LLLA) underperform even compared to the vanilla DNN in terms of in-distribution accuracy. In order to provide improved uncertainty estimates (as we will soon show), they trade clean data accuracy. This type of behaviour not observed in RegMixup.
Small-scale (CIFAR) Covariate Shift Experiments For C10-C and C100-C, as typical in the literature, we report the accuracy averaged over all the corruptions and degrees of intensities in 3 It is evident that our approach produces a remarkable improvement in the average accuracy compared all the baselines (except AugMix, we discuss later why that might be the case). For instance, for C100-C WRN experiments, our method achieves an accuracy improvement of 6.9 % over DNN, of 3.86% % over DE, and of 2.45 over Mixup. Similarly, for C10-C WRN, our method achieves an improvement of almost 6.53 % over DNN, of 4.81 % over DE, and of 1.45 % over Mixup.
For natural covariate shift datasets C10.1 and C10.2 as well, RegMixup outperforms all the baselines (including AugMix). For instance, on C10.2, it obtains an improvement of 3.26% % over DNN, of 1.5% over Mixup, and of 2.46 % over the expensive DE.
Why Aug performs extraordinarily well on synthetically corrupted C10-C and C100-C but not on natural distribution shift C10.1 and C10.2? Looking at TableBoone can observe Aug- Mix's extremely good performance on the C10- C and C100-C. However, the same time, the model is underperforming with respect to Reg- Mixup on C10.1 and C10.2. Similarly, AugMix is outperformed by RegMixup on all ImageNet CS experiments (and OOD will be shown soon). This seems to suggest that although the augmentations used during the training of Aug- Mix are not exactly same as that of the corrupted test dataset, they tend benefit from synthetic forms of covariate shifts, hence the dramatic improvement these particular scenarios.
Covmriat Shift Accuracs
WRN25-10
C186-C
Methadk
DNN
76.50
90:25
75
53.54
75.15
53.31
50.07
Visup
Reghtling
92.79
61.18
91.58
M6.72
67.64
DNN-SSN
18.36
0:001
86.72
52.51
34.55
55-26
$2.96
30.5
DINN-SIES
93:58
55:24
32.54
75.40
85.61
13.49
DUO
21.6
50.4
KEAC-LLLA
78.56
00.65
86 65
52.57
75.08
89.34
$3.50
50.65
AagMic
90.02
91.6
$5.9
08.15
92.17
55.58
17.63
85.00
Table Accuracies (%) on covariate shifted sam- ples for models trained on C10 and C100.
IND Ace.
Covartate Ston Ace
0OD Det.
ImspeNet
Axe(T)
(1)
AUROC
DNN
36,00
36.1
2.76
64.53
24.72
55.97
Misup
17.15
39.5
3.20
64.38
26.34
55.54
RegMiaup (our)
77.08
2926
5.86
68.66
26,98
$7.65
AugMis
76.88
263
5691
(E(5
78.22
38.44
211
66.83
21.08
53.29
Large-scale (ImageNet- 1 K) Experiments on Table 4: ImageNet accuracies (%) on IND and CS samples, and OOD detection performance. RegMixup scales to ImageNet- and exhibits improved accuracy with respect to both Mixup, DNN, and AugMix. In particular, is 1.08 % better than DNN, 0.53% better than Mixup and 0.80% % better than AugMix. DE in this case is the best performing one.
In-distribution Data As shown in Table 4
Large-scale (ImageNet-1K)) Covariate Shift Experiments In Table 4 we report the results for common ImageNet- IK covariate-shift test sets. As it can be seen, RegMixup is either the best performing one among all the single-mon model approaches, or it is the absolute winner including DE.
8

---

WRNZE-10
RN50
CIFARIO (n-Distributionn
CIFARLOU Distriiitibtion))
CIFARIO (l-DDisribbutio)
CIFARI00 (In-Distributioo))
Ont-af-Distribution
C100
SVHN T-ImaggeNet
SVHN
T-ImageNNet
C100
SVHN
T-ImageNet
C10
SVHN
T-ImageNet
Methods
AUROC (1)
AUROC (1)
AUROC(1)
AUROC(1)
DNN
88.61
96.00
86.44
81.06
79.68
80:99
88.61
93.20
87.82
79.33
82.45
79.80
Mixup
K3.17
$7.53
$4,02
78.57
78.68
80.61
84.24
$5.40
54.89
77.02
76.86
80.35
RegMixup (our)
89.63
96.72
90.19
81.27
89.32
83.13
89.63
95.39
90.04
79.44
58.66
82.56
DNN-SN
88.56
87.71
81.10
83.45
82.26
88.19
92.46
57.55
79.20
50,78
79.90
DNN-SRN
85.51
82.41
88.82
93.54
$7.82
78.77
$2.39
79.70
SNGP
90.61
90.01
79.05
86.78
82.60
KFAC-LLLA
89.33
91.17
87.81
81.04
80.32
81.57
89.54
93.13
$8.32
79.30
$2.80
80.17
Aug-Mix
89.78
91.3
88.99
81.10
76.60 64.64
80,56
DE (5x)
21.25
97.53
89.52
93.26
85.07
33.40
91:38
96.90
90.5
51.93
85.08
82.15
Table 5:
detection results (%) for WideResNet28- 10 and ResNet50 for models trained on C10 and C100. See Appendix for the cross-validated hyperparameters.
For instance, on ImageNet-A, RegMixup performs 2.67% better than Mixup and 3.20% better than DNN. Similarly, on ImageNet-V2 it performs 1.08% better than Mixup and 1.13% % more than DNN. RegMixup also outperforms AugMix on all the considered datasets, while is outperformed by DE on ImageNet-V2 V2 (by 1.02% )) and performs competitively on ImageNet-SK.
These experiments clearly show the strong generalization of RegMixup under various in in-distribution and CS experiments. They also show that it does not trade clean data accuracy to do so.
4.2
Detection Experiments
Following the standard procedure (Liu et al. 2020a]. we report the performance in terms of AUROC for the binary classification problem between in- and out-distribution samples. The predictive uncertainty of the model is typically used to obtain these curves. Given an uncertainty measure (normally entropy, refer Appendix C for extensive discussion), it is important for models to be more uncertain on OOD samples than on in-distribution samples to be able to distinguish them accurately. This behaviour would lead to better AUROC.
We report the OOD detection results for small-scale experiments (CIFAR 10/100 in Table and for large-scale experiments (ImageNet) in Table 4 Compared to the single-model approaches, it is clear that RegMixup outperforms all the baselines in both small-scale and large-scale experiments, except only in one situation (C100 OOD, trained WRN) where SNGP outperformed it by an AUROC of 0.98. In fact, on ImageNet experiments, RegMixup outperformed all the baselines including DE. We would also like to highlight that RegMixup provides significantly better performance compared to Mixup on all these experiments. For example, the improvement is more than 9% when SVHN is treated as the OOD dataset for WRN trained on either C1O or C100 (refer Table 5). Similarly, can observe dramatic improvements on other experiments as well.
5 Conclusive Remarks
We proposed RegMMxup, an extremely simple approach that combines Mixup with the standard cross- entropy loss. We conducted wide range of experiments and showed that RegMixup significantly improved the reliability of uncertainty estimates of deep neural networks, while also provided a notable boost in the accuracy. We showed that RegMixup did not just outperform Mixup. it also outperformed most recent ttate-of-the-ant approaches in providing reliable uncertainty estimates.
We hope that our work opens possibilities to explore situations where ERM and VRM can explicitly be combined together for practical benefits. An example would be label smoothing [Müller et al.) 2019) that can be seen and the marginal distribution of the input samples (e.g., images) is approximated using deltas. In Appendix H we conduct preliminary analysis using CutMix Yun et al. 2019 and recent techniques to train Vision Transformers Dosovitskiy et al. [202]] that alternate between Mixup and CutMix. Another possible future direction would be sampling in order to ensure that the unperturbed samples as well are used during training with high probability. The observation that RegMixup ends up acting as a soft proxy to entropy maximizer for interpolated samples, a potential extension of our work regards the possibility of exploring different interpolation techniques to enforce high entropy behaviour on those regions of the input space.
instance of VRM where the vicinal distribution is over the labels
bi-modal vicinal distribution along with importance
Area Under Receiver Operating Characteristic curve.
9

---

Acknowledgements
This work is supported by the UKRI grant: Turing AI Fellowship EP/W002981/1 and EPSRC/MURI grant: EP/N019474/1. We would like to thank the Royal Academy of Engineering, FiveAl and Meta AI. Francesco Pinto's 's PhD funded by the European Space Agency (ESA). Meta AI authors are neither supported by the UKRI grants nor have any relationships whatsoever to the grant. Thanks to John Redford, Jon Sadeghi, Kemal Oksuz, Zygmunt Lenyk, Guillermo Ortiz-Jimenez, Amartya Sanyal, Pau de Jorge, and David Lopez-Paz for their valuable comments on the paper.
10

---

References
Arsenii Ashukha, Alexander Lyzhov, Dmitry Molchanov, and Dmitry Vetrov. Pitfalls of in-domain uncertainty estimation and ensembling deep learning. arXiv preprint arXiv:200..06470 2020.
Christopher M. Bishop. Pattern Recognition and Machine Learning (Information Science and Statistics). Springer-Verte Verlag, Berlin, Heidelberg. 2006. ISBN 0387310738.
Olivier Chapelle, Jason Weston, Léon Bottou, and Vladimir Vapnik. Vicinal risk minimization. In 13th International Conference on Neural Information Processing Systems, 2000.
Tianqi Chen, Emily Fox, and Carlos Guestrin. Stochastic gradient hamiltonian monte carlo. In ICML, 2014.
Youngseog Chung, Willie Neiswanger, Ian Char, and Jeff Schneider. Beyond pinball loss: Quantile methods for calibrated uncertainty quantification. In Advances in Neural Information Processing Systems, 2021.
Jeremy Cohen, Elan Rosenfeld, and Zico Kolter. Certified adversarial robustness via randomized smoothing. In ICML, 2019.
Felix Dangel, Frederik Kunstner, and Philipp Hennig. Backpack: Packing more into backprop. In International Conference on Learning Representations, 2020. URL https::/oppnrreiiee net/ Forum/id-BJIrF24tta
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: large-scale hier- archical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn. Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. ICLR, 2021.
Alain Durmus, Umut Simsekli, Eric Moulines, Roland Badeau, and Gaël RICHARD. Stochastic gradient richardson-romberg markov chain monte carlo. In D. Lee, M. Sugiyama, U. Luxburg, Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems, 2016.
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In Proceedings of the 33rd International Conference on International Conference on Machine Learning Volume 48, ICML' 16, page 1950-1059, JMLR.org, 2016.
Yonatan Geifman and Ran El- Yaniv. Selectivenet: deep neural network with an integrated reject option. In International Conference Machine Learning, pages 2151-2159, PMLR, 2019.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning Volume 70, ICML'17, page 1321-1330. JMLR.org, 2017.
Dongyoon Han, Jiwhan Kim, and Junmo Kim. Deep pyramidal residual networks. CVPR, 2017.
Marton Havasi, Rodolphe Jenatton, Stanislav Fort, Jeremiah Zhe Liu, Jasper Snoek, Balaji Lak- shminarayanan, Andrew Mingbo Dai, and Dustin Tran. Training independent subnetworks for robust prediction. In International Conference on Learning Representations, 2021. URL https: //tpereve//penreieeee net wwt/ For.torrmmndd-GoKKkkkaa
K. He, X. Zhang. S. Ren, and J. Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016. doi: 10.1 10.1109/CVPR.2216.999
Hendrycks, Basart, Mu, S Kadavath, and others. The many faces of robustness: A critical analysis of out-oof-istrrbution generalization. arXiv preprint arXiv, 2020a.
11

---

Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corrup- tions and perturbations. Proceedings of the International Conference on Learning Representations, 2019.
Dan Hendrycks, Mantas Mazeika, Saurav Kadavath, and Dawn Song. Using self-supervised learning can improve model robustness and uncertainty. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, 2019a.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. July 2019b.
Dan Hendrycks, Norman Mu, Ekin D. Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshmi- narayanan. AugMix: A simple data processing method to improve robustness and uncertainty. Proceedings of the International Conference on Learning Representations (ICLR), 2020b.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: critical analysis of out-of-distribution generalization. ICCV, 2021.
Marius Hobbhahn, Agustinus Kristiadi, and Philipp Hennig. Fast predictive uncertainty for clas- sification with bayesian deep networks, 2021. URL httpp:///peenrereew..etet net/forummid
KcImcc3j-qS
Gao Huang, Yixuan Li, Gcoff Pleiss, Zhuang Liu, John Hopcroft, and Kilian Q. Weinberger. Snapshot ensembles: Train 1, get for free, 2017a.
Gao Huang, Zhuang Liu, and Kilian Q Weinberger. Densely connected convolutional networks. CVPR, 2017b.
Durk Kingma, Tim Salimans, and Max Welling. Variational dropout and the local reparameterizzttoo trick. In Advances in Neural Information P Processing Systems, 2015.
Agustinus Kristiadi, Matthias Hein, and Philipp Hennig. Being bayesian, even just a bit, fixes overconfidence in ReLU networks. In Proceedings of the 37th ICML, pages 5436-5446, 2020.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Guyon, U. V. Luxburg, Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, 2017.
Ya Le and Xuan S. Yang. Tiny imagenet visual recognition challenge. 2015.
Mathias Lecuyer, Vaggelis Atlidakis, Roxana Geambasu, Daniel Hsu, and Suman Jana. Certified robustness adversarial examples with differential privacy. In IEEE Symposium on Security and Privacy 2019.
Jeremiah Zhe Liu, Zi Lin, Shreyas Padhy, Dustin Tran, Tania Bedrax- Weiss, and Balaji Lakshmi- narayanan. Simple and principled uncertainty estimation with deterministic deep learning via distance awareness. In NeurlPS, June 2020a.
Weitang Liu, Xiaoyun Wang, John D. Owens, and Yixuan Li. Energy- out-of-distribution detection, 2020b.
Shangyun Lu, Bradley Nott, and Aaron Olson. Harder or different? a closer look at dis- tribution shift in dataset reproduction. http://www: gaw.by..t.......... wcl. ac www.uk/bbbaajj//d12200/
101. pdf, Accessed: 2021-11-10.
Zhiyun Lu, Eugene Ie, and Fei Sha. Uncertainty estimation with infinitesimal jackknife, its distribution and mean-field approximation, 2020.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In ICLR, February 2018a.
12

---

Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In International Conference on Learning Representations, 201 8b. wwt.:ee/ooorrum.tct1BRe....
URL http:///ppppppeeeeeeee//eeeoootr
Jishnu Mukhoti, Viveka Kulharia, Amartya Sanyal, Stuart Golodetz, Philip HS Torr, and Puneet K Dokania. Calibrating deep neural networks using focal loss. In NeurIPS, 2020.
Rafael Müller, Simon Kornblith, and Geoffrey Hinton. When does label smoothing help? In H Wallach, H Larochelle, A Beygelzimer, F dAlché-Bue Buc, E Fox, and R Garnett, editors, Advances Curran Associates, Inc.,
in Neural Information Processing Systems, volume 32, pages 2019.
Yuval Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Ng. Reading digits in natural images with unsupervised feature learning. 2011.
Christine Oshorne. Statistical calibration: review. International Statistical Review / Revue Internationale de Statistique, 59(3):309-336 1991. ISSN 03067734, 17515823. URLhtty
7/www, //www.jstor. arg/stabble/443699
Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, Sculley, Sebastian Nowozin, Joshua V. Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift, 2019.
Joaquin Ceiioneo-CCandele Masashi Sugiyama, Anton Schwaighofer, and Neil D. Lawrence. Dataset Shift in Machine Learning. The MIT Press, 2009, ISBN 0262170051.
Rahul Rahaman and Alexandre H. Thiery. Uncertainty quantification and deep ensembles, 2020.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do CIFAR- 10 classifiers generalize to CIFAR-10? June 2018.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? CoRR, abs/1902.10811 2019. URL http::/arxiv. hasrgg/bb/1999 10811
Hippolyt Ritter, Aleksandar Botev, and David Barber. scalable laplace approximation for neural
networks. In International Conference on Learning Representations, 2018. URL https:// opennreviww. beet/forumiid-Skeee
Sanyal, P H S Torr, and P K Dokania. Stable rank normalization for improved generalization in neural networks and GANs. In ICLR, 2020.
Murat Sensoy, Lance Kaplan, and Melih Kandemir. Evidential deep learning to quantify classification uncertainty. In Advances in Neural Information Processing Systems, 2018
Kendrick Shen, Robbie Jones, Ananya Kumar, Sang Michael Xie, Jeff Z.. HaoChen, Tengyu Ma, and Percy Liang. Connect, not collapse: Explaining contrastive learning for unsupervised domain adaptation, 2022. URL httph:t/////// http//xxvvvvvoo... /2g0 00570
Yuge Shi, Jeffrey Seely, Philip S. Torr, N. Siddharth, Awni Hannun, Nicolas Usunier, and Gabriel Synnaeve. Gradient matching for domain generalization. arXiv preprint arXiv:2 2021.
Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, and Ludwig Schmidt. Measuring robustness to natural distribution shifts in image classification. July 2016.
Sunil Thulasidasan, Gopinath Chemnupati, Jeff Bilmes, Tanmoy Bhattacharya, and Sarah Michalak. On mixup training: Improved calibration and predictive uncertainty for deep neural networks. In Advances in Neural Information Processing Systems, 2019.
Joost van Amersfoort, Lewis Smith, Yee Whye Teh, and Yarin Gal. Uncertainty estimation using a single deep deterministic neural network. In ICML, 2020.
V. Vapnik. Principles of risk minimization for learning theory. In Proceedings of the 4th International Conference on Neural Information Processing Systems, NIPS'91, 91, page 831-838, San Francisco, CA, USA, 1991. Morgan Kaufmann Publishers Inc. ISBN 1558602224.
13

---

Haohan Wang. Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representations by penalizing local predictive power. In Advances in Neural Information Processing Systems, pages 10506-10518, 2019.
Jindong Wang, Cuiling Lan, Chang Liu, Yidong Ouyang, Wenjun Zeng, and Tao Qin. Generalizing to unseen domains: A survey domain generalization. arXiv preprint arXiv: 1xXi::2033.0307,, 2021.
Yeming Wen, Dustin Tran, and Jimmy Ba. Batchensemble: an alternative approach efficient ensemble and lifelong learning. In International Conference on Learning Representations, 2020. net .et.://ooruummnd-SSAlyyyyy.
URL htt.:tttt///pperrviiiee
Yeming Wen, Ghassen Jerfel, Rafael Muller, Michael W Dusenberry, Jasper Snoek, Balaji Lak- shminarayanan, and Dustin Tran. Combining ensembles and data augmentation can harm your calibration. In International Conference on Learning Representations, 2021. URL
https://openreevew
Ross Wightman. Pytorch image models.
https::/githuu..com//t
pytorch- ima-e- ieaiiiaaageeeeeeeee models 2019.
Huaxiu Yao, Yu Wang, Sai Li, Linjun Zhang, Weixin Liang, James Zou, and Chelsea Finn. Improving out-of-distribution robustness via selective augmentation, 2022. URL htttps://openrevies hett/forumPiddzxheelkiiii
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. ICCV, 2019.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In Edwin R. Hancock Richard C. Wilson and William A. P. Smith, editors, Proceedings the British Machine Vision Conference (BMVC), pages 87.1-87.12. BMVA Press, September 2016. ISBN 1-901725-59-6. doi: 10.5244/C. 30.87. URL https://dx doi.
Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference Learning Representations, 2018. URL https
// woppareviee..ee/foorroooooeeeeee idd.rrpp--dd
Ruqi Zhang, Chunyuan Li, Jianyi Zhang, Changyou Chen, and Andrew Gordon Wilson. Cyclical stochastic gradient meme for bayesian deep learning. In International Conference on Learning Representations, 2020. URL
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See the "Methods considered for Comparison" section
(c) Did you discuss any potential negative societal impacts of your work? [No] Not
expected
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either the supplemental material or as URL.)? [Yes] Code is on github. The appendix contains an explicative pseudocode
14

---

(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] They are specified either the Experiments section of the main paper or the Appendix
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [N/A]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Appendix
4. If you using existing assets (e.g., code, data, models) curating/releasing assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [No]
(c) Did you include any assets either in the supplemental material URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
15


Reviews:
[{"review_id": "oUm-S5Pw06y", "reviewer": "Reviewer_FwuB", "summary": "This paper presents RegMixup, which employs Mixup training as an additional regularize of the standard training. It can be treated as a special case of employing Mixup, but with a possibility of 0.5.  Some interesting findings are presented, for instance, RegMixup scales well with a beta distribution of (10, 10), and RegMixup provides high entropy barrier separating in-distribution from out-of-distribution samples. Extensive experiments are conducted on CIFAR-10, CIFAR-100, and ImageNet.  Calibration, OOD, and distribution/covariance shift results are presented. Overall, the presented method is simple yet insightful. ", "questions": "My main concerns are listed in weaknesses, especially points 1 and 2, here is one more question/suggestion:\n\n1: In Figure 2 and lines 108-109, authors claimed that Mixup employs a small value of α. But I believe many papers [1,2,3] employ beta (1,1) as the default setting. Authors may need to provide some supporting references or consider changing this claim\n\nOverall I'm happy to change my rating to Borderline Accept if authors can strongly address weaknesses 1 and 2, where weakness 2 limits the application of RegMixup and weakness 1 limits the generality. If all of my concerns are well-addressed, I'm OK to raise my rating further into Weak Accept. \n\n[1]: CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features,  ICCV 19.\n\n[2]: You Only Cut Once: Boosting Data Augmentation with a Single Cut, ICML 22.\n\n[3]: Lots of transformer papers and recent network architecture papers.", "limitations": "Limitations are discussed. Negative societal impact are not discussed. A potential negative societal impact is that the interpretability of RegMixup is still limited and hence needs to be carefully applied to sensitive applications.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1: The proposed method is simple, and can be easily used for training robust image classifiers. \n\n2: Good presentation, well-written.\n\n3: Experiments are extensive. Results are solid. \n\n\nWeaknesses:\n1: No experiments on CutMix (and other variants, but CutMix should be enough). Does RegMixup also help CutMix?\n\n2: The training of RegMixup seems to see 2x samples per iteration. Thus the running speed is slow (as authors claimed 1.5 x slower). When compared to other methods, RegMixup seeing 2x samples may lead to unfair comparison. \n\n3: Might be good to validate the performance of RegMixup on transformer and other training recipes (used in transformer models). \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "m7f_Fjti0QY", "reviewer": "Reviewer_86F1", "summary": "This paper proposes a simple change to Mixup data augmentation to also train with unperturbed images in addition to augmented images. Making this small change induces alpha cross-validation to choose stronger Mixup (closer to 50/50 image interpolation), yielding more image diversity without losing access to in-distribution samples during training. Extensive experiments show improvement in both in-distribution and out-of-distribution (OOD) accuracy, and OOD detection. This method avoids the common tradeoff of in-distribution and OOD accuracy.", "questions": "Do you have further or more concrete directions for future work in mind?", "limitations": "The authors do acknowledge a few experiments in which their method is outperformed, and partial explanations for why (e.g. AugMix).", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: The proposed method is very simple and easy to implement, as well as computationally inexpensive. It is validated with extensive experiments covering a wide set of models and datasets, and shows good results on multiple tasks.\n\nWeaknesses: The paper does not read well due to many grammatical/writing issues. My first impression was negative because of this. As an example, the word \"of\" is missing in the title (\"out of distribution\"). Please copy edit the final version.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Q79ErFcXrFw", "reviewer": "Reviewer_2EUq", "summary": "This paper proposes a simple modification of the well-known Mixup that uses the combination of vanilla training loss and Mixup training loss as the training objective. This modification helps to overcome the two issues of Mixup: 1) Mixup chooses extremely small $\\alpha$ during cross-validation and 2) Mixup can not distinguish in-distribution and out-of-distribution samples. Experiment results show that RegMix outperforms Mixup and many other baselines in in-distribution accuracy, Covariate Shift accuracy, and out-of-distribution detections.", "questions": "1. This paper states that one of the limitations of Mixup is that it uses small $\\alpha$ for good generalization. And this paper uses $\\alpha=0.3$ for CiFAR-10 & CIFAR-100, and $\\alpha=0.1$ for ImageNet. But as I know in the case of image classification, e.g., CIFAR-10, ImageNet-1k, the commonly used $\\alpha$ is 1 which leads to a uniform distribution. Could the authors provide results of these datasets under the commonly used $\\alpha$ and provide some explanation on why Mixup prefers small $\\alpha$ in this case? I believe this will provide a more solid sound. \n\n2. Since the choice of $\\alpha$ is the limitation of MIxup and benefit of RegMix, I would suggest putting the $\\alpha$ used for these two methods in the main body instead of in the Appendix.\n\n3. (Minor one) It would be better to provide the comparison of a failure case of Mixup and a success case of RegMix for OOD.", "limitations": "1. Mixup was developed in 2018 and several better mixup-based methods have been proposed, e.g. the common training pipeline of ViT now uses a combination of Mixup and CutMix. I was wondering how these methods perform against CS and OOD and if RegMix together with these new mixup-based methods can still lead to better results.\n\n2. It would be better to provide some theoretical justifications to show why RegMix can help overcome the issues of Mixup. Specifically, why Mixup prefers small $\\alpha$ but RegMix prefers a large one.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. This paper suggests a simple modification of Mixup that fixes the two issues of Mixup and leads to better CS and OOD performance.\n2. The paper is well-written. It first defines the issues of Mixup and shows how RegMix is able to overcome these issues.\n3. Extensive experiments have been carried out on various image datasets and several tasks.\n\nWeaknesses:\n1. The proposed method is not that different from the original Mixup.\n2. The proposed method is designed for the earliest mixup-based method. And we are not sure if it also works for the recent progress on mixup-based methods, e.g., CutMix, PuzzleMix.\n3. There is no theoretical justification for why RegMix works.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept