An ensemble view on mixup
Authors: David Lopez-Paz
Ishmael Belghazi
Diane Bouchacourt
Elvis Dohmatob
Badr Youbi Idrissi
Levent Sagun
Andrew M Saxe
Venue: Submitted to ICLR 2023 (2023)
Keywords: mixup
ensembles
generalization
calibration
ood
detection

Abstract: Deep ensembles are widely used to improve the generalization, calibration, uncertainty estimates and adversarial robustness of neural networks. In parallel, the data augmentation technique of mixup has grown popular for the very same reasons. Could these two techniques be related? This work suggests that both implement a similar inductive bias to “linearize” decision boundaries. We show how to obtain diverse predictions from a single mixup machine by interpolating a test instance with multiple reference points. These “mixup ensembles” are cheap: one needs to train and store one single model, as opposed to the K independent members forming a deep ensemble. Motivated by the limitations of ensembles to model uncertainty far away from the training data, we propose a variant of mixup that builds augmented examples using both random interpolations and extrapolations of examples. We evaluate the efficacy of our proposed methods across a variety of in-domain and out-domain metrics on the CIFAR-10 and CIFAR-10-NEG datasets.

Under review as a conference paper at ICLR 2023
AN ENSEMBLE VIEW ON MIXUP
Anonymous authors Paper under double-blind review
ABSTRACT
Deep ensembles are widely used to improve the generalization, calibration, uncer- tainty estimates and adversarial robustness of neural networks. In parallel, the data augmentation technique of mixup has grown popular for the very same reasons. Could these two techniques be related? This work suggests that both implement a similar inductive bias to "linearize" decision boundaries. We show how to obtain diverse predictions from single mixup machine by interpolating a test instance with multiple reference points. These "mixup ensembles" are cheap: one needs to train and store one single model, as opposed to the independent members forming deep ensemble. Motivated by the limitations of ensembles to model uncertainty far away from the training data, we propose a variant of mixup that builds augmented examples using both random interpolations and extrapolations of examples. We evaluate the efficacy of our proposed methods across a variety of in-domain and out-domain metrics the CIFAR-10 and CIFAR- 10-NEG datasets.
INTRODUCTION
In an unprecedented tour de force, deep learning has surpassed human performance on a variety of applications such image classification, speech recognition, natural language processing, and game playing (LeCun et al., 2015; Silver et al., 2016; Goodfellow et al., 2016). While this is nothing short of an incredible achievement, there is a growing interest in the research community to evaluate machine learning models beyond average accuracy (Stock and Cisse, 2018). This is because, regardless of their impressive accuracy in in-domain, deep learning models crumble under distribution shifts (Arjovsky et al., 2020; Gulrajani and Lopez-Paz, 2020), have limited ability say "I don't 't know" (Tagasovska and Lopez-Paz, 2019; Belghazi and Lopez-Paz, 2021), and are fooled by "adversarial" perturbations imperceptible to humans (Goodfellow et al., 2015). In sum, we have built machines extremely predictive under the conditions they are trained on, but unreliable under changing circumstances.
Deep ensembles (Lakshminarayanan al., 2016) are a popular technique improve the generaliza- tion (Wilson and Izmailov, 2022), calibration (Guo et al., 2017), uncertainty estimates (Belghazi and Lopez-Paz, 2021; Ovadia et al., 2019), and adversarial robustness (Pang al., 2019; Abbasi et al., 2020; Adam and Speciel, 2020) of deep neural network models. To construct a deep ensemble, practi- tioners (i) initialize collection of K neural networks random, called ensemble members; (ii) train them all to convergence on the same data; (iii) use as prediction rule the average of the outputs of the K trained neural networks. As such, deep ensembles are K times more expensive than single neural network models, both at training and evaluation time. Moreover, while the uncertainty estimates provided by deep ensembles result in better-calibrated decision boundaries in- beteeeeen classes, how to signal low confidence far away from training data remains an open question (Amersfoort et al., 2020).
On separate thread of research, the regularization technique mixup (Zhang et al., 2018) has raised in popularity for exhibiting the same benefits as deep ensembles. In nutshell, mixup trains neural network classifiers corresponding label mixtures. More specifically, deep neural networks trained with mixup achieve better generalization (Chun et al., 2020: Zhang et al., 2021a), calibration (Thulasidasan et al., 2020; Zhang et al., 2021b), uncertainty estimates (Lee et al., 2021), and adversarial robustness (Pang et al., 2020; Archambault et al., 2019; Zhang et al., 2021a; Lamb et al., 2021).
convex combinations of random pairs of examples, and the
In this work, we take the parallels above as hints to consider mixup as an implicit ensemble method. Our main intuition is summarized in Figure 1. In the zero-training-eroor case, the members of deep

---

Under review as a conference paper at ICLR 2023
ERM
Enserable
miisup
valae of laugoot sultoon
valle of largosr wifluuee
0.88
0.8-
2.2
lacpos
0.4-
6.4-
01.4
2.0
0.0
6.0
010
0.50
-0.25
1.00
0.20
0.50
0.50
-0.23
0.00
0.25
0.50
0.50
-122
0.00
0.20
0.50
4
10
contered 65 pred change
centered predi change
A centered of good chonge
8000
Ensemble
Figure 1: Deep ensembles and mixup exhibit smoother decision boundaries than ERM (first three panels). On the one hand, ensembles "linearize" their decision boundaries by averaging zero-training- error predictors that fluctuate randomly in-between training examples. On the other hand, mixup optimizes for this linear behavior explicitly during training. As we add more members to deep ensemble, the resulting predictor exhibits a lower mixup loss, while it has never been trained to minimize such statistic (fourth panel). Do ensembles and mixup improve generalization because of the same inductive bias?
ensemble fluctuate randomly in-between training examples, because each member is initialized at random. After ensembling, these random fluctuations cancel each other, smoothening (linearizing) decision boundaries, often improving generalization. This is the very inductive bias directly optimized by mixup! In fact, as we increase the number of members deep ensemble, the resulting predictor minimizes the mixup loss implicitly, each of the members is an ERM model.
Contributions our intuitions are correct, how could we leverage a single mixup machine like ensemble? We propose recipe that interpolates single test instances with multiple "reference" examples, providing a multiplicity of predictions (Section 3). mixup ensembles are times cheaper than deep ensembles, as only one machine trained and deployed. We also discuss the limitations of deep ensembles when it comes to detecting out-of-distribution examples that lay far away from the training data. Motivated by this shortcoming, we propose Ex Ex-mixup, variant of mixup that constructs augmented points allowing both the interpolation and extrapolation of pairs of training examples (Section 4). We conclude with series of numerical experiments verify the efficacy of the proposed methods (Section 5) and some concluding thoughts (Section 6).
2 BASICS ON DEEP ENSEMBLES
Let us start our exposition by providing the necessary background about deep ensembles (Lakshmi- narayanan al., 2016). Constructing deep ensemble of neural networks (Lakshminarayanan et al., 2016) is as follows:
Instantiate neural networks of equal architecture at different random initializations (O,,...,0s 0x). These neural networks, denoted by (. fex J. are referred to as the members of the ensemble.
2. Train the K neural networks via Empirical Risk Minimization (ERM) the training data.
3. Deploy the final prediction rule the average of the K neural networks f(x)
K 1 =k=1 EE fee (z). Use "one minus the largest softmax score" (1 (1- max. (=))) as measure of predictive uncertainty (Hendrycks and Gimpel, 2016; Lakshminarayanan et al., 2016),
Without loss of generality the sequel considers classification problems with C classes. Thus, ensemble members-as well the ensemble diself-output C'ddimensional softmax vectors, belonging to the C-dimensional simplex C.
When using modern neural network architectures, one assume that each of the K members in the ensemble achieves zero training error (Zhang et al., 2017). However, since the members are initialized at random, they may disagree in those regions of the input space where the labeling is noisy (high aleatoric uncertainty) or there was lack of training data (high epistemic uncertainty) (Kendall and Gal, 2017; Hüllermeier and Waegeman, 2020; Tagasovska and Lopez-Paz, 2019). Therefore,
2

---

Under review as a conference paper at ICLR 2023
ERM
Deep Ensemble
Deep Ensemble
1.00
ensemble memmber
0.99
0.98
0.97
ensemble member
Mixup
Extra- Extra-Mixup
Test'Tmme- Mixup Ensemble
1.00
ensemble member
0.99
0.98
0.97
ensemble member
Figure 2: Predictive uncertainty (darker) of the methods discussed in this work. ERM trains single machine to minimize the prediction error across training data. Deep Ensemble train K ERM models and returns their average the final prediction. mixup trains ERM model random convex interpolations of examples. Ex-mixup trains an ERM on both (i) random convex combinations of examples, and (ii) random extrapolations of examples and the uncertainty label. We observe that ERM provides sharp decision boundary with no uncertainty awareness. Deep Ensembles and mixup describe uncertainty along the decision boundary. Ex-mixup provides full account of uncertainty (awareness of being far away of the training data). The ensemble members extracted from a single mixup machine are diverse (in terms predicted label disagreement).
the ensemble exhibits some degree uncertainty awareness: when the members of the ensemble disagree, the resulting prediction should be regarded as one with low-confidence (Lakshminarayanan al., 2016; Ovadia al., 2019).
It is classic result that ensembling accurate diverse predictors improves generalization er- ror (Hansen and Salamon, 1990; Ueda and Nakano, 1996). However, the ensemble may remain confident in those regions of the space where all the members extrapolate in the same manner. This is illustrated in Figure 2, depicting the classification of two-dimensional moon shapes. While the Deep Ensemble is more uncertain along the decision boundary than plain ERM predictor, both confident as they can be as we move far away from the training data in most directions.
Deep ensembles require training and deploying K neural networks, computational and storage ask K times exceeding those of single ERM predictor. This shortcoming, combined with the advantages of using deep ensembles, motivate recent strands of research, those of MIMO networks (Havasi et al., 2021) and model soups (Wortsman et al., 2022) networks.
On the hand, MIMO networks train single classifier to jointly predict K targets from K inputs. For instance, if K = 3 the implemented classifier once, and predicts the corresponding three labels jointly. Architecturally, MIMO networks require K input and K output linear layers. Thus, the number of total parameters in MIMO network increases ResNet18 ImageNet classifier. When deployed, MIMO networks can provide K predictions about single test instance, which compete in diversity those provided by deep ensembles (Havasi al., 2021).
= f(x1, I2, .z)) accepts three instances
with K for instance, a 48% increase for K
On the other hand, model soups train K independent models (fo,...., fox of equal architecture but different con-architectural hyper-parameterr (such as random initialization, learning rate, weight decay). After these K models are trained, we construct single model. the "soup" f = Σ fo,- by averaging the optimized parameters of the K trained models. Then, building model soup requires
3

---

Under review as a conference paper at ICLR 2023
Table 1: Summary of different learning methods, together with the number of machines necessary be trained (#tr) and deployed (#te), number of predictions available per test instance (#preds), and level of uncertainty awareness. For each method we list the formula obtain the k-th prediction pk about a single test instance x.
method
prediction Ps:
#preds
uncertainty
FRM
fo(a)
no
Mixup
idem
1
in-domain (in-out)-domain
Extra-Mixup
idem
I
TestTime-Mixin
(folder + (1 Au)oe) Ax)
K
in-domain
Deep Ensemble
Sa (x)
K
in-domain
MIMO
O(K)
O(K)
K
in-domain
MC-Dropout
foo (2)
K
in-domain
Soup
{ IE (a)
1
no
training K networks and deploying one. However, a model soup can only provide single prediction per test instance. Table summarizes the discussed methods so far.
EXTRACTING ENSEMBLES FROM SINGLE MIXUP MACHINES
The mixup data augmentation technique (Zhang et al., 2018) trains classifiers random convex combinations (#. ÿ) of training examples:
(z, v) = mixup(x,, Vis Tyy V/ Wy,A) == (A H, +
(1 - æj. (1.
Uj).
In the previous, (xr,, y/) and (ry, My) are a random pair of training examples, and À ~ Beta(a, a) is a random mixing coefficient. The mixup parameter controls the strength of mixing, recovering the vanilla ERM training as a - O. Once deployed, the mixup machine evaluated normally at test instances. Therefore, mixup trains and deploys single machine, with the capability of providing single prediction per test instance.
mixup training has gained traction due improving generalization error (Chun et al., 2020; Zhang et al., 2021a), uncertainty estimates (Thulasidasan et al., 2020; Zhang et al., 2021b; Lee et al., 2021), 1), and adversarial robustness (Pang et al., 2020; Archambault et al., 2019; Zhang al., 2021a; Lamb et al., 2021). Figure shows how mixup produces smooth, well-calibrated decision boundary in wwo-dimensional classification problem. As stated in the introduction, these are the same suspected reasons behind the success of deep ensembles. Could these two techniques be related?
In the following, we outline simple procedure leverage single mixup machine to produce K predictions about a single test instance. To this end, consider a small collection of K reference points {(r, { These could be training or held-out examples or, when adversarial robustness is a worry, kept secret during deployment. By virtue of the mixup condition, the machine trained with mixup should satisfy [(Axar + > Jaa) A (x:) (1 >x) (xx), where As ~ Beta(a, a)) uses the same mixing hyper- hyper- parameter > 0 as in training time. By rearranging, we use the reference point 2y as an "anchor" to obtain the prediction fx(x) about the test instance a, namely:
f(x) ~0 l()xx (1 Aa) )))) (1 \x) =: fx(
we are now able to construct an ensemble of predictions (J(())..... JRk [x())) about using single mixup machine. To produce valid simplex vector in AC, we perform only softmax operation as the last step in Equation (1), In practice, we choose the reference points by running k-kmeans feature space, and selecting the raw training samples closest to each centroid. We call this TestTime Time-mixup mixup (TT-mixup) ensemble of K members.
Repeating with all reference points {Ix 1K
The mixing coefficients À offer an opportunity maaximum-a-posteriin accuracy (Aę + 1). Figure 2 shows that the member predictions of TT-mixup are diverse, even more than the ones provided by the expensive Deep ensemble.
trade-off prediction diversity (Ax + 0) and
We found this process was briefly mentioned in unpuhlished manuscript (I.ce al., 2021).
4

---

Under review as a conference paper at ICLR 2023
MIMO versus mixup One can view MIMO (wlog K 2) as a generalization of mixup, where on trains a classifier on "mixed" data (ž. y) = ((z,, T2), (3/. B2)). While mixup fixes the construction of mixed data as random convex combination, MIMO employs the more general concatenation operator to construct augmented examples, delegating their mixing to the first convolutional layer. While MIMO's generality comes the expense of additional parameters, the retrieval of its member predictions f& (x) = f(r, r)& is more transparent. These and other techniques fall under common framework of "learning from multiple examples".
4 WHAT HAPPENS OUT-OOF-DISTRIBUIONN
Given the importance of out-of-distribution (ood) performance and uncertainty estimation in our enquire, cannot help but notice in Figure 2 worrisome behaviour for ERM, Deep Ensemble, and mixup machines. The machines produced by these three training algorithms are maximally confident about their predictions as we move far away from training data in most directions (the confidence turns white away from the center). This is an niggling limitation in safety-criticel applications (Kendall and Gal, 2017), where we would desire that our machine learning systems signal low-confidence and abstain from prediction in those extreme regions of the input space.
This discussion is an instance of striving for machine learning methods able to model both aleatoric and epistemic uncertainty (Tagasovska and Lopez-Paz, 2019). On the one hand, we observe aleatoric uncertainty those regions of the input space where the true labeling mechanism is noisy. In Figure 2. this corresponds to the decision boundary separating the two overlapping classes-no matter how much additional data we collect, predicting close to the boundary is prone to some irreducible uncertainty. On the other hand, we should observe high epistemic uncertainty in those regions of the space where we have not observed any training data. Therefore, epistemic uncertainty is inversely proportional to the input density of the training distribution. In Figure epistemic uncertainty should increase (color should darken) as we move far away from the center in any direction.
To address this issue, extend mixup training to construct augmented data not only interpolating. but also extrapolating, random pairs of training examples. More specifically, we consider training our classifier of choice on mixed examples built as:
mmxupp(x (z,, Bi, Izz Us. A) mixxp((x;, Di, 2x; Tzz A) with equal probability.
or
(a. 0)
(2)
Like in mixup, in the previous equation (I,, and (Iy, My) are a random pair of training examples, X ~ Beta(a, a:) is a random mixing coefficient, and = (1/C...., ........11 1/C) signals highest uncertainty.
The first case of the augmentation protocol (2) is vanilla mixup, where we interpolate between random pairs of instances and their labels. The second case of (2) allows extrapolate "over-shoot" beyond the first example x, of the selected random pair. When this happens, the accompanying label an interpolation between the label associated to the first selected example x; and the highest-uncertainty label u. Intuitively, we are providing signal to the machine as to become uncertain outside of the convex hull described by the training data. We call this training scheme Ex-mixup-vl. Figure illustrates the impact of Ex-mixup-vl training on the confidence surface of the resulting classifier. Now, we observe increasing uncertainty anywhere but where the training data lives. This captures both alcatoric uncertainty in-between classes) and epistemic uncertainty (far away from data).
4.1 A SECOND VERSION EX-MIXUP
In Ex-mixup-vl, choosing the highest uncertainty label to be = (1/C...., 1/C) may lead to unpredictable behavior when moving very far away from the training data. particular, since the value = (I/C,..., .., 1/C') is non-saturated softmax vector, the output of the network is free to drift to a saturated softmax vector far-away from the training data. To address this issue, our experiments also implement second version of Ex-mixup, termed Ex- Ex-mixup-v2.
Ex-mixup-v2 v2 instantiates an additional, dedicated (C + 1)-th uncertainty label. Therefore, the labels of normal training points y4 are extended to be y = (ya. 0) A++1 and the highest-uncertaen label is set to u = (O.... 0, 1) E CC+ Figure 3 shows the different behavior of Ex-mixup-v and Ex ex-mixup-y2 in the two moons problem when moving far away from the training data.
5

---

Under review as a conference paper at ICLR 2023
mixup
Extra-mixup-vl
Extra-mixup-v2
Figure 3: Comparison of uncertainty landscapes (darker is less confident) for binary classification problem as provided by mixup. Ex-mixup-vl, and Ex-mixup-v2. Mixup provides limited uncertainty awareness, orthogonal mixup-v1 is more uncertainty-aware but the use of the softmax non-saturating value (1 5) to signal uncertainty results in unpredictable confident far away from the training data. Ex-mixup-v2 fixes this issue by extending the problem to three classes, where high-uncertainty extrapolations are labeled using the softmax saturating value (0, 0, 1).
the 1-dimensional manifold described by the decision boundary. Ex-
Lastly, Ex-mixup-v2 allows the separate modeling of aleatoric and epistemic uncertainty. On the one hand, the (C' 1)-th softmax score signals epistemic uncertainty. On the other hand, the entropy of the first C logits signals aleatoric uncertainty. When the (C + 1)tth score equals one, we are in situation of maximum epistemic uncertainty and null aleatoric uncertainty. When the (C' 1) 1- score is zero, there is null epistemic uncertainty. and an alcatoric uncertainty proportional to the entropy the first logits.
EXPERIMENTS
We conduct experiments based on the CIFAR-10 dataset (Krizhevsky, 2009). In addition to the original training (tr) and testing (tc) splits, we add the CIFAR-10-Neg dataset (Kalpun et al., 2022) as third split (ne), containing difficult examples. We divide each split into in-domain data (first five classes, accuracy metrics) and out-domain data (second five classes, ood metrics).
5.1
METHODS
We implement six learning techniques:
ERM trains a single machine to minimize prediction error across training examples.
TT-Aug trains single ERM machine; however, during evaluation time it acts an ensemble by evaluating different random data augmentations of the test instance.
Ensemble trains K ERM predictors, each initialized at random.
mixup trains single ERM machine mixed data.
TT-mixup trains single mixup machine; however, during evaluation time it acts an ensemble by using the interpolation recipe from Section 3, setting Å 0.8.
Ex-mixup (-v1,-v2 trains single machine extrapolated mixed data (Section 4).
All methods are optimized for 500 epochs, batch size of 128, and SGD with learning rate of 0.1, momentum of 0.9, and a weight decay 10-1 The learning rate divided by ten every 150 epochs. Each method is trained from 100 random initializations and data orderings. The number of members in ensembles (as well as the number of evaluations in TT-mixup) is K 10.
5.2 METRICS
We measure variety of metrics evaluate generalization, robustness, and ensemble diversity.

---

Under review as a conference paper at ICLR 2023
Table 2: Main results for all methods on CIFAR datasets. Best results at te/ne splits in orange For all methods, we evaluate classification performance (avg acc), adversarial robustness (adv acc), ood detection (OOD acc), loss raw data (loss), violation of the mixup condition (mixup loss), and expected calibration error (ECE). For ensembles, we also report diversity in-domain (diversity) and out-domain (ood diversity).
method
ERM
Ensemble
mixup
Ex-mixup-vl
Ex-mixup-12
TT-mixup
TT-Ang
avg ace (ir)
1,00 + 0.0001
1.000 0,000
1,000 +: 0,000
0,998 : 0.0022
0.999 = 0.001
1.000 + 0.000
1,000 1: 0.000
avg ace (tc)
0.91 = 0.007
0.958 = 0.001
0.955 + 0.006
0.951 = 0.006
0.951 = 0.006
0.959 = 0.003
0.951 + 0.003
ace (nc)
0.657 + 0.014
0.686
0.007
0.686 + 0.016
0.681 + 0.017
0.685 0.016
0.721 + 0.011
0.674 0.014
worst acc (tr)
1.000 + 0.000
1,000 0.000
0.099
0.907
0.003
0.908
0.002
1.000
0.000
1,000
worst ace (te)
0.909 1: 0.011
0,930
0.002
0.930 1 0.009
0.919 1 0.011
0.919 1 0.011
0.931 1. 0.004
0,918 0.005
werst ace (me)
0.462 =:
0.018
0,405 0.040
0.045
0.509
0.044
0.507
0.033
adv ace (tr)
0.374 + 0.031
0.528
0.012
0.001 + 0.042
0.004 + 0.061
0.562 + 0.073
0.579 + 0.022
0.461 + 0.024
adv ace (tc)
0.336 0. 0.028
0.018
0.589
0.521 0.048
0.189
D
0.500
0.018
0,409 0.021
adv acc (nc)
0.060 + 0.013
0.011 0.141
091 + 0.026
0.130 + 0.028
0.139 0.030
0.141 + 0.014
0. 102 + 0.008
ood acc (tr)
03.845 0.010
0.002
0.785
0,788 0.025
0.612 D.
0.823 0.000
0.879 0.009
ood acc (tc)
0.761 = 0.011
0.791
+ 0.003
0.731 + 0.009
0.732 = 0.022
0.596 = 0.089
0.757 = 008
0.783 + 0.009
and (ne)
0.444 0.012
0.438
0.003
0.482 + 0.012
0.470 0.017
0.489 0.030
0.465 0.006
0.447 0.009
diversity (tr)
0.975 0.000
0.966 + 0.002
0.975 0.000
diversity (le)
0,934 0.002
0:947 0.002
0.945 0.01/2
OOD diversity (tr)
diversity (me)
N/A
0.785 0.001
0,810 0,001
N/A
0.876 + 0.009
0.876 0. 008
0,854 + O. 0.004L
0.850 1: 0.OISS
0OD diversity (te)
0.810 0.004
0.875 0.007
0.854 0.003
OOD diversity (ne)
0.825 0.003
0.893 1: 0.007
0.867 1. 0.004
data loss (tr)
0.000 = 0. 0.000
0.000 0.000
0.139 + 0.021
0.306 + 0.008
0.093 0.070
0.111 0.021
0.000 0.000
data loss (te)
01.008 = 0.039
0.138 1 0.0112
0,240 1: 0.022
0.396
: 0,092
0.230 = 0.075
0.204 + 0..224
0,186 1: 0.011
loss (nc)
2.146
0.032
0,818
0,911
1.063
0.768 0.028
1,405
mixup loss (tr)
2.226 1. 0.086
1,708
0.038
0.661
0.021
0.799 1 0.054
1.021 1 0.128
0.712 0.037
1.837 0.069
mixup
2:442
1.723
0.130
0.036
0.063
1.101
0.028
2,045
mixup loss (nc)
2.780 + 0. 288
1.173
+ 0.180
0.963 + 0.067
1.074
0.080
1.313 = 0.182
0.930 = 0.057
1.926 :: 0.277
ECB (tr)
0.000 = 0.000
0.000
0.000
0.127
0.017
0.257
0.009
0.084
0.103 0.021
0,000 0.000
ECB (te)
0.045 + 0.006
0.010
+ 0.002
0.106 + 0.017
0.228
0.065
0.008 + 0.052
0.094 + 0.021
0.020 + 0.001
ECB (ne)
0.297 1. 0.013
0,158 0.007
0,103
0.022
0.088
0.025
0.140 0.038
0.064 0.018
0,250 0.014
Classification accuracy (avg acc) percentage of in-domain images correctly classified.
Accuracy on adversarial examples (adv acc) measures the percentage of adversarially corrupted in-domain images that are correctly classified. The adversarial examples are produced using the fast sign gradient method with & = 0.1 (Goodfellow et al., 2015).
Accuracy at ood detection (ood acc) measures the binary classification accuracy on the task of distinguishing between in-domain and out-domain images. An image is considered out-domain if the entropy of the ensemble exceeds its 95th percentile (as computed across the training data).
Cross entropy loss on raw data (data loss).
Mixup violation (mixup loss) measures the cross-entropy loss on mixup examples.
Expected calibration error (ECE), which is minimized when the classifier is wrong (100 x p)% of the times that it predicts a softmax probability of p (Guo et al., 2017).
Ensemble diversity (diversity) measures the average label prediction disagreement between pairs of ensemble members, across in-domain images.
Ensemble diversity ood (diversity ood) analogous the above, for out-domain images.
5.3 RESULTS
Table 2 summarizes our main results. Overall, mixup methods outperform ERM ensembles for most performance metrics, except for ood acc (te) and ECE (te). mixup is the overall best alternative when it comes to (average, worst, adversarial) accuracy, with TT-mixup taking the lead on the (ne) split for average and worst accuracy. We observe that the diversity between member predictions is similar for Ensembles and TT-mixup. When it comes to the proposed Ex-mixup alternatives, none of them seem provide a worthy edge over vanilla or TT-mixup. This may be related to the fact that
7

---

Under review as a conference paper at ICLR 2023
0.-
8%% 0t: (60)
RRM
ECE [te)
eett sxe
66-
EXS (enstabk)
0.9
0.22
6.1
0.1
Tou Etseecrtio
6.3
0.00
1 ood 100000
Aani voe (te)
wherop lons (te)
misup loss (ts)
11:00
0.5
8/ 0x [ue)
0. 200
Eee
6:70
6.16
0.650
0:30
mixep line (tx)
mibepp lona (ts)
minsu loss (bo)
minop loos (ts)
Figure 4: Relationship between mixup loss in-domain (split te) and performance metries on the te/ne splits. Vanilla and TT-mixup outperform ERM and Ensembles.
"learning in high dimension always amounts to extrapolation" (Balestriero et al., 2021) (in contrast to the optimistic intuitions provided by Figure 3).
One interesting remark that Ensembles respect the mixup condition much better than single ERMs. While we lack formal justification, this is likely because the averaging of several functions initialized at random results in linearizing fluctuations in- betweeeen training points (where they all match)- the exact criterion explicitly enforced by mixup (recall Figure 1). Thus, it may be that Ensembles and mixup are tightly related in the way they improve in-domain and out-domain performance. In sum, seems like most benefits from Ensemble methods be inherited by single mixup machine used as-is, employing the TT--mxup protocol for improved ood detection and ECE.
Some ablations First, increasing the number of ensemble members A is beneficial for both methods and metrics except adversarial accuracy. Second, we observe that À = 0.8 is good trade-off when using the TT-mixup protocol. Third, there is benefit in adversarial accuracy by keeping the reference points secret (forcing the adversary to use a different set of reference points). Fourth, combining TT-mixup with TT-Aug does induce a consistent performance gain across metrics. Finally, Figure shows the relationship between mixup loss and four metrics on the te/ne splits. This illustrates the strong relationship between how much the mixup satisfied and different in-domaín and out-domain performance metrics.
Going beyond simple averaging While TT-Mixup also develop variants that uses a weighted sum of the members. For each member, we compute its weight the sum of its cross-entropy with each of the other members' predictions. Then the member's weight is computed as a softmax over the negative of the summed cross-entropies, with temperature parameter. As the temperature parameter increases, this converges a uniform distibution over each member, i.c. taking the average as in TT-Mixup. This can be viewed as using the Maximum Expected Utility principle as in Premachandran al. (2014), where the member's expected utility (or negative expected cross-entropy) is computed using the other members' predictions. Results are shown in Table 3 for the main accuracy metries. As expected, a temperature 10 gets results close to regular TT-Mixup, with 1,4% improvement the (ne) split worst accuracy (0.521 versus 0.507 for Tese Mixup). As the temperature decreases, the strategy works better on the split ood (ne) accuracy, matching mixup performance (0.486 versus 0.482), but at the expense of ood (tr) and (te) accuracies. Nevertheless, low temperatures e.g. a value 0.5 performs on par or outperform Mixup on the main accuracies reported.
average of the different members,
8

---

Under review as a conference paper at ICLR 2023
Table 3: Weighted sum of member with TT-Mixup, for different temperature parameters. We fix À 0.8 and k = 20.
temperature
0.1
0.5
1.0
10.0
avg acc (tr)
1.000 + 0.000
1.000 = 0.000
1.000 = 0.000
1.000 + 0.000
avg acc (te)
0.958 = 0.003
0.958 0.003
0.958 + 0.003
avg acc (ne)
0.722 = 0.013
1.200
0.724 0.012
0.010
worst (tr)
0.999 = 0.000
0.999 0.000
0.999 = 0.000
0.999
worst acc (te)
0.925 0.010
0.926 + 0.006
0.927 0.007
0.927 0.007
worst (ne)
0.527 + 0.040
0.525 + 0.028
0.527 + 0.042
0.521
0.038
ood acc (tr)
0.768 0.009
0.787 = 0.008
0.792 0.008
0.815 0.005
ood acc (te)
0.723 = 0.009
0.735 = 0.007
0.737 0.007
0.751 0.006
ood acc (ne)
0.486 + 0.012
0.482 = 0.010
0.477 = 0.008
0.466 + 0.007
CONCLUSION
We have offered two ensemble perspectives on mixup. On the one hand, we have introduced TT- mixup, test-time protocol extract K diverse predictions about single test instance from single mixup machine. This enables the benefits of the widely used deep ensembles at K times less training and deployment cost. On the other hand, have introduced Ex-mixup, novel mixup augmentation protocol to foster better ood awareness far away from the training data. While Ex-mixup does not seem to offer any competitive advantage in our experiments. TT-mixup improves both in-domain and out-domain performance. Interestingly, our implementation of TT-mixup pays no attention the selection of mixing coefficients (fixed X 0.8). These are promising venues to improve TT-mixup and bring its performance even closer to the gold standard of deep ensembles. sum, we hope that this work sheds some light to the inner workings of mixup, and sparks further investigation on how single classifiers can be better leveraged during deployment for increased performance, robustness, and uncertainty awareness.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Mahdieh Abbasi, Arezoo Rajabi, Christian Gagne, and Rakesh B. Bobba. Toward adversarial robustness by diversity in an ensemble of specialized deep neural networks. arXiv, 2020. URL httpps:/aaai /arxiii org/ wwt///gg//////o///e//.///...//// 2205 08321.
George Adam and Romain Speciel. Evaluating ensemble robustness against adversarial attacks. arXiv, 2020. URL
http///wwwi...r//a////..../. 05750.
Joost van Amersfoort, Lewis Smith, Yee Whye Teh, and Yarin Gal. Uncertainty estimation using single deep deterministic neural network. arXiv, 2020. URL 2003. 02037.
wrgg aabs/
Guillaume P. Archambault, Yongyi Mao, Hongyu Guo, and Richong Zhang. Mixup as directional adversarial training. arXiv, 2019, URL http::s///axivi.o/ abs/1906 06875.
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv, 2020. URL https: /ttpaaawww..gaaaaaa/aa/aaaaa 028933.
Randall Balestriero, Jerome Pesenti, and Yann LeCun. Learning in High Dimension Always Amounts to Extrapolation. arXiv, 2021. URL U
2110.100..0 09485.
Mohamed Ishmael Belghazi and David Lopez-Paz. What classifiers know what they don't? arXiv, 2021.
06217.
Sanghyuk Chun, Seong Joon Oh, Sangdoo Yun, Dongyoon Han, Junsuk Choe, and Youngjoon Yoo. An empirical evaluation on robustness and uncertainty of regularization methods. arXiv, 2020.
URL /ttps://arxiv.cor org/
03879.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016. URL org.
http::/www.deeplleamiinggg...
lan Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv, 2015. URI. htt.tt:/// arxiv. orgg/bbs///11.....
Ishaan Guirajani and David Pape In search of lost domain generalization. arXiv, 2020. URL pectps://arxiae wwwp//tg////./ooo.../.........e/ 014044........
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural wrw.v.orggaaao htbb////6 04599.
networks. arXivv, 2017. URL
Lars Kay Hansen and Peter Salamon. Neural network ensembles. IEEE Transactions Pattern Anal- ysis and Machine Intelligence, 1990. URRLhttps://leeeppattt iete.
eee. doc.mentt
58871.
Marton Havasi, Rodolphe Jenatron, Stanislav Fort, Jeremiah Zhe Liu, Jasper Snoek, Balaji Laksh- minarayanan, Andrew M. Dai, and Dustin Tran. Training independent subnetworks for robust prediction. arXiv, 2021. URL https: lttpa/aai...rraa///a/aaoa 06610.
Dan Hendrycks and Kevin Gimpel. baseline for detecting misclassified and oft-of-iistiibttion examples in neural networks. arXiv, 2016. URL / al610.
1610. 02136. 02136.23366..
Eyke Hüllermeier and Willem Waegeman. Aleatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods. arXiv, 2020. URL https: org/ /abs/ 9910.99457
Gal Kalpun, Nikhil Ghosh, Saurabh Garg, Boaz Barak, and Preetum Nakkiran. Deconstructing distributions: A pointwise framework of learning. arXiv, 2022. URL WWRLhttss://axivv.oom
09931.
Alex Kendall and Yarin Gal. What uncertainties do need in bayesian deep learning for computer vision? arXiv, 2017. URL https::////. / aariiiiiv /ttp//aww.....aaaaa/a/aa// 100./7103.0000 04977.
Alex Krizhevsky. The cifar- dataset, 2009. URL https: www.nto..ctoooooooeoooo - c cifar far. htmll.
10

---

Under review as a conference paper at ICLR 2023
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles. arXiv, 2016. URL http:::: /ttptpps///xxxv.ooo/// hos/1612.01474
Alex Lamb, Vikas Verma, Kenji Kawaguchi, Savya Khosla, Juho Kannala, and Yoshua Bengio. Interpolated adversarial training: Achieving robust neural networks without sacrificing too much accuracy. arXiv, 2021. URL httttt//a//atttt http://awwvv.......a.//o
wrw.orr//aao.. attp///s..1..././///// 06784.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning, Nature, 2015. URL https:
// www.r. com/ htnm//aticles../eeee htt.lles/nature1453333
Hansang Lee, Haeil Lee, Helen Hong, and Junmo Kim. Test-time mixup augmentation for uncertainty estimation in skin lesion diagnosis. MIDL, 2021. URL htttps://openrevieww forum??
id=aGfL5C9WRx
Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, D Sculley, Sebastian Nowozin, Joshua V. Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. arXiv, 2019. URL 1906. 02530.
/bs/
Tiayyu Pang, Kun Xu, Chao Du, Ning Chen, and Jun Zhu. Improving adversarial robustness via promoting ensemble diversity. arXiv, 2019. URL https: https://arxiv..rrrt www.:oogg///a.aa....oo....s.aa/a hatpg//as/11990 1901. 08846.
Tianyu Pang, Kun Xu, and Jun Zhu. Mixup inference: Better exploiting mixup to defend adversarial org http//b.//11111111
attacks. arXiv, 2020. URL
Vittal Premachandran, Daniel Tarlow, and Dhruv Batra. Empirical minimum bayes risk prediction: How to extract an extra few In CVPR, 2014. URL https: http://ooooooooooo../////.../oo 11t1110..11111111
2014.137.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre,, George Van Den Dricssche, Julian Schritwieser, loannis Antonoglou, Veda Panneershelvam,, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. Nature, 2016. URL https: /www. nature. wwm/ aatiaaatiilssssee https//ntueeeeeeeneeeee
Pierre Stock and Moustapha Cisse. Convnets and imagenet beyon accuracy: Understanding mistakes 11443.
and uncovering biases. arXiv, 2018. URL https
Natasa Tagasovska and David Lopez-Paz. mongle-model uncertainties for deep learning. arXiv, 2019. URL http://axvvvvv. U
00908.
Sunil Thulasidasan, Gopinath Chennupati. Jeff Bilmes, Tanmoy Bhattacharya, and Sarah Michalak. On mixup training: Improved calibration and predictive uncertainty for deep neural networks. arXiv, 2020. URL https: http://awxxv...aaa...aaoo 110//9905000 11001.
Naonori Ueda and Ryohei Nakano. Generalization error of ensemble estimators. International Con- ference Neural Networks, 1996. URL htw.t:/t////// /ttettss://eeexppppe http://eeeeeprrreee// ieee. oee.oorg//ccummtttt
548872.
Andrew Gordon Wilson and Pavel Izmailov. Bayesian deep learning and probabilistic perspective of generalization. arXiv, 2022. URL htt.tttps///xxiiii org/ 2002 08791.
Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S. Morcos, Hongseok Namkoong. Ali Farhadi, Yair Carmon, Simon Korbblith, and Ludwig Schmidt. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. arXiv, 2022. URL https: /tte:///rrxii att.iiiiiivv///ii///aaa./aa.iiia org/abs httpg//ab///000/ 05482.
Chiyuan Zhang, Samy Bengio. Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. arXiv, 2017. URL abs/ /1s/16611 03530.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Papez-Paz. mixup: Beyond empirical risk minimization. In ICLR, 2018. URL htt.tppss//e.xi...o///aa//////// 09412.
11

---

Under review as a conference paper at ICLR 2023
Linjun Zhang, Zhun Deng, Kenji Kawaguchi, Amirata Ghorbani, and James Zou. How does mixup help with robustness and generalization? arXiv, 2021a. URL https://arxiv..oo abs/ 2010. 04819.
Linjun Zhang, Zhun Deng, Kenji Kawaguchi, and James Zou. When and how mixup improves calibration. arXiv, 2021b. URL https: http:///axvvvvvooo wwg.oog///aaooo /tgg/as//////aa//aa//g/////aaerr 2202.0100002 06289.
12


Reviews:
[{"review_id": "mtf2jZljeu", "reviewer": "Reviewer_1zoz", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper relates mixup to ensemble and conjectures that mixup works as an implicit ensemble, which could potentially explain the benefits of mixup, e.g., improving generalization, robustness, etc. Based on this conjecture, the authors proposed two variants of mixup: TT-mixup which improves training and test efficiency upon ensemble, and Ex-mixup to potentially increase OOD accuracy and detection. Experiments were done on CIFAR-10 to evaluate the performances of the proposed methods.", "summary_of_the_review": "I tend to vote for rejecting this paper. Although the idea of connecting mixup with ensemble is interesting, the justifications for their proposed methods are somewhat unclear, and these methods do not perform consistently better than previous ones in the experiments. The clarity of this paper also needs improvement.", "strength_and_weaknesses": "-Strengths:\n\n1. The relationship between mixup and ensemble is interesting and novel, and has the potential to explain why mixup works.\n\n2. The authors proposed several versions of their two mixup variants and did ablation studies, which could give more insights into the use of mixup variants.\n\n-Weaknesses:\n\n1. The relationship between ensemble and mixup and the two proposed mixup variants might lack enough justification. I will explain these three points separately below:\n\n1.1 For the relationship between ensemble and mixup, the authors claimed in the paper that both of them encourage the model to have some linear behavior. However, the trained neural networks using these methods are always still highly non-linear except on very simple datasets. Besides, it seems not clear from the paper why this linear behavior is related to the performance (e.g., generalization, robustness) of the models trained.\n\n1.2 For TT-mixup, the reference points are selected by running $K$-means on feature space, and it might be unclear how to choose $K$ and why the cluster centers are good candidates for the \"anchors\".\n\n1.3 As for Ex-mixup, extrapolation and interpolation are two ways to encourage linearity of model output. However, the authors treat the extrapolation points as out of distribution, which might not be always the case for real datasets. The selection of parameters also seems not well-justified, e.g., $2x_i-x_j$ for the extrapolation and setting the additional uncertainty entry to 1 for extrapolated data points.\n\n2. The experimental results are mixed (different models are good at different tasks and there doesn't seem to be a very clear pattern), and the proposed methods don't show a clear advantage over existing results. Moreover, the results are only on CIFAR-10 and its variants, which might make the conclusions somewhat limited.\n\n3. This paper is not written in a very clear way, and there are some places that are hard to understand. The detail about this will be provided in the Clarity part in the next section.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "k1iJc5OZXI4", "reviewer": "Reviewer_S9F4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper first analyzes the relationship between ensemble and mixup and empirically shows that both have some similarities in terms of their decision boundary --- there is some uncertainty region for mixup and ensemble, that is not seen in vanilla training. Then, the paper proposes a Test-Time mixup technique that allows mixup-trained models to simulate ensemble and generate multiple predictions for a single instance, which is then averaged. Finally, the paper talks about how mixup works in out of distribution generalization, and proposes to extrapolate the data instead of interpolating. Two methods are proposed, both extrapolating the input but for the labels one interpolating with an uniform distribution over all classes and another an one-hot on an inserted new dimension. The paper analyzed the mixup variants proposed and ensemble and showed that (1) the proposed Test-Time mixup works better on data points that is negatively correlated to test set performance and (2) the extrapolating methods do not work as well as original (interpolating) mixup.", "summary_of_the_review": "This paper is overall interesting and brings some new insights to the understanding of mixup and deep ensemble. However, extra efforts may be required to justify its claims, which I believe can greatly improve the paper.", "strength_and_weaknesses": "Strengths:\n- The paper produces quite a few analyses, including (1) drawing attention to the similarity in the effect of ensemble and mixup, (2) how ensemble and mixup do not seem to work well on out-of-distribution inputs (despite later empirically they worked pretty well on the dataset evaluated), (3) how extrapolating mixup generates good looking uncertainty in a decision boundary plot (again despite the empirical experiment).\n- The paper is well-written and easy to follow.\n\nWeaknesses:\n- The paper seems more like an exploration of mixup methods, covering aspects on its relation with ensemble and ood generation. I did not find affirmative answers on some of the questions I had in mind when reading the paper: (1) how or why mixup is similar to ensemble learning (apart from figure 1, I was hoping for more intuition/proof on why ensemble, something that produces a more soft boundary on hard examples, and mixup, that does not really differentiate hard and easy examples, have similar boundaries as claimed by the paper. Or is the similarity really of that high a degree?) (2) why extrapolations worked well on figures, but not on the dataset evaluated.\n\n- The paper also proposed a Test-Time mixup idea, that simulates ensemble as it proposes several predictions that can be averaged. However, it is not exactly clear the advantage of this idea over vanilla mixup. It seems that works better on instances that were selected to correlate negatively to the test time performance (based on other models), is this a sign of good generalization or robustness? It is not quite clear. To validate whether the Test-Time mixup idea works, it would be better if more experiments on different datasets/backbone models are performed.\n\n\nQuestions:\n1. How is Figure 2 drawn --- how are uncertainty measured for instances not belonging to the dataset.\n2. A possible explanation for why the empirical experiment on ood in table 2 disagree with the plots for extrapolating: the ood samples that extrapolating has low confidence of might correspond to random images (pure noises), instead of actual images with different digits.\n\n\n\nTypos:\n* Section 3 last paragraph: where on trains a classifier -> where one trains a classifier\n* In Table 2, for worst acc (ne), Ex-mixup-v2 has a higher value (0.509) than TT-mixup (0.507), and should be colored orange.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "KVHGWXlQkXJ", "reviewer": "Reviewer_AwxV", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studied the ensemble perspective of the mixup. They proposed novel variant of mixup, i.e. TT-mixup and Ex-mixup. TT-mixup shows highly competitive result with the K-ensemble model, however taking K times less training and deployment cost. The experiment on the small data (CIFAR10) validates the efficacy of the proposed model.", "summary_of_the_review": "Overall, the idea is interesting and applicable to many practical problems, but the evaluation is weak and theoretical justification is missing.\nthe contribution of the Ex-mixup is confusing. \n\nPlease refer to strength and weakness section for details.", "strength_and_weaknesses": "Strength:\n\n1: The mixup ensemble is interesting, and it provides an efficient solution and shows a competitive result compared to the ERM ensemble.\n\n2: The proposed TT-mixup and Ex-mixup are interesting, but the evaluation is somewhat weak. \n\n3: The motivation is clear. The result in the Table-2 is comprehensive and Figure-3 seems interesting, especially the right figure for the Extra-mixup-v2. \n\nWeaknesses: \n\n1: The paper evaluated the model empirically. However, a proper theoretical justification can be provided for the ensemble kind of model. Is it possible to provide some bound/relation between the mixup and ensemble? \n\n2: Most of the study, like generalization ability, OOD robustness, calibration, uncertainty estimates adversarial robustness are exist in the literature. So, in terms of these study the paper is not novel, but the proposed mixup is novel. \n\n3: The results are evaluated only over a small dataset i.e. CIFAR10/CIFAR10-Neg. Which are not sufficient to evaluate the model efficacy. I request to the author to please provide the result for the ensemble and TT-mixup result for the CIFAR100 dataset. It contains a larger class and more similar classes. \n\n4: In the section-4 Eq:2, term “u” is not clear. How it is used in the model? Please provide the details. \n\n5: Reproducibility may be a key concern, mixup itself has a lot of randomness and here parameters and implementation details are not complete. Also, datasets split details are missing. Overall, reproducibility is hard. I request to the author to please provide the code with the details descriptions. What are OOD samples?. \n\n6: In the Figure:4, on the X-axis there are number 1,2 what are that numbers?\n\n7: Ex-mixup-v1 and Ex-mixup-v2 showing consistently poor result, when it will be useful? Did the author tried over some extrapolated data? In the Figure-3 (Extra-mixup-v2) it shows very impressive result but same pattern are not evident in the Table-2 why? \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "5bk-ys3RBd", "reviewer": "Reviewer_3JQa", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes forming an \"ensemble\" at test time from a single mixup-trained model by averaging over multiple mixup predictions between the test example and K \"reference\" examples. They also experiment with improving the OOD uncertainty by training the mixup model with additional \"extrapolation\" examples in which they apply mixup between a training example and an extrapolation of that example, using the original label and an C+1 uncertainty label, with C being the number of classes. Empirically, they show that their original test-time mixup model (TT-mixup) outperforms even deep ensembles on the CIFAR-10 test set and CIFAR-10-Neg on accuracy, ECE, and diversity.", "summary_of_the_review": "Overall, this is a well-written paper with a precise set of methods, a nice set of empirical results that demonstrate the effectiveness of the method against the gold-standard deep ensembles, and a clear takeaway (use test-time mixup). One main critique is that the test-time efficiency claim should be expanded upon to better reflect that the method is still quite costly at test time.\n\n\n#### Minor\n- p. 2: Near the end of the introduction, \"ERM\" is used before defining it in section 2. Though ERM is a common abbreviation, it's still nicer to define it upon first usage.\n- p. 2: Capitalize \"mixup\" in the sentence beginning as \"mixup ensembles are K times cheaper ...\" in the Contributions section.\n- p. 4, Table 1: What is \"idem\"?\n- p. 5: MIMO also has the benefit of cheaper training and test step time than k applications of the model.\n- p. 7: ECE original reference is Naeini et al, 2015.", "strength_and_weaknesses": "This paper has several strengths. At a high-level, the paper is clear and precise in the writing, with well-explained methods. Empirically, the experiments evaluated the methods on a reasonable set of metrics (accuracy, NLL (\"data loss\"), ECE, OOD accuracy), and all results included error bars, which is great. The takeaway from the paper is clear (use test-time mixup), and the paper also included a negative result (that the extrapolation mixup proposal wasn't generally beneficial), which is also refreshing to see.\n\nIn terms of weaknesses, it would be nice to better recognize that the proposed test-time mixup incurs a cost of K applications of the model at test time per example. Since both this test-time mixup and deep ensembles are trivially vectorized (not discussed in the paper), the main efficiency gain at test time is the potential for improved memory bandwidth due to having only one copy of the weights loaded in device memory. The factor of K predictions can still be prohibitively costly in terms of computation, and it would be great to better include that in the paper. Additionally, it could have been nice to see results for the combination of ensembles and mixup to better understand how they could complement each other, regardless of the increased inefficiency; this is a minor critique though.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject