Combining Ensembles and Data Augmentation Can Harm Your Calibration
Authors: Yeming Wen
Ghassen Jerfel
Rafael Muller
Michael W Dusenberry
Jasper Snoek
Balaji Lakshminarayanan
Dustin Tran
Venue: ICLR 2021 Poster (2021)
Keywords: Ensembles
Uncertainty estimates
Calibration

Abstract: Ensemble methods which average over multiple neural network predictions are a simple approach to improve a model’s calibration and robustness. Similarly, data augmentation techniques, which encode prior information in the form of invariant feature transformations, are effective for improving calibration and robustness. In this paper, we show a surprising pathology: combining ensembles and data augmentation can harm model calibration. This leads to a trade-off in practice, whereby improved accuracy by combining the two techniques comes at the expense of calibration. On the other hand, selecting only one of the techniques ensures good uncertainty estimates at the expense of accuracy. We investigate this pathology and identify a compounding under-confidence among methods which marginalize over sets of weights and data augmentation techniques which soften labels. Finally, we propose a simple correction, achieving the best of both worlds with significant accuracy and calibration gains over using only ensembles or data augmentation individually. Applying the correction produces new state-of-the art in uncertainty calibration and robustness across CIFAR-10, CIFAR-100, and ImageNet.

Published as a conference paper at ICLR 2021
COMBINING ENSEMBLES AND DATA AUGMENTATION CAN HARM YOUR CALIBRATION
Yeming Wen"!, Ghassen Jerfel*?, Rafael Maulllrr,,Mihanne W. Dusenberry?, Jasper Snoek?, Balaji Lakksmiinaraaaaan & Dustin Tran?
Equal contribution, UUniversity of Texas, Austin, GGogge Brain
ABSTRACT
Ensemble methods which average over multiple neural network predictions are a simple approach to improve model's calibration and robustness. Similarly, data augmentation techniques, which encode prior information in the form of invariant feature transformations, are effective for improving calibration and robustness. this paper, we show a surprising pathology: combining ensembles and data augmentation can harm model calibration. This leads to trade-off in practice, whereby improved accuracy by combining the two techniques comes at the expense of calibration. On the other hand, selecting only one of the techniques ensures good uncertainty estimates at the expense of accuracy. We investigate this pathology and identify a compounding under-confidence among methods which marginalize over sets of weights and data augmentation techniques which soften labels. Finally, we propose a simple correction, achieving the best of both worlds with significant accuracy and calibration gains over using only ensembles or data augmentation individually. Applying the correction produces new state-of-the art in uncertainty calibration across 10, CIFAR- 100, and ImageNet.'
INTRODUCTION
Many success stories in deep learning (Krizhevsky et al., 2012; Sutskever et al., 2014) are in restricted settings where predictions are only made for inputs similar the training distribution. In real-world scenarios, neural networks can face truly novel data points during inference, and in these settings it can be valuable to have good estimates of the model's uncertainty. For example, in healthcare, reliable uncertainty estimates can prevent over-confident decisions for rare or novel patient conditions (Dusenberry et al., 2019). We highlight two recent trends obtaining state-of-the-arr in uncertainty and robustness benchmarks.
Ensemble methods are a simple approach to improve model's calibration and robustness (Lakshmi- narayanan et al., 2017). The same network architecture but optimized with different initializations can converge to different functional solutions, leading to decorrelated prediction errors. By averaging predictions, ensembles can rule out individual mistakes (Lakshminarayanaa et al., 2017; Ovadia et al., 2019). Additional work has gone into efficient ensembles such as MC-dropout (Gal and Ghahramani, 2016), Batch and its variants (Wen al., 2020; Dusenberry al., 2020; Wenzel et al., 2020). These methods significantly improve calibration and robustness while adding few parameters to the original model.
Data augmentation is an approach which is orthogonal to ensembles in principle, encoding additional priors in the form of invariant feature transformations. Intuitively, data augmentation enables the model to train on more data, encouraging the model to capture certain invariances with respect to its inputs and outputs: data augmentation may also produce data that may be closer an out-of- distribution target task. It has been key factor driving state-of-the-an for example, Mixup (Zhang al., 201 8; Thulasidasan et al., 2019a), AugMix (Hendrycks et al., 2020), and test-time data augmentation (Ashukha et al., 2020).
common wisdom in the community suggests that ensembles and data augmentation should naturally combine. For example, the majority of uncertainty models in vision with strong performance are
!Contact: ywen caen@uteexs.edu Code: https: / / wtw.bbccoo/g.... goog] rttddwaddd/teeee/e/eeeeeee experimental mental / z pation__ oittiinnmiuu xup.

---

Published as a conference paper at ICLR 2021
built upon baselines leveraging standard data augmentation (He et al., 2016; Hendrycks al., 2020) (e.g., random flips, cropping); Hafner et al. (2018) cast data augmentation as an explicit prior for Bayesian neural networks, treating it as beneficial when ensembling; and Hendrycks et al. (2020) highlights further improved results in AugMix when combined with Deep Ensembles (Hansen and Salamon, 1990; Krogh and Vedelsby, 1995). However, we find the complementary benefits between data augmentations and ensembels are not universally true. Section 3.1 illustrates the poor calibration of combining ensembles (MC-dropout, BatchEnsemble and Deep Ensembles) and Mixup on CIFAR: the model outputs excessive low confidence. Motivated by this pathology, in this paper, we investigate in more detail why this happens and propose method resolve it.
Contributions. In contrast to prior work, which finds individually that ensembles and Mixup improve calibration, we find that combining ensembles and Mixup consistently degrades calibration perfor- mance across three ensembling techniques. From a detailed analysis, we identify a compounding under-confidence, where the soft labels Mixup introduce a negative confidence bias that hinders its combination with ensembles. We further find this to be true for other label-based strategies such as label smoothing. Finally, we propose CAMixup to correct this bias, pairing well with ensembles. CAMixup produces new
calibration on both CIFAR-10/ 10/R-0001000 (e.g., 0.4% and 2.3% on CIFAR- 10 and CIFAR- 10C). building on Wide ResNet 28-10 for competitive accuracy (c.g.. 97.5% and 89.8%) and on ImageNet (1 (1.5%), building ResNet-50 for competitive accuracy (77.4%).
2
BACKGROUND ON CALIBRATION, ENSEMBLES AND DATA AUGMENTATION
2.1
CALIBRATION
Uncertainty estimation is critical but ground truth is difficult to obtain for measuring performance. Fortumately, calibration error, which assesses how well model reliably forecasts its predictions over population, helps address this. Let (Ý, P) denote the class prediction and associated confidence (predicted probability) of classifier.
Expected Calibration Error(ECE): One notion of miscalibration is the expected difference between confidence and accuracy (Nacini et al., 2015): Ey||P(Y P = p) - pl]. ECE approximates this by binning the predictions in [0, 1] under .M equally-saaced intervals, and then taking a weighted average of each bins' accuracy/confidecce difference. Let Bm be the set of examples in the m th bin whose predicted confidence falls into interval (m-1 mi The bin Bm 's accuracy and confidence are:
Acc(B
Σ 1 = v.), Conf(Bm)) #:EBm.
Σ
P..
Bm
B
z.E.B..
where d and yr are the predicted and true labels and p; the confidence for example z₁. Given n examples, ECE is Em=1
|Bm.
Acc(Bm) Conl(B.))
2.2 ENSEMBLES
Aggregating the predictions of multiple models into an ensemble is a well-established strategy to improve generalization (Hansen and Salamon, 1990; Perrone and Cooper, 1992; Dietterich, 2000).
BatchEnsemble: BatchEnsemble takes a network architecture and shares its parameters across ensemble members, adding only rank-1 perturbation for each layer in order to decorrelate member predictions (Wen et al., 2020). For given layer, define the shared weight matrix among K ensemble members as W E Rmxd tuple trainable vectors F2 E R and s; E RR." are associated with each ensemble member k. The new weight matrix for each ensemble member in BatchEnsemble is
(2) where denotes the element- wise product. Applying rank-1 1 perturbations via and adds few additional parameters to the overall model. We use an ensemble size of 4 in all experiments.
W WoWl Fk, where F2 = rESs E R
Gal and Ghahramani (2016) interpret Dropout (Srivastava et al., 2014) model, leading to its application for uncertainty estimates by sampling multiple dropout masks at test time in order to ensemble its predictions. We use ensemble size of 20 in all experiments.
MC-Dropout:
ensemble
Deep Ensembles: Composing an ensemble of models, each trained with a different random initial- ization, provides diverse predictions (Fort et al., 2019) which have been shown to outperform strong
2

---

Published as a conference paper at ICLR 2021
baselines uncertainty estimation tasks (Lakshminarayanan et al., 2017). We use ensemble size of 4 in all experiments.
In this work, focus the interaction between data augmentation strategies and BatchEnsemble, MC-Dropout, and deep ensembles. Other popular ensembling approaches leverage weight averaging such Polyak-Ruppert (Ruppert, 1988), checkpointing (Huang et al., 2017), and stochastic weight averaging (Izmailov et al., 201 8) to collect multiple sets of weights during training and aggregate them to make predictions with only single set.
2.3 DATA AUGMENTATION
Data augmentation encourages a model to make invariant predictions under desired transformations which can greatly improve generalization performance. For example, in computer vision, random left- right flipping and cropping are de-facto approaches (He et al., 2016). We highlight two state-of-the-art techniques which we study.
Mixup: Mixup (Zhang et al., 2018) manipulates both the features and the labels in order to encourage linearly interpolating predictions. Given example (Ii, Vt). Mixup applies
c = Az; + (1 - \)æy, y = Ay, + (1 - A)y,-
(3)
Here, zj is sampled from the training dataset (taken from the minibatch), and ~ Beta(a, a) for fixed hyperparameter
Mixup was shown to be effective for generalization and calibration of deep neural networks (Zhang al., 2018; Thulasidasan et al., 2019b). Recent work has investigated why Mixup improves generalization (Guo et al., 2018; Shimada et al., 2019) and adversarial robustness (Beckham et al., 2019; Pang et al., 2020; Mangla et al., 2020). Given Mixup's simplicity, many extensions have been proposed with further improvements (Yun et al., 2019; Berthelot et al., 2019; Verma et al., 2019; Roady et al., 2020; Chou et al., 2020).
AugMix: Searching sampling over set of data augmentation operations lead to significant im- provement on both generalization error and calibration (Cubuk et al., 2019b;a). AugMix (Hendrycks et al., 2020) applies consistency loss to encourage similarity across the augmentations. AugMix achieves state-of-the-ar calibration across in- and out-of-distribtiion tasks. Let O be the set of data augmentation operations and & be the number of AugMix iterations. Mix samples wi,....; Wy ~ Dirichlet(a,....... , a) for a fixed hyperparameter a >> 0 and op...... ОPд from O. Given an interpolation parameter m, sampled from Beta(a, a), the augmented input Taugmia is:
of augmentations, each with random weighting, with Jensen-Shanme
Laugmiz = M.orig + (1 m Taug.
Taug
wdop, (Forig).
(4)
3 MIXUP-ENSEMBLE PATHOLOGY
We seek to understand the effect of data augmentations on ensembles. In particular, we hope to verify the hypothesis of compounding improvements when combining the seemingly orthogonal techniques of data augmentation and ensembles. To our surprise, we find that augmentation techniques can be detrimental to ensemble calibration.
3.1 THE SURPRISING MISCALIBRATION OF ENSEMBLES WITH MIXUP
Ensembles are the most known and simple approaches to improving calibration (Ovadia et al., 2019; Lakshminarayanan al., 2017), and Thulasidasan al. (2019b) showed that Mixup improves calibration in single network. Motivated by this, Fig. 1 applies Mixup to each ensemble member on CIFAR- CAAAA--OOCIFAR-OOO with WideResNet 28-10 (Zagoruyko and Komodakis, 2016). Here, we searched over Mixup's 's optimal hyperparameter a (Eq. 3) and found that 1 gives the best result, which corroborates the finding in Zhang et al. (2018). All data points in Fig. are averaged over random seeds.
Figs. Ia and 1b demonstrate improved test accuracy (Red (ensembles without Mixup) to Blue (ensembles with Mixup)). However, if we shift focus Figs. Ic and Id's calibration error, it is evident that combining Mixup with ensembles leads to worse calibration (Red to Blue). This is counter- intuitive as would expect Mixup, which improves calibration of individual models (Thulasidasan
3

---

Published as a conference paper at ICLR 2021
4.2
10
19.5
19.0
5
18.5
3.6
18.0
4
17.5
3.2
17.0
2
16.5
1
2
BE
MC
DE
BE
MC
DE
BE
MC
DE
BE
MC
DE
(a) CIFARIO-Erro (%)
(b) CIFARI00-Error (%)
(c) CIFARIO-ECE (%)
(d) CIFARI00-ECE (%)
Figure 1: WideResNet 28-10 18-10 CIFAR-IO/CIFAR-III Red: Ensembles without Mixup; Blue:
Ensembles with Mixup; Orange: Individual models in ensembles without Mixup. (a) & (b): Applying
Mixup different ensemble methods leads consistent improvement on test accuracy. (c) & (d): Applying Mixup to different ensemble methods harms calibration. Averaged over 5 random seeds.
et al., 2019a), to also improve the calibration of their ensemble. Fig. I confirms this pattern across Batchensemble (BE), MC-dropout (MC), and deep ensembles (DE). This pathology also occurs on ImageNet, as seen in Table 1.
Why do Mixup ensembles degrade calibration? To investigate this in more detail, Fig. plots a variant of reliability diagrams (DeGroot and Fienberg, 1983)
2.023
BatchEnsemble. We bin the predictions into M = 15 equally spaced intervals based their confidence (soft-
Accuracy Conffidence
0.000
max probabilities) and compute the difference between the average confidence and the average accuracy in Eq. for each bin. Fig. 2 tracks this difference over vary- ing confidence levels, positive difference (Ace- Conf) implies under-confidence with respect to the true fre- quencies; negative implies over-confidence; and zero implies perfect calibration.
-2015
Single
-1,20
MixupSingle
MixuBBE
BatchEnsemble
Contidence (max prob)
The backbone model in Fig. is BatchEnsemble with an ensemble size of 4 (we also found this consistent for MC-Dropout and Deep-Ensemble). The figure presents Single: vanilla WideResNet 28-10 10; Mix-
Figure 2: Reliability diagrams on CIFAR- 100 with a WideResNet 28-10.
4 methods:
upSingle: WideResNet 28- model trained with Mixup; BatchEnsemble: vanilla BatchEnsemble WideResNet 28- 10 model; MixupBE: BatchEnsemble WideResNet 28-10 model trained with Mixup. Fig. 2 shows that only models trained with Mixup have positive (Ace Conf) values on the test set, which suggests that Mixup encourages under-confidence. Mixup ensemble's under-confidence also greater in magnitude than that of the individual Mixup models. This suggests that Mixup ensembles suffer from compounding under-confidence, leading worse calibration for the ensemble than the individual Mixup models' calibration. This is contrary to our intuition that ensembles always improves calibration.
To further visualize this issue, Appendix C's Fig. 8 investigates the confidence (softmax probabilities) surface of deep ensembles and Mixup when trained on a toy dataset consisting of 5 clusters, each with different radius. We ensemble over 4 independently trained copies of 3-layer MLPs. Deep ensemble's 's predictive confidence is plotted over the entire input data space Fig. 8c. The resulting predictions extremely confident except at the decision boundaries. Deep Ensemble still displays high confidence the area nearest to the origin which is expected to have lower confidence level. On the other hand, Fig. 8d shows that Mixup-Ene -nsup-e is only confident in very constrained area around the training clusters, leading to an overall under-confident classifier which confirms our postulation of compounding under-confidence.
4

---

Published as a conference paper at ICLR 2021
3.2 Is THE PATHOLOGY SPECIFIC TO MIXUP?
At the core of the issue is that Mixup conflates data uncertainty (uncertainty inherent to the data generating process) with model uncertainty. Soft labels can conrect for over-confidence in single models which have no other recourse to improve uncertainty estimates. However, when combined with ensembles, which incorporate model uncertainty, this correction may be unnecessary. Because image classification benchmarks tend to be deterministic, soft labels encourage predictions on training data to be less confident about their true targets even if they are correct. We validate this hypothesis by showing it also applies to label smoothing.
Label Smoothing: Like Mixup, label smoothing applies soft labels: it smoothens decision boundaries by multiplying data point's 's true class by (1 ---a,, a), with probability a spread equally across other classes. Using the same experimental setup before, we apply increasing levels of label smoothing to ensembles of Res 28- 10 models trained on CIFAR-10. Fig. 3 demonstrates the harmful effect of label smoothing on CIFAR- 10 ECE, particularly when aggressive (coeff 0.2). In the concurrent work, Qin et al. (2020) found that label smoothing plus ensemble leads to worse calibration. They showed that adjusting model confidence successfully corrects the compounding underconfidence.
Error and ECE of Label Smoothing WideResNet 28-10 trained on CIFAR-10
77.00
MC Cropout
MC Dropout
59.95
Deep Ensermibes
BatchEnsemble
5.0
Deep Ensembles
BatchE nsemb
ECE
42.62
4.5
16.15
7.68
3.5
0.1
0.2
0.5
0.7
0.9
0.1
0.2
0.5
0.7
0.9
Smoothing Intersity
Label Smoothing Intensity
Figure 3: ECE and Error on CIFAR- 10 with label smoothing on MC Dropout, Deep Ensembles, and BatchEnsemble. ECE degrades with label smoothing, particularly when it is more aggressive (2 0.2).
4 CONFIDENCE ADJUSTED MIXUP ENSEMBLES (CAMIXUP)
In this section, we aim to fix the compounding ander-confidence. issue when combining Mixup and ensembles without sacrificing its improved accuracy on both in- and out-of-distribution data.
4.1 CLASS BASED CAMIXUP
Mixup encourages model under-confidence as shown in Fig. 2. Notice that Mixup assigns a uniform hyperparameter all examples in the training set. To improve Mixup, start from the intuition that in classification, some classes are prone to be more difficult than others to predict. This can be confirmed by Fig. 4a, which provides examples of per-class test accuracy. Ideally, prefer our model to be confident when it is predicting over easy classes such as cars and ships. For harder classes like cats and dogs, the model is encouraged to be less confident achieve better calibration.
Therefore, instead of a uniform Mixup hyperparameter for all classes, we propose to adjust the Mixup hyperparameter of each class by the difference between its accuracy and confidence. CAMixup's intuition is that we want to apply Mixup on hard classes on which models tend to be over-confident. On easy examples, we impose the standard data-augmentattin without Mixup. This partially prevents Mixup models from being over-confident on difficult classes while maintaining its good calibration on -uito--distribution inputs.
Denote the accuracy and confidence of class as Ac)(Ci) and Conf(C.). We adjust Mixup's À in Eqn. 3 by the sign of Acc(C.) - Conf(C:), which are defined as Aco(C,) [.. Exjec; 1 = i)
and Conf(C:) Jे. Exyec; Pi.
00 Acc(C.) > Con)(C;)
A a Acc(C,) < Conf
(5)
We focus on classification, where classes form natural grouping of easy to hard examples. However, the idea he used metadata that we'd 'd like to halance uncertainty estimates, e.g., gender and age groups
5

---

Published as a conference paper at ICLR 2021
Average Accuracy
CNN
Soltmax protabililies
Member2
Member2
93.53%
200
Member3
Memberd
or 79%
Accuracy Confidence
150
07.80%
Underconfidence
Overcomfidence
Disable Mixup
Enable Mixup
Cars(97.8) Cats(90.6) Dogs(93.5) Ships(97.9)
(a) Proposed CAMixup method.
(b) epochs in which Mixup was applied to each class.
Figure 4: Left: An illustration of the proposed CAMixup data augmentation. Selected per-class test accuracies are showed in brown. Overall test accuracy is 96.2% on CIFAR-10; Right: Number of epochs (out 250) where CAMixup enables Mixup for selected classes in BatchEnsemble. CAMixup tends to assign Mixup hard classes. Counts accumulated individually for each ensemble member (ensemble size 4).
If the model is already under-confidem. class (Acc(C;) Con Mixup is not applied to examples in the class, and A = O. However, if Acc(C,) s Conf(C,), the model over-confident this class, and Mixup is applied to reduce model confidence. We compute the accuracy and confidence on a validation dataset after each training epoch.
4.0
19.5
10
3.8
19.0
18.5
18.0
4
3.4
17.5
17.0
2
3.0
16.5
1
BE
MC
DE
BE
MC
DE
BE
MC
DE
BE
MC
DE
(a) CIFAR 10-Error (%)
(b) CIFAR 100-Error (%)
(c) CIFARIO-ECE (%)
(d) CIFAR 100-ECE (%)
Figure 5: WideResNet 28-10 10 on
Red: Ensembles without Mixup; Blue: En- sembles with Mixup; Green: Our proposed CAMixup improves both accuracy & ECE of ensembles.
Notice that X; is dynamically updated the end of each epoch. To understand which classes more often assigned Mixup operation, Fig. 4 calculates the number of times that A; > 0 throughout training. The maximum number of times is the number of total training epochs, which 250 in the BatchEnsemble model. We find that CAMixup rarely enables Mixup to easy classes such as cars and ships: the number of times is less than 10% of the total epochs. For harder classes like cats and dogs, CAMixup assigns Mixup operation almost every epoch, accounting for more than 80% of total epochs. summary, Fig. shows that CAMixup reduces model confidence on difficult classes and encourages model confidence easy classes, leading to better overall calibration. Appendix D.I's Fig. 9a also shows that CAMixup effectively shifts the confidence to the lower region.
Fig. 5 presents results of CAMixup on CIFAR-10 and CIFAR-100 test set, where compare the effect of Mixup and CAMixup on different ensembling strategies (BatchEnsemble,, MC Dropout, DeepEnsemble). Adding Mixup to ensembles improves accuracy but worsens ECE. Adding CAMixup to ensembles significantly improves accuracy of ensembles in all cases. More importantly, the calibration results in Figs. 5c and 5d show that CAMixup ensembles significantly better calibrated

---

Published as a conference paper at ICLR 2021
than Mixup ensembles, for instance, CAMixup reduces ECE by more than 5X for BatchEnsemble over Mixup. We observe a minor decrease in test accuracy (at most 0.2%) when comparing CAMixup ensembles with Mixup ensembles, but we believe that this is worthwhile trade-off given the significant improvement in test ECE.
Table 1 presents similar experiments applied to ResNet-50 on ImageNet, using BatchEnsemble as the base ensembling strategy. These results are state of the
Table I: Batchensemble with en-
semble size 4 on ImageNet.
the best of our knowledge: Dusenberry et al. (2020) report 1.7% ECE with Rank- 1 Bayesian neural nets and 3.0% with Deep Ensembles; Thulasidasan et al. (2019a) report 3.2% for ResNet-50 with Mixup, 2.9% for ResNet-50 with an entropy-regularized loss, and 1.8% for ResNet-50 with label smoothing.
ACC
ECE
Batchle MixupBE CAMixupBE
77.0
2.0%
77.5
2.1%
77.4
1.5%
4.2 PERFORMANCE UNDER DISTRIBUTION SHIFT
Here, we assess model resilience covariate shift by evaluating on the CIFAR-10-4 and SEFAR-10-C
18
benchmarks (C stands for corruptions) proposed by Hendrycks and Dietterich (2019a), which apply 15 types of corruptions each with levels of intensity.
28
14
We evaluate the performance of CAMixup vs Mixup when applied to different ensembles, and report aver- age error on ECE across different types of corruptions and intensities.
24
22
10
20
18
Fig. 6a shows that Mixup improves accuracy the corrupted dataset because of its strong regularization effect. However, the models tend to be over-confident as one moves further from the original distribution (higher corruption intensities), so encouraging under- confidence is not an issue. This explains why Mixup ensembles maintain low ECE on distiiriinttion test data in Fig. 6b.
16
BE
MC
DE
BE
MC
DE
(a) CIFARIO-C-Eror (b) CIFARIO-C-ECE
Figure 6: WideResNet 28-10 on CIFAR-1O-C. Red: Ensembles without Mixup: Blue: En- sembles with Mixup; Green: Ensembles with CAMixup (ours).
Fig. 6b also shows that CAMixup's calibration on out-of-distribution data (CIFAR-1O-C) is also on par with Mixup ensembles. We observe the same result on 100---100-C (Appendix D.1's Fig. 9). Thus, we successfully improve model calibration on in-distribution datasets without sacrificing its calibration dut-o--distribution datascts.
COMPOUNDING THE BENEFITS OF CAMIXUP WITH AUGMIX ENSEMBLES
We have investigated why certain data augmentation schemes may not provide complementary benefits to ensembling. We proposed class-adjusted Mixup (CAMixup) which compounds both accuracy and ECE over vanilla ensembles. We believe that the insights from our work will allow the community and practitioners compound SOTA performance. We provide two concrete examples.
5.1 AUGMIX
We show how CAMixup can compound performance over ensembles of models trained with AugMix, which were shown by Hendryeks et al. (2020) to achieve state-of-the-ae accuracy and calibration on both clean and corrupted benchmarks. We primarily focus on improving BatchEnsemble and we investigate if adding better data augmentation schemes closes the gap between memory-efficient ensembles (BatchEnsemble) and independent deep ensembles.
As discussed in Section 2.3, AugMix only uses label-preserving transformations. Therefore AugMix provides complementary benefits to ensembles (and CAMixup). This is consistent with calibration improvements in the literature with ensemble methods, which apply standard data augmentation such as random flips, which also do not smoothen labels.
We consider combination of AugMix and Mixup as it allows the model to encounter both diverse label- preserinn augmentations and soft labels under a linearly interpolating regime. The combination
7

---

Published as a conference paper at ICLR 2021
BatchEnsemble
MixupBE
CAMixupBE
Aig
Aug Mixup
AugCAMixup
8.
B-
88
(a) CIFAR-10 10 Error
(b) CIFAR-10 ECE
(c) CIFAR-1OX 100 Error
(d) CIFAR- ECE
Figure Performance on Batchensemble under dataset shift. Mixup and Muxup improve accuracy and calibration under shift but significantly worsen in-distribution calibration. Our proposed CAMixup and AugCAMixup improve accuracy and calibration.
CIFAR-10
CIFAR-100 ECE(L)
Method/Metric
Acc(t)
ECE(L)
cA/CECE
Acc(t)
cA/EECE
AugMix BE
97.36
1.02%
89.49/2.6%
83.57
2.96%
67.12/7.1%
AugMixup BE
97.52
1.71%
90.05/2.8%
83.77
4.19%
69.26/4.8%
AugCAMixup BE
97.47
0.45%
89.81/2.4%
83.74
2.35%
68.71/4.4%
Table 2: Results for Wide ResNet-28-11 10 BatchEnsemble on in- and out-of-distribution CIFAR-10/100 with various data augmentations, averaged over seeds. AugMix: AugMix + BatchEnsemble; AugMixup: AugMix + Mixup BatchEnsemble; AugCAMixup: AugMix + CAMixup BatchEnsem- ble. Adding Mixup to AugMix model increases test accuracy and corrupt accuracy at the cost of calibration decay on testset. CAMixup bridges this gap with only a minor drop in accuracy.
AugMixup (AugMix + Mixup) can be written
x = À AugMix(z:) + (1 - A) AugMix(22), y = * yı + (1 A) y22-
(6)
Consistent with earlier results Mixup, Table 2 shows combining AugMixup with BatchEnsemble improves accuracy but worsens ECE, leading to under-confidence on in-distribution data. (Ap- pendix D.2's Fig. 10). With our proposed fix CAMixup, the combination AugCAMixup (AugMix + CAMixup) improves calibration while retaining the highest accuracy for ensembles. Fig. 7 shows detailed results on CIFAR- JO-C and CIFAR-100-6 Similar to Mixup, AugMixup improves calibra- tion under shift but worsens calibration on in- in-distribution. However, our proposed AugCAMixup improves accuracy and calibration of ensembles on both clean and corrupted data.
To the best of our knowledge, these results are state-of- tftte-of-the-aate arr in the literature: Dusenberry et al. (2020) report 0.8% ECE and 1.8% ECE for CIFAR- 10 and CIFAR- 100 along with 8% and 11.7% ECE for corruptions; Guo et al. (2017) report 0.54% and 2.3% ECE for the smaller Wide ResNet 32 on CIFAR-10 and CIFAR- 100 with temperature scaling (93% and 72% accuracy), and Ovadia et al. (2019) demonstrated that temperature scaling does not extend to distribution shift.
5.2 TEMPERATURE SCALING
In concurrent work, Rahaman and Thiery (2020) consider the interplay between data augmentation and ensembling calibration. They also find that Mixup ensembles can be under-confident, and propose temperature scaling as solution. Their core contribution is the same but differ in slight ways: further this analysis by showing the compounding under-confidence extends to other techniques applying soft labels such as label smoothing, and we propose CAMixup as a solution. Post-hoe calibration techniques like temperature scaling complementary the core conflation issue with Mixup. Corroborating findings Ovadia et al. (2019), Appendix G shows combining CAMixup and temperature scaling can further improve test calibration error; does not improve -ut-of-distribution calibration. Another concurrent work showed that calibrated ensemble members do not always lead to calibrated ensemble predictions (Anonymous, 2021).
proposal and do not address
6 CONCLUSION
Contrary to existing wisdom the literature, we find that combining ensembles and Mixup con- sistently degrades calibration performance across three ensembling techniques. From detailed
8

---

Published as a conference paper at ICLR 2021
analysis, we identify compounding under-confidence, where Mixup's soft labels (and more broadly, label-based augmentation strategies) introduce negative confidence bias that hinders its combination with ensembles. To correct this, propose CAMixup, which applies Mixup to only those classes on which the model tends to be over-confident, modulated throughout training. CAMixup combines well with state-of-the-art methods. It produces state-of-the-ee calibration across CIFAR-10, CIFAR- 100, and ImageNet while obtaining competitive accuracy. Appendix H points out potential future work and limitations of CAMixup.

---

Published as a conference paper at ICLR 2021
REFERENCES
Chirag Agarwal and Sara Hooker. Estimating example difficulty using variance of gradients. arXiv preprint arXiv:2008. 11600, 2020.
Anonymous. Should ensemble members be calibrated? In Submitted to International Confer- ence on Learning Representations, 2021. URL https: http///ppeeeiiieeeeeeeeeeeeeeeee wtw.:/uueeettttttteeeemmmmmommme forum?ick WIWLFuDkvKp. under review.
/
Arsenii Ashukha, Alexander Lyzhov, Dmitry Molchanov, and Dmitry Vetrov. Pitfalls of in-domain uncertainty estimation and ensembling in deep learning. In International Conference Learning Representations, 2020.
Christopher Beckham, Sina Honari, Alex Lamb, Vikas Verma, Farnoosh Ghadiri, R. Devon Hjelm, and Christopher Joseph Pal. Adversarial mixup resynthesizers, ArXiv, abs/1903.02709, 2019.
David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel. Mixmatch: A holistic approach semi-supervised learning. In Advances in Neural Information Processing Systems, pages 5049-5059, 2019.
Hsin-Ping Chou, S. Chang, J. Pan, Wei Wei, and D. Juan. Remix: Rebalanced mixup. ArXiv, abs/2007.03943, 2020.
Ekin D. Cubuk, Barret Zoph, Jonathon Shlens, and Quoe V. Le. Randaugment: Practical automated data augmentation with reduced search space. arXiv: Computer Vision and Pattern Recognition, 2019a.
Ekin Dogus Cubuk, Barret Zoph, Dandelion Mané, V. Vasudevan, and Quoe V. Le. Autoaugment: Learning augmentation strategies from data. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 113-123, 2019b.
Morris H. DeGroot and Stephen E. Fienberg. The Comparison and Evaluation of Forecasters. The Statistician, 32(1/2):12, March 1983. ISSN 00390526. doi: 10.2307/2987588, URL https: // //www. jstor org wttpg//saal///////eeeeeeeeeeeeee 10. 00100///2877587111iii... crossref..
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei Li. Imagenet: A large-scale hierarchical image database. 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248-255, 2009.
Thomas G. Dietterich. Ensemble methods in machine learning. Multiple Classifier Systems, 2000.
Michael W Dusenberry, Dustin Tran, Edward Choi, Jonas Kemp, Jeremy Nixon, Ghassen Jerfel, Katherine Heller, and Andrew M Dai. Analyzing the role of model uncertainty for electronic health records. arXiv preprint arXiv:1906 176.04190 2019.
Michael W. Dusenberry, Ghassen Jerfel, Yeming Wen, Yi-an Ma, Jasper Snoek, Katherine Heller, Balaji Lakshminarayanan, and Dustin Tran. Efficient and scalable Bayesian neural nets with rank-1 factors. In ICML, 2020.
Stanislav Fort, Huiyi Hu, and Balaji Lakshminarayanan. Deep Ensembles: A Loss Landscape
Perspective. arXiv preprint arXiv:1912.02757, 2019.
Yarin Gal and Zoubin Ghahramani. Dropout Bayesian approximation: Representing model uncertainty in deep learning. In International Conference on Machine Learning, 2016.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On Calibration of Modern Neural Networks. In International Conference Machine Learning (ICML), volume es.LG. Comell University Library, August 2017. URL http: / /awiiv.iivvv..///ia//aaa/aaaaa org/ abs /abbs/100 1766..76000 04599v2. regulariza-
Hongyu Guo, Yongyi Mao, and Richong Zhang. Mixup as locally linear tion. In AAAI, 2018
Hongyu Guo, Yongyi Mao, and Richong Zhang, Augmenting data with mixup for sentence classifica- tion: An empirical study, arXiv preprint arXiv: MrXiiv:900.000 09051, 2019.
10

---

Published as a conference paper at ICLR 2021
Danijar Hafher, Dustin Tran, Alex Irpan, Timothy Lillicrap, and James Davidson. Reliable uncertainty estimates in deep neural networks using noise contrastive priors. arXiv preprint 1.Xiv:8807,092880 2018.
Lars Kai Hansen and Péter Salamon. Neural network ensembles. IEEE Trans. Pattern Anal. Mach. Intell., 12:993-1001, 1990.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Computer Vision and Pattern Recognition, 2016.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness common corruptions and perturbations. In International Conference on Learning Representations, 2019a. . net. URL https: /ttps:///ppperee http:///neeeeeee.ooooooooaaaaaaa wtw.:/mmn----/2tiiccccpe.o
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In International Conference on Learning Representations, 2019b.
Dan Hendrycks, Norman Mu, Ekin D. Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshmi- narayanan. Augmix: A simple data processing method to improve robustness and uncertainty. ArXiv, abs/1912.02781, 2020.
Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q. Weinberger. Deep networks with stochastic depth. ECCV, 2016.
Gao Huang, Yixuan Li, Gcoff Pleiss, Zhuang Liu, John E Hopcroft, and Kilian Q Weinberger. Snapshot ensembles: Train get for free. arXiv preprint arXiv:1704.00100 2017.
Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson Averaging weights leads to wider optima and better generalization. In Uncertainty in Artificial Intelligence, 2018.
Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolu- tional neural networks. In Neural Information Processing Systems, pages 1097-1105, 2012.
Anders Krogh and Jesper Vedelsby. Neural network ensembles, cross validation, and active learning. In Advances in neural information processing systems, pages 231-238, 1995.
Ananya Kumar, Percy Liang, and Tengyu Ma. Verified uncertainty calibration. ArXiv, abs/1909. 2019.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Neural Information Processing Systems, 2017.
Puneet Mangla, Vedant Singh, Shreyas Jayant Havaldar, and Vineeth N. Balasubramanian. Varmixup: Exploiting the latent space for robust training and inference. ArXiv, abs/2003.0655 2020.
Mahdi Pakdaman Naeini, Gregory F. Cooper, and Milos Hauskrecht. Obtaining Well Calibrated Probabilities Using Bayesian Binning. In AAAI Conference on Artificial Intelligence, volume 2015, pages 2901-2907, January 2015. URL https: www.ncbi... mlmm nih. gov/pmc////// articles 4PC 00 009000000000000000000000 / i 6 91.9999..pm. .ddf.
Jeremy Nixon, Mike Dusenberry, Linchuan Zhang, Ghassen Jerfel, and Dustin Tran. Measuring Calibration in Deep Learning. arXiv:1904.0l 0..69....6.5111 [cs, stat), April 2019. URL http:://aaaat org/abs / 904. 01 685.
Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, D Sculley, Sebastian Nowozin, Joshua V Dillon, Balaji Lakshminarayanan, and Jasper Snock. Can you trust your model's uncertainty? Evaluating predictive uncertainty under dataset shift. In Neural Information Processing Systems, 2019.
Tianyu Pang, Kun Xu, and Jun Zhu. Mixup inference: Better exploiting mixup to defend adversarial attacks. ArXiv, abs/1909 11515, 2020.
11

---

Published as a conference paper at ICLR 2021
Michael P. Perrone and Leon N. Cooper. When networks disagree: Ensemble methods for hybrid neural networks. 1992.
Yao Qin, Xuezhi Wang, Alex Beutel, and Ed Huai hsin Chi. Improving uncertainty estimates through the relationship with adversarial robustness. ArXiv, abs/2006. 16375, 2020.
Rahul Rahaman and Alexandre H Thiery. Uncertainty quantification and deep ensembles. arXiv
preprint arXiv:2007.08792, 2020.
Ryne Roady, T. Hayes, and Christopher Kanan. Improved robustness to open set inputs via tempered mixup. ArXiv, abs/2009.04659 2020.
Alejandro Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. FitNets: Hints for thin deep nets. CoRR, abs/ 0tos//441..5550 2015.
David Ruppert. Efficient estimations from a slowly convergent Robbins- Monro process. Technical report, Cornell University Operations Research and Industrial Engineering, 1988.
Takuya Shimada, Shoichiro Yamaguchi, Kohei Hayashi, and Sosuke Kobayashi. Data interpolating prediction: Alternative interpretation of mixup. ArXiv, abs/1906.08412, 2019.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1):1929-199) 1958, 2014.
Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber. Highway networks. CORR,
abs/1505.00387, 2015.
Iyya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. In Neural Information Processing Systems, 2014.
Sunil Thulasidasan, Gopinath Chennupati, Jeff A Bilmes, Tanmoy Bhattacharya, and Sarah Michalak.
On mixup training: Improved calibration and predictive uncertainty for deep neural networks, In 2019a.
Advances in Neural Information Processing Systems, pages
Sunil Thulasidasan. Gopinath Chennupati, Jeff A. Bilmes, Tanmoy Bhattacharya, and Sarah Ellen Michalak. On mixup training: Improved calibration and predictive uncertainty for deep neural networks. In NeurlPS, 2019b.
Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey J Gordon. An empirical study of example forgetting during deep neural network learning. arXiv preprint arXiv: 1812.05159, 2018
Juozas Vaicenavicius, D. Widmann, Carl R. Andersson, F. Lindsten, J. Roll, and Thomas Bo Schön. Evaluating model calibration in classification. In AISTATS, 2019.
Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, David Lopez-Paz, and Yoshua Bengio. Manifold mixup: Better representations by interpolating hidden states. In International Conference on Machine Learning, pages 6438-6447. PMLR, 2019.
Yeming Wen, Dustin Tran, and Jimmy Ba. BatchEnsemble: An alternative approach to efficient ensemble and lifelong learning. In International Conference on Learning Representations, 2020.
Florian Wenzel, Jasper Snock, Dustin Tran, and Rodolphe Jenatton. Hyperparameter ensembles for robustness and uncertainty quantification. In Neural Information Processing Systems, 2020.
D. Widmann, F. Lindsten, and D. Zachariah. Calibration tests in multi-class classification: A unifying framework. ArXiv, abs/1910.1 1385, 2019.
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE International Conference Computer Vision, pages 6023-6032, 2019.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint arXiv:1605.07144 2016.
Hongyi Zhang, Moustapha Cissé, Yann Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. ArXiv, 1b0/1700.09412, 2018.
12

---

Published as a conference paper at ICLR 2021
A DATASET DETAILS
CIFAR & CIFAR-C: We consider two CIFAR datasets, CIFAR-10 and CIFAR-100 (Krizhevsky, 2009). Each consists of a training set of size 50K and test set of size 10K. They are natural images with 32x32 pixels. Each class has 5,000 training images and 500 training images on CIFAR-10 and CIFAR- 100 respectively. In our experiments, we follow the standard data pre-processing schemes including zero-padding with 4 pixels on each sise, random crop and horizon flip (Romero et al., 2015; Huang et al., 2016; Srivastava et al., 2015). If a training method requires validation dataset such as CAMixup, we separate 2, 500 images from 50K training images as the validation set.
It's important to test whether models are well calibrated under distribution shift. CIFAR-IO 10 corruption dataset (Hendrycks and Dietterich, 2019a) is designed to accomplish this. The dataset consists of 15 types of corruptions to the images. Each corruption types have 5 intensities. Thus, in total CIFAR-IOC has 75 corrupted datasets. Notice that the corrupted dataset is used as a testset without training on it. Ovadia et al. (2019) benchmarked a number of methods on CIFAR-10 corruption. Similarly, we can apply the same corruptions CIFAR-I00 dataset obtain CIFAR-1006
ImageNet & ImageNet-CC: We used the ILSVRC 2012 classification dataset (Deng et al., 2009) which consists of a total of 1.2 million training images, 50,000 validation images and 150,000 testing images. Images span over 1,000 classes. We follow the data augmentation scheme in He et al. (2016), such as random crop and random flip, to preprocess the training images. During testing time, we apply a 224x224 center crop to images. Similarly to CIFAR-C, we apply 15 corruption types with 5 intensities each to obtain ImageNet-C (Hendrycks and Dietterich, 2019b).
B HYPERPARAMETERS
SECTION 3
We kept the same set of hyperparameters as the BatchEnsemble model in Wen et al. (2020). All hyperparameters can be found in Table 3. The most sensitive hyperparameter we found is whether to use ensemble batch norm, which applies separate batch norm layer for each ensemble member; and the value of random_ candom_sign __int,, which controls the standard deviation of Gaussian distributed initialization of and r. We kept BatchEnsemble CIFAR-10 10 the same as Wen et al. (2020), which does not deploy ensemble batch norm. We enable ensemble batch norm on CIFAR- 100 and ImageNet. This allows us to use larger standard deviation in the initialization. The random siig_sig_n__n init is -0.5 on CIFAR-10 and -0.75 on CIFAR-100 and -0.75 on ImageNet. In the code, we use negative value to denote the standard deviation of Gaussian distribution (positive value instead initializes with Bernoulli distribution under that probability). In our case, we only use negative sandom_signnjnne which means we only consider Gaussian distributed initialization in this work.
sign_init
Dataset
CIFAR-I0
CIFAR- 100
ensemble_sin size base gase__earninngaaa _rae
4
0.1
per_core_ jer_core_batchhh batch_ size
64
num cores num_cores
8
Ir Ir_decay_rate raaio
0.1
train_epochs
250
Ir decaycy_oppe _eecchs
[80, 160, 200]
12
0.0001
0.0003
random sign init
0.5
0.75
SyncEnsemble BN
False
Truc
Table 3: Hyperparameters we used in Section 3 regarding to BatchEnsemble. The difference between CIFAR-10 and CIFAR-100 is 12, random_sisgn and whether to use SyncEnnemble_BH BN.
C EXCESSIVE UNDER-CONFIDER UNDER-CONFIDENCE ON SYNTHETIC DATA
To further understand the confidence surface of Mixup + Ensembles, we provided a visualization in Fig. 8. We trained on a synthetic dataset consisting of 5 clusters, each with different radius. We ensemble over 4 independently trained copies of 3-layer MLPs. We plotted the softmax probabilities surface of Mixup-Single model, Desp-Ensemble and MixupeEnsemble. The softmax probabilities
13

---

Published as a conference paper at ICLR 2021
34
20
-30
-40
-20
20
-2
(a) Synthetic Data
(b) Mixup-Single
(c) Deep--nsemmlee
(d) Mixup-Ensemble
Figure 8: Softmax probabilities surface of different ensemble methods (ensemble size 4) in the input space after training on synthetic data. Deep ensemble over-confident in the area around origin.
Mixup-Ensemble leads to gloabl under-confidencc.
0.0075
22
Accuracy- Confidence
0.0025
50
18
0.0000
16
46
-0.0050
BatchEnsemble MixupBE
42
-0.0150
CAMixupBE
10
0.166 03338 0.6511 0.715 27527 a,505 09178
Confidence (max prob)
0.9255
MC
BE
MC
(a) Reliability CIFAR100
(b) CIFAR F00-Error (%)
(c) CIFAR100-ECE (%)
Figure 9: Left: Reliability diagrams on CIFAR-100 with a WideResNet 28-10. Our proposed
CAMixup successfully fixes the under-confidence of Mixup BatchEnsemble, leading better calibra-
tion. (b) & (c): Red: Ensembles without Mixup; Blue: Ensembles with Mixup; Green: Our proposed CAMixup does not harm the out-of-distribttion performance.
represent the model confidence. Fig. 8c shows that confident except at the decision boundaries. Fig. 8b displays a lower confidence than Deep-Ensemble. This is beneficial in the single model context because single deep neural networks tend to be over-confident and Mixup can partially correct this bias. On the other hand, Fig. 8d shows that Mixup- Ensemble is only confident in very constrained area around the training clusters, leading to an overall under-confident classifier which confirms our postulation compounding under-confidence.
predictions are extremely
D MORE CALIBRATION RESULTS OF BAKUP-BACCHENSEMBLL.
In Section 3.1, we demonstrated that combining Mixup and ensembles leads to worse calibration on testset. this appendix section, we complement the above conclusion with analysis on corrupted datasets and with data-augmentation techniques like AugMix.
D.1 SUPPLEMENTARY RESULTS ON CAMIXUP
In this section, we provided supplementary results on CAMixup. Fig. 2 shows that combining Mixup and BatchEnsemble leads to excessive under-confident In Fig. 9a, we showed that our proposed CAMixup fixes this issue by correcting the confidence bias. This explains why CAMixup achieves better calibration in-distribution testset. As demonstrated in Section 4.2, Mixup improves model out-off df--iiitibittion performance because of its strong regularization effect. We showed that our proposed CAMixup inherits Mixup's 's improvement CIFAR-10-C Fig. 9b and Fig. 9c show that this conclusion seamlessly transfers to CIFAR-100-C. We also supplement Fig. 5 with Table 4 and Table 5, illusrating detailed numbers.
14

---

Published as a conference paper at ICLR 2021
BatchEnsemble
0.0015
MixupBE
Accuracy- Confidence
AugMixBE
Accuracy Confiidncce
Big
0.0025
Aug CAMixupBE
BatchEnsemble
-0.0050
MixupBE
-0.0075
AugMixBE
AugMixupBE
-0.01.0
AugCAMIxupBE
-0.000
0.5522 0.5818
0,6500 0.7199 0,7815 0.8400 0.8179 Confidence (max prob)
0.9922
0.5244
0.5838 04511 0.7163 0.7927 2,600 0.175 2.9565
Confidence (max prob)
(a) Reliability CIFARIO
(b) Reliability CIFARIO0
Figure 10: Reliability diagrams on CIFAR-10 and CIFAR-100. Both plots show that AugMix does lead under-confidence when combined with ensembles. However, we combine AugMix with Mixup (AugMixup), the compounding under-confidence issue still exists, leading to suboptimal calibration. Our proposed AugCAMixup corrects this underconfidence bias.
D.2 SUPPLEMENTARY RESULTS ON AUGMIX
We show that Mixup does not combine with ensembles without sacrificing in in-distribution calibration in Section 3.1. As discussed in Section 2.3, AugMix only uses does not modify the labels. Intuitively, it does not reduce model confidence. We support this intuition with Fig. 10. It shows that AugMix does not lead to under-confidence. Therefore it can be combined with ensembles without any calibration issue.
transformations and
In Table 2, showed that combining AugMix and Mixup leads to worse calibration due to the under-confidence although AugMix itself does not. To better understand the insights beyond staring at scalars, provided the reliability diagram analysis confidence issue of AugMixup (Augmix + Mixup) still exists. It suggests that applying CAMixup Augmix can correct the under-confidence bias as what we showed Fig. 10a and Fig. 10b. Our proposed CAMixup allows to compound performance of ensembles and data augmentation to achieve the best possible performance.
well. In Figure 10, showed that the under-
CIFAR-10
Method/Metric
Acc(t)
ECE())
cA/CE
BatchEnsemble
96.22 =0.07
1.8 +0.2
77.5 /12.9 +1.2 %
Mixup BE
96.98+0.08
6.4 +0.4%
10.0::..49933:11 %
CAMixup BE
96.94 =0.10
1.2 +0.2 %
81.1+0.44 /9.7
Table CIFAR-10 results for Wide ResNet-28-10 BatchEnsemble (Wen al., 2020) (BE), averaged
over 5 seeds. This table is used supplement Fig. 5.
CIFAR-100
Method/Metric
Acc(1)
ECE(L)
cA/cE
BatchEnsemble
81.85+0.09
2.8+0.1%
54.110.3/19.110.889
Mixup BE
83.12+40.08
9.7++0.5 %
59.3+00.38..
CAMixup BE
83.02+00.10
2.31
39.7+0.3/8.9
Table 5: CIFAR- 100 results for Wide ResNet-28-10 BatchEnsemble (Wen et al., 2020) (BE), averaged
over seeds. This table is used supplement Fig. 5.
15

---

Published as a conference paper at ICLR 2021
Dataset
CIFAR-10
CIFAR-100
Metric
Acc(t)
ECE())
cA/cR
Acc(t)
ECE())
cA/cR
Deep Ensembles
96.66
0.78%
76.80/9.8%
82.7
2.1 %
54.1/13.8%
Mixup DE
97.11
6.15%
83.33/8.0%
83.90
9.42%
61.02/8.9%
CAMixup DE
96.95
1.92%
83.01/4.4%
83.68
5.22%
59. 59.18/8.6%
AugMix DE
97.39
0.59%
89.50/3.3% %
84.15
5.13%
68.21/6.7%
AugMixup DE
97.56
2.71% %
90.03/4.3%
84.85
6.86%
69.31/7.66
AugCAMixup DE
97.48
1.89%
89.94/4.7%
84.64
5.29%
69.19/5.9% %
on deep ensembles. We can conclude that Mixup worsens ensemble predictions in deep ensembles well in BatchEnsemble. This suggests we can use CAMixup on deep ensembles as well. However, the improvement is not as obvious as it is on Batchens leading to the fact that Aux is the most calibrated (in- and out-of-distribution) data augmentation strategy on deep ensembles.
Table 6:
E DEEP ENSEMBLES WITH MIXUP
We showed that CAMixup improves Mixup Ensemble calibration testset without undermining its calibration under distribution shift in Section 4. In this section, we show that the improvement can also be observed on deep ensembles. In Fig. 11, we showed the under-confidence bias we observed on Mixup+ Basebble also exists on Mixup + deep ensembles, with an even more obvious trend. Beyond commonly used ECE measure, we also explore other calibration measures. They further confirmed our under-confidence intuition. We provide some brief explanation on how to calculate ACE, SCE and TACE.
ACE measure is the same as ECE except for the binning scheme. Rather than equally divide the confidence evenly into several bins, ACE choses an adaptive scheme which spaces the bin intervals so that each contains an equal number of predictions. SCE is the same ECE except that it accounts for all classes into calibration measure rather than just looking at the class with maximum probability, The softmax predictions induce infinitesimal probabilities. These tiny predictions can wash out the calibration score. TACE is proposed to set a threshold to only include predictions with large predictive probability, to address the above issue.
We present the results of Mixup, CAMixup, AugMix, AugMixup and AugCAM on deep ensembles in Table 6. We notice that the improvement of CAMixup on deep ensembles is smaller than its improvement on Batchhnsemble. We postulate that this because Mixup deep ensembles much badly calibrated than Mixup BatchEnsemble, For example, AugMixup deep ensembles achieve 2.71% and 6.86% ECE on CIFAR-IO and CIFAR-106 In the meanwhile, AugMixup + BatchEnsemble achieve 1.71% and 4. 19%. Thus, even if CAMixup can improve the calibration of Mixup deep ensembles, it still cannot beat AugMix + deep ensembles. As a result, when we say we close the calibration gap between BatchEnsemble and deep ensembles, we are comparing AugCAMixup BatchEnsemble (BatchEnsembll + CAMixup Augmix) to AugMix deep ensembles. This is because AugMix deep ensembles achieve the best calibration among all variants we tried. How to completely fix the under-confidence in deep ensembles is natural extension of this work. Since we focus on bridging the calibration gap between BatchEnsemble and deep ensembles, we delegate the complete fix in deep ensembles to the future work.
F METRICS OTHER THAN ECE
ECE is the standard metric in calibration, but it a biased estimate of true calibration (Vaicenavicius et al., 2019). Heavily relying ECE metric might lead to inconsistent conclusion. In this section, we computed the calibration eгтor with recently proposed calibration estimator which reduces bias in ECE, including debiased calibration estimator (Kumar et al., 2019) (DCE) and SKCE (Widmann et al., 2019). fig. 12 shows that our conclusion in the main section are also supported by these two recently proposed calibration estimators. In particular, the improvement of proposed CAMixup over
16

---

Published as a conference paper at ICLR 2021
1.0
3000
342
44:
-0000
3350
6.000
02300
3000
$..00
1.002
3760
(a) Reliability on testset.
140
340
M4
-0.00
1.000
20:00
3.50
(b) Reliability on corrupt level
-000
-000
60R
-3000
(c) Reliability on corrupt level 3.
3400
-0000
-0.00
317
E00
5C50
(d) Reliability on corrupt level
Figure 11: WideResNet-28-111 10 Deep Ensembles with Mixup CIFAR-10, We plotted the reliability diagram ensemble and individual predictions. Besides ECE, we also plotted other calibration metrics such ACE, SCE and TACE proposed in Nixon et al. (2019). All metrics verify the conclusion that Mixup + Ensembles leads to mnder-confidence on testset.
17

---

Published as a conference paper at ICLR 2021
BatchEnsemble
Method/Metric
Acc(1)
SKCE(L)
cA/SSKEE
Acc(1)
SKCE(L)
cA/eSKCE
Vanilla
96.22
3.4e-4
77.5/0.026
96.66
3.4e-5
54.1/0.018
Mixup
96.98
4e-3
80.0/0.024
97.11
4.4e-3
59.3/0.0068
CAMixup
96.94
1.3e-4
81.1/0.019
96.95
4.3e-4
59.7/0.0032
Table Results for Wide ResNet-28--Ue BatchEnsemble (Wen al., 2020) and Deep Ensembles on CIFAR-10 10 and CIFAR-10-C, averaged over 3 seeds. This table is used to supplement Fig. 12
Mixup on testset is even larger than what ECE reflects Fig. 5. Table 7 demonstrates the specific numbers used in Fig. 12.
2.5
0.4
2.0
0.3
5
1.5
0.2
1.0
(a) CIFARIO-SKCH SKCE (%) (b) CIFARIO-DCE (%) (c) CIFARIO-C-SKCE (%) (d) CIFARIO-C-DCE (%)
Figure 12:
WideResNet 28-10 on CIFAR- 10 and CIFAR-IO-C, averaged over 3 random seeds. SKCE:
Squared kernel calibration error computed Widmann al. (2019). DCE: Debiased calibration emoor in Kumar et al. (2019). Red: Ensembles without Mixup: Blue: Ensembles with Mixup; Green: Ensembles with CAMixup (ours). Both SKCE and DCE give consistent rankings on calibration error to the ranking in Fig. 5 and Fig. 6. This plot shows that our proposed CAMixup is effective in reducing Mixup calibration error when combined with ensembles.
G CAMIXUP WITH TEMPERATURE SCALING
See Fig. 13.
18

---

Published as a conference paper at ICLR 2021
H LIMITATIONS AND FUTURE WORK
We describe limitations of our work, signalling areas for future research. One limitation CAMixup is that all examples in the same class still share the same Mixup coefficient. This leaves room for developing more fine- aine-grained adaptive Mixup mechanisms, such as adapting the Mixup coefficient per example. This relates data point given a deep network? (Toneva et al., 2018; 8; Agarwal and Hooker, 2020) Another limitation showed that CAMixup still cannot fully fix the miscalibration of Mixup + deep ensembles in Appendix E, due to the fact that Mixup deep ensembles leads to even worse calibration than Mixup + BatchEnsemble. This raises harder question which CAMixup cannot completely solve but also leaves more research room to further understand why Mixup is worse on deep ensembles and how to address it. Thus, we leave the question on how address the above issues to future work. Next, we determine whether to use Mixup based on the reliability (Mean Accuracy Mean Confidence) of each class on validation set. One concern that CAMixup does not scale well a large number of classes. Fortunately, we showed that this works problems up to 1000 classes (ImageNet). Additionally, Mixup has been most successful the vision domain, hence our focus; and with preliminary success on tabular data and natural language processing (Zhang et al., 2018; Guo et al., 2019). Assessing whether CAMixup and ensembling techniques translate to text is an interesting area.
open research question: how do you measure the training difficulty of
We took first step in developing more fine-grn ine-grained adaptive Mixup mechanism. Recall that class based CAMixup calculates the reliability (Aceuracy Con- fidence) at the end of each epoch, then it decided whether to apply Mixup in each class (illustrated in Fig. 4). This requires extra computation on valida- tion dataset and it assigns uniform Mixup coefficient within one class. By leveraging recently developed forgetting count (Toneva et al., 2018), we can ad- just Mixup coefficient for each example based on its forgetting counts. The intuition associated with high forgetting counts, it indicates the model tends to forget this example. To achieve better calibration, we should place low confidence example. The algorithm of forgetting counts based CAMixup presented in Algorithm 1. In summary, we first calculate the forgetting counts for each train- ing example and obtain the median of these counts as the threshold. Then, CAMixup applies Mixup to the training example whose forgetting counts are higher than the median.
Algorithm CAMixup
Forgetting
Count Based
initialize prevacc; 0, E D initialize forgetting 7 = 0, E initialize MixupCoeffij while training do
B D # sample minibatch Apply Mixup on B based on MixupCoeff
an examples
for example; E B do compute acc; if prevace, > acc; then
this
Tii = T[] + prevacc, acc,
end end for gradient update classifier on B rank sort(T) threshold = [aak( for example; E do
We provided a preliminary results on CIFAR-10 in Fig. 14. It demonstrates that forgetting counts based CAMixup outperforms class based CAMixup on most metrics across BatchEnsemble and MC-dropout. One exception that underperforms on test calibra- tion MC-dropout. We could not observe the same improvement on CIFAR-100. We postulate that the reliability of forgetting count CIFAR-100 is not good as it is on CIFAR-10, leading to the inconsistent results. We leave the question how improve forgeting count based CAMixup on CIFAR- into future work.
if T > threshold then MixupCoef[[i] =
else MixupCoeff(i end if end for end while
19

---

Published as a conference paper at ICLR 2021
Valicaton Eror
Vardudion NLL
Vuldziion ECE
24
636
22
28
0:32
2.
8:38
Text [S]
345
6:37
340
6:36
2.45
1.50
3.2
CAMBADD
OURbu
T5: OMMbup
Comppted
6:12
18.4
18.6
6.64
(a) CIFAR-10
0.50
8:58
111
1:64
11.
146
8:64
CM
GPP
Text Emoo
Teat NLL
Test E02
33
15:33
1.5
5.4
18.35
18
Corrupted Eror
Comppted
Compugte
14
232
228
643
10
T5-CMPop
CAMBUU
TS+
Grtbpp
TS+
(b) CIFAR-100
Figure 13: Combining CAMixup and Temperature Scaling further improves test ECE, does not calibration however.
make further improvements
20

---

Published as a conference paper at ICLR 2021
3.6
25
1.2
3.5
24
1.1
18
1.0
3.4
16
0.9
22
3.3
0.8
21
0.7
0.6
3.1
19
BE
MC
BE
MC
BE
MC
BE
MC
(a) CIFAR10-Eroo (%)
(b) CIFARIO-C-Emor (%)
(c) CIFARIO-ECE (%)
(d) CIFAR1O-C-ECE (%)
Figure 14: WideResNet 28-10 on CIFAR-IO and CIFAR-IO-C. Green: Class based CAMixup. Purple:
Forgetting count based CAMixup. Forgetting count based CAMixup outperforms class based Mixup most metrics across BatchEnsemble and MC-dropout.
21


Reviews:
[{"review_id": "wXP-qwALGvW", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\n- This paper found that combining ensembles and data augmentation can harm model calibration. Inspired by this finding, it proposes a simple correction by only applying mixup to certain classes. Empirical experiments show some improvements. \n\nPros:\n- The paper is clearly written and easy to follow. The motivation and approach is intuitive.\n- Experiments are conducted on both small and large datasets.\n\nCons:\n- The findings do not seem too surprising to me. The vanilla deep neural networks are often in the over-confident regime, so either ensemble or mixup itself seems very effective in improving model calibration. But applying these two together might lead the model into the under-fitting regime. \n- It occurs to me that the proposed AugCAMixup is not good enough. From Table 2 we can tell that it basically sacrifices final accuracy with improvement in ECE. This means that applying mixup to all the classes is more preferrable than applyting to a subset of the classes in terms of accuracy. Why don't the authors consider using rescaling to fix the under confident issue?\n- The baselines compared in this submission seem too limited. Some rescaling based methods should be considered, for instance, augmixup + rescaling.\n- How's the performance on metrics other than ECE? I think it is important to include other metrics as ECE can hide some problems.\n \n------\npost-rebuttal update\n\nI appreciate the authors for the responses. While the other reviewers give high reviews about this manuscript, I would keep my original reveiw for the following reasons. 1) This manuscript is incremental in nature. I agree with R3 that \"understanding which techniques can be combined and which cannot (and how to fix it) is important for developing the field and feels like a small step to the right direction\", but a somewhat unsurprising result lacks technical novelty. In one of the responses the authors wrote \"The spirit of this work is to point out that not all data augmentations can be combined well with ensembles.\" I think this potentially means that the conclusion the authors made on mixup could not even transfer to other data augmentations. This even further limits the contribution of this manuscript. So I guess the above argument from R3 is not convincing, or one could try to study the effect of batch norm and ensemble and wrote another good paper. 2) The empirical performance of the method is limited and thus whether the proposed method is useful is a question.", "extra_scores": {}, "extra_text": {}}, {"review_id": "upSsI5_wrEo", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "After the discussion, my concerns were fixed. The paper explores the interesting relations of Mix Up and Uncertainty, which is useful and will be the right fit for the conference.\n\n**Summary:**\n\nThe work studies how a better calibration of individual members of an ensemble affects the calibration of an ensemble. It is demonstrated that i) better calibration of individual members of the ensemble may lead to the worse calibration of the ensemble predictions ii) this is the case when mix-up / label smoothing are used during training. \n\nTo fix the issue Confidence Adjusted mixup Ensembles (CAMixup) is proposed. The CAMixup is an adaptive mixup data augmentation based on per-class calibration criteria. The core idea of CAMixup is to use powerful (unconfidence encouraging) mixup data-augmentation on examples of overconfident classes, and do not use mixup for the under-confident classes. The confidence criteria are computed once in an epoch.\n\nThe empirical results are provided in- and out-of-domain for CIFARs(C) and ImageNet(C).\n\n**The concerns:**\n\n1) ECE is a biased estimate of true calibration with a different bias for each model, so it is not a valid metric to compare even models trained on the same data [Vaicenavicius2019]. In other words, the measured ECE has no guaranty to have something to do with the real calibration but reflects the bias of the measured metric. Other metrics, that are based on histogram estimates have the same problem. Please put extra attention to this concern.\n\nWhat I suggest is the following:\n\na. Mesure NLL for in- and out-of-domain data. It seems to be still an adequate (indirect) criterion of calibration, and is an adequate criterion of uncertainty estimation. According to [Ashukha2020], the NLL needs a pre-calibrated model with temperature scaling for in-domain data (called calibrated NLL / calibrated LL). \n\nb. To use the squared kernel calibration error (SKCE) proposed in [Widmann2019] along with de facto standard, but biased ECE. The SKCE is an unbiased estimate of calibration. There might be some pitfalls of this metric that I'm not aware of, but the paper looks solid and convincing. Also, please put attention to Figure 83 in the arХiv version.\n\nYes, ECE is the standard in the field, but it is the wrong standard that prevents us from meaningful scientific progress, so we should stop using it.\n\n2) The standard deviation needs to be reported everywhere. Especially the differences between close values like (97.52, 97.52, 97.47 in Table 2) may appear not statistically significant. The same touches Fig 5 and other figures that are reported. Otherwise, it is impossible to stay how solid results are.\n\n**Minor comments:**\n\n1) Maybe it worth to provide plot mean-λi vs epoch to illustrate this \"Notice that λi is dynamically updated at the end of each epoch.\"?\n\n2) Figure 4(a) is done slightly disorderly.\n\n3) In the paper, ECE is measured in percentages. As far as I can tell ECE is dimensionless quantities. It is not clear what is intended.\n\n**Final comment:** I put the \"marginally below acceptance threshold\" score, but I'm willing to increase it after the update with corrections (and hope that these corrections will be done).  I like the direction and CAMixup, but in-domain results are not very consistent (see Fig 5 (d)), the ECE has uncontrollable model-specific biases that ruin all the presented results. \n\n[Widmann2019] Widmann D, Lindsten F, Zachariah D. Calibration tests in multi-class classification: A unifying framework. In Advances in Neural Information Processing Systems 2019 (pp. 12257-12267). https://arxiv.org/pdf/1910.11385.pdf\n\n[Ashukha2020] Ashukha A, Lyzhov A, Molchanov D, Vetrov D. Pitfalls of in-domain uncertainty estimation and ensembling in deep learning. ICLR, 2020.", "extra_scores": {}, "extra_text": {}}, {"review_id": "2cjCLhgp4Mt", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##########################################################################\nSummary: \n\nThe paper identifies the negative effects on calibration and the robustness of the deep models when data augmentation and the ensembles are combined. The paper proposes a technique that mitigates this drawback and improves the calibration and robustness across benchmarks.\n\n##########################################################################\n\nReasons for score: \n \nOverall, I vote for accepting. The observations in the paper and the improvements to the state-of-the-art are certainly very encouraging. My minor concerns are on clarity of the writing, references at some places. Hopefully the authors will address these concerns in the rebuttal period. \n##########################################################################\n\n\nPros:\n\n+ Overall, the paper is well written, easy to follow and understandable. \n\n+ The paper is positioned in the body of the literature and does contribute to further improve the state-of-the-art.\n\n+ The pathological effect of augmentations combined with ensembles is an interesting observation. The proposed solution does improve the state-of-the-art both for in-distribution and out-of-distribution data.\n\n+ In section 3.2 the label smoothing experiment is interesting. \n\n+ The additional experiments in Appendix and the discussion on limitations and potential for future directions is useful and much appreciated.\n\nCons:\n\n- My first major concern is omitting the key references when the terms are introduced for the first time or some of the assertions. Some of them are listed below. \n\n- “Ensemble methods are a simple approach to improve a model’s calibration and robustness.” Who found this? This statement requires a citation.\n\n- “For example, the majority of uncertainty models in vision ...” requires a citation.\n\n- “Deep Ensembles” -- need citation, etc. Please fix all such instances\n\n- “and Hendrycks et al. (2020) highlights further improved results in AugMix when combined with Deep Ensembles. However, we find their complementary benefits are not universally true.” From the above sentence, it is not clear what the findings of Hendrycks et al is and for which of their findings the authors in this paper are contradicting with.  I know the paper is about robustness and uncertainty and the authors refer to that. Be clear in the presentation, think that these papers will be read by ML enthusiasts.\n\n- “... poor calibration of combining ensembles and Mixup on CIFAR”, here it is worth to introduce the kind of ensemble(s) that are used in combining.\n\n- DeepEnsembles, BatchEnsemble, MC-Dropout each of them have different ensemble sizes in the experiments. What is the reasoning behind those hyperparameter selections?\n\n- The calibration pathology is observed for ensembles with mixup, is it true for other data augmentations such as rotation, cropping, etc?\n\n- “Ensembles are the among the most known and simple ...” this sentence doesn’t flow well.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "1FMBcl27PS", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work analyses the interaction between data-augmentation strategies such as MixUp and model ensembles with regards to calibration performance. The authors note how strategies such as mixup and label smoothing, which reduce a single model's over-confidence, lead to degradation in calibration performance when such models are combined as an ensemble. Specifically, all techniques, taken individually, improve calibration by reducing overconfidence. However, in combination they lead to under-confident models and, therefore, worse calibration. Based on this analysis, the author's provide a simple technique which yields SOTA calibration performance on CIFAR-10, CIFAR-10-C, CIFAR-100 and CIFAR-100-C and ImageNet. The authors propose to dynamically enable and disable MixUp based on whether the model is over/under confident on a particular class, as judged on a validation dataset. \n\nI think this work provides useful insight and a simple and effective solution. Additionally, it is clearly written and very easy and pleasant to read.\n\nThe authors may find this concurrent work on ensemble calibration to be relevant: https://openreview.net/forum?id=wTWLfuDkvKp\n\nThe only question is have is why the authors think that models are overconfident on hard examples/classes and properly calibrated on easy ones. My intuition would be the opposite - that models are overconfident in areas where there are too few training examples and/or areas where there is no data uncertainty. If there is no data uncertainty then overconfidence would not be a problem. So mainly the issue is due to data sparsity in areas of non-zero data uncertainty. Would be good to expand the discussion. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper analyses the interaction between data-augmentation strategies  and model ensembles with regards to calibration performance. The authors note how strategies such as mixup and label smoothing, which reduce a single model's over-confidence, lead to degradation in calibration performance when such models are combined as an ensemble. They propose a simple solution. The paper merits publication.
Decision: Accept (Poster)