AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty
Authors: Dan Hendrycks*
Norman Mu*
Ekin Dogus Cubuk
Barret Zoph
Justin Gilmer
Balaji Lakshminarayanan
Venue:  (2020)
Keywords: robustness
uncertainty

Abstract: Modern deep neural networks can achieve high accuracy when the training distribution and test distribution are identically distributed, but this assumption is frequently violated in practice. When the train and test distributions are mismatched, accuracy can plummet. Currently there are few techniques that improve robustness to unforeseen data shifts encountered during deployment. In this work, we propose a technique to improve the robustness and uncertainty estimates of image classifiers. We propose AugMix, a data processing technique that is simple to implement, adds limited computational overhead, and helps models withstand unforeseen corruptions. AugMix significantly improves robustness and uncertainty measures on challenging image classification benchmarks, closing the gap between previous methods and the best possible performance in some cases by more than half. 

Published as a conference paper at ICLR 2020
A SIMPLE DATA PROCESSING METHOD TO IMPROVE ROBUSTNESS AND UNCERTAINTY
AUGMIX:
Dan Hendrycks
Norman Mu"
Ekin D. Cubuk Google
DeepMind
Google
hendrycks@bbrrkeeeey.ee
normanmuggoog com
cubukégoogla com
Barret Zoph
Justin Gilmer
Balaji Lakshminarayanani DeepMind
Google
Google gilmerr@ooogle.com
larzetzcph@google.comm
com
ABSTRACT
Modern deep neural networks can achieve high accuracy when the training distribu- tion and test distribution are identically distributed, but this assumption is frequently violated in practice. When the train and test distributions are mismatched, accu- racy can plummet. Currently there few techniques that improve robustness unforeseen data shifts encountered during deployment. In this work, we propose a technique improve the robustness and uncertainty estimates of image classifiers. We propose AUGMIX, a data processing technique that is simple to implement, adds limited computational overhead, and helps models withstand unforeseen cor- ruptions. AUGMIX significantly improves robustness and uncertainty measures on challenging image classification benchmarks, closing the gap between previous methods and the best possible performance in some cases by more than half.
INTRODUCTION
Current machine learning models depend on the ability of training data to faithfully represent the data encountered during deployment. In practice, data distributions evolve (Lipton et al., 2018), models encounter new scenarios (Hendrycks & Gimpel, 2017), and data curation procedures may capture only narrow slice of the underlying data distribution (Torralba & Efros, 2011). Mismatches between the train and test data are commonplace, yet the study of this problem is not. As it stands, models do robustly generalize across shifts in the data distribution. models could identify when they are likely to be mistaken, or estimate uncertainty accurately, then the impact of such fragility might be ameliorated. Unfortunately, modern models already produce overconfident predictions when the train- ing examples are independent and identically distributed to the test distribution. This overconfidence and miscalibration is greatly exacerbated by mismatched training and testing distributions.
Small corruptions to the data distribution are enough to subvert existing classifiers, and techniques to improve corruption robustness remain few in number. Hendrycks & Dietterich (2019) show that classi- fication error of modern models rises from 22% on the usual ImageNet test set to 64% on ImageNet-C. a test set consisting of various corruptions applied to ImageNet test images. Even methods which aim to explicitly quantify uncertainty, such probabilistic and Bayesian neural networks, struggle under data shift, as recently demonstrated by Ovadia et al. (2019). Improving performance in this setting has been difficult. One reason is that training against corruptions only encourages networks to memorize the specific corruptions seen during training and leaves models unable to generalize to new corruptions (Vasiljevic et al., 2016; Geirhos al., 2018). Further, networks trained on translation augmentations remain highly sensitive to images shifted by single pixel (Gu et al., 2019; Hendrycks & Dietterich, 2019). Others have proposed aggressive data augmentation schemes (Cubuk et al., 2018), though at the cost of a computational increase. Chun et al. (2019) demonstrates that many techniques may improve clean accuracy at the cost of robustness while many techniques which improve robustness harm uncertainty, and contrariwise. In all, existing techniques have considerable trade-offs.
Equal Contribution. Corresponding author.

---

Published as a conference paper at ICLR 2020
MixUp
AugMix
Figure I: A visual comparison of data augmentation techniques. AUGMIX produces images with variety while preserving much of the image semantics and local statistics.
In this work, we propose a technique to improve both the robustness and uncertainty estimates of classifiers under data shift. We propose AUGMIX, a method which simultaneously achieves new state-of-the-art results for robustness and uncertainty estimation while maintaining or improving accuracy on standard benchmark datasets. AUGMIX utilizes stochasticity and diverse augmentations, a Consen-Shannor Divergence consistency loss, and a formulation to mix multiple augmented im- ages to achieve state-of-the art performance. On CIFAR-10 10 and CIFAR- 100, method roughly halves the corruption robustness error of standard training procedures from 28.4% to 12.4% and 54.3% to 37.8% error, respectively. On ImageNet, AUGMIX also achieves state-of oftth--artt corrup- tion robustness and decreases perturbation instability from 57.2% to 37.4%. Code is available at https:
woog//oooleeeeoooooooooooooooee wwoogll-rreearchhe research/ /awggiaa.......//
2 RELATED WORK
Robustness under Data Shift. Geirhos et al. (2018) show that training against distortions often fail generalize to unseen distortions, as networks have a tendency to memorize properties of the specific training distortion. Vasiljevic al. (2016) show training with various blur augmentations can fail generalize to unseen blurs or blurs with different parameter settings. Hendrycks & Dietterich (2019) propose measuring generalization to unseen corruptions and provide benehmarks for doing so. Kang et al. (2019) construct an adversarial ver- sion of the aforementioned benchmark. Gilmer et al. (2018) 8); Gilmer & Hendrycks (2019) argue that robustness data shift a pressing problem which greatly affects the reliability of real-world machine learning systems.
ImageNet-C Corruptions
Calibration under Data Shift. Guo et al. (2017); Nguyen & O' Connor (2015) propose metrics for determining the cal- ibration machine learning models. Lakshminarayanan et al. (2017) find that simply ensembling classifier predictions im- proves prediction calibration. Hendrycks et al. (2019a) show that pre-training can also improve calibration. Ovadia al. (2019) demonstrate that model calibration substantially dete- riorates under data shift.
Data Augmentation. Data augmentation can greatly improve generalization performance. For image data, random left-right flipping and cropping are commonly used He et al. (2015). Random occlusion techniques such as Cutout can also improve accuracy on clean data (Devries& & Taylor, 2017: Zhong et al., 2017). Rather than occluding a portion of an image, CutMix replaces portion of image with portion of different image (Yun et al., 2019). Mixup also uses information from two images. Rather than implanting one portion of an image inside another, Mixup produces elementwise convex combination of two images (Zhang et al., 2017; Tokozume et al.,
Figure 2: Example ImageNet-C cor- ruptions. These corruptions are en- countered only at time and not during training.
2

---

Published as a conference paper at ICLR 2020
Figure 3: A cascade of successive compositions can produce images which drift far from the original image, and lead to unrealistic images. However, this divergence can be balanced by controlling the number steps. increase variety, we generate multiple augmented images and mix them.
2018). Guo et al. (2019) show that Mixup can be improved with adaptive mixing policy, as to prevent manifold intrusion. Separate from these approaches are learned augmentation methods such Augmmngment (Cubuk al., 2018), where group of augmentations is tuned to optimize performance on a downstream task. Patch Gaussian augments data with Gaussian noise applied to randomly chosen portion of image (Lopes et al., 2019). A popular way make networks robust to Ep adversarial examples is with adversarial training (Madry et al.. 2018), which we use in this paper. However, this tends increase training time by an order of magnitude and substantially degrades accuracy on non-adversarial images (Raghunathan et al., 2019).
3 AUGMIX
AUGMIX is data augmentation technique which improves model robustness and uncertainty esti- mates, and slots in easily existing training pipelines. high level, AugMix is characterized by its utilization of simple augmentation operations in concert with a consistency loss. These augmentation operations are sampled stochastically and layered produce high diversity of augmented images. We then enforce a consistent embedding by the classifier across diverse augmentations of the same input image through the use Sonsen-Shannor divergence as a consistency loss.
Mixing augmentations allows to generate diverse transformations, which are important for inducing robustness, as a common failure mode of deep models in the arena of corruption robustness the memorization of fixed augmentations (Vasiljevic et al., 2016; Geirhos et al., 2018). Previous methods have attempted to increase diversity by directly composing augmentation primitives in a chain, but this can cause the image quickly degrade and drift off the data manifold, as depicted in Figure 3. Such image degradation can be mitigated and the augmentation diversity can be maintained by mixing together the results of several augmentation chains in convex combinations, A concrete account of the algorithm is given in the pscudocode below.
Algorithm AUGMIX Pseudocode
1: Input: Model p. Classification Loss L, Image Tocig Operations O {rotate,.... .... posterize)
2: function AugmentAnndMix(Taag.. h: = 3, a 1)
3: Fill Tang with zeros
4: Sample mixing weights (W1, W,...., wk) Dirichlet(a,, a, a.......
5: for L,..., do
6:
Sample operations оp, op2, оP ~ O
7:
Compose operations with varying depth op12 op, o op and op = op3 op2 o op.
8: Sample uniformly from one of these operations chain ~ {op,, O1 OP123
9:
Taug += w; chain(
> Addition is elementwise
10:
end for
11: Sample weight m ~ Beta(a, a)
12: Interpolate with rule Taugmix MZZrrg + (1 m))
13: return Taugmix
14: end function
15: Fangmixi = AugmentAAndMiix(rraan
b Fangmiiit is stochastically generated
Fangmix2 AugmentAndMix((ret)
D Tauggiial Touggii22
17: Loss Output: C(p(y) Torg), v) + À Jensen- Sareen-Shanoo(((( Wong)) p(y) Faugmix1) p(y) Faugmixz)
3

---

Published as a conference paper at ICLR 2020
shear_y
translate
Xaria
M,-00.1 -0.12 12
X
Xass
w,-0.2
1-m=0.8
rolate
W. Mr sa
posterize
mw0.2
equalize
posterize
Figure 4: A realization of AUGMIX. Augmentation operations such as translate x and weights such as m are randomly sampled. Randomly sampled operations and their compositions allow to explore the semantically equivalent input space around an image. Mixing these images together produces a new image without veering too far from the original.
Augmentations. Our method consists of mixing the results from augmentation chains composi- tions of augmentation operations. We use operations from AutoAugment. Each operation is visualized in Appendix C. Crucially, we exclude operations which overlap with ImageNet- C corruptions. In par- ticular, we remove the contrast, color, brightness, sharpness, and Cutout operations so that our set of operations and the ImageNet-C corruptions are disjoint. turn, we do not use any image noising nor image blurring operations so that ImageNet-C corruptions are encountered only at test time. Operations such as rotate can be realized with varying severities, like 2 or -15 For operations with varying severities, we uniformly sample the severity upon each application. Next, we randomly sample augmentation chains, where constructed by composing from one to three randomly selected augmentation operations.
by default. Each augmentation chain is
Mixing. The resulting images from these augmentation chains are combined by mixing. While considered mixing by alpha compositing, we chose use elementwise convex combinations for simplicity. The d-dimensional vector of convex coefficients randomly sampled from a Marichlet(a,..... ax) distribution. Once these images are mixed, we combine the result of the augmentation chain and the original image through a second random convex combination sampled from a Beta(a, a) distribution. The final image incorporates several sources of randomness from the choice of operations, the severity of these operations, the lengths of the augmentation chains, and the mixing weights.
"skip connection" to
Jensen-Shannon Divergence Consistency Loss. We couple with this augmentation scheme loss that enforces smoother neural network responses. Since the semantic content of an image is approximately preserved with AUGMIX. we should like the model to embed Forig, Tamgmixt Fangmix2 similarly. Toward this end, we minimize the Sensen-Shannon divergence among the posterior distributions of the original sample Forig and its augmented variants. That is, for Porig p(y Torgg); Paugmixt = ply Tamgmix1); Paugmix2 = plyl (amgmix)), we replace the original loss L with the loss
C(Porig. y) + A (S Paugmix1 Paugmix2).
(1)
To interpret this loss, imagine sample from one of the three distributions Porig Paugmixl Paugmix2. The gensen-Shannon divergence can be understood to measure the average information that the sample reveals about the identity of the distribution from which it was sampled.
This loss can be computed by first obtaining M (Porig + Paugmixt + Paugmix2) /3 and then computing
KL (ponggl MM] + KL [Paggmmmix || [M] + KL (pmggniis M.)
IS (porig P Paugmix2)
(2)
Unlike an arbitrary KL Divergence between Porig and Paugmix the Jensen Censen-Shannon divergence is upper bounded, in this case by the logarithm of the number of classes. Note that we could in- stead compute JS (ponig Paugmixt.. though this does not perform as well. The gain of training with JS( (Patigi Paugmix1: ; Paugmix2' Paugmix3) is marginal. The model to be stable, consistent, and insensitive across to a diverse range of inputs (Zheng et al., 2016; Kannan al., 2018: Xie et al., 2019). Ablations are in Section 4.3 and Appendix A.
Consistency Loss impels to
4

---

Published as a conference paper at ICLR 2020
4 EXPERIMENTS
Datasets. The two CIFAR (Krizhevsky & Hinton, 2009) datasets contain small 32 32 3 color natural images, both with 50,000 training images and 10,000 testing images. CIFAR-1O has 10 categories, and CIFAR-100 has 100. The ImageNer (Deng et al., 2009) dataset contains 1,000 classes of approximately 1.2 million large-scale color images.
In order to measure model's resilience data shift, we evaluate on the CIFAR-IO-C, CIFAR-I00 C, and ImageNer-CC C' datasets (Hendrycks & Dietterich, 2019). These datasets are constructed by corrupting the original CIFAR and ImageNet test sets. For each dataset, there are a total of 15 noise, blur, weather, and digital corruption types, each appearing at 5 severity levels or intensities. Since these datasets are used to measure network behavior under data shift, we take care not to introduce these 15 corruptions into the training procedure.
The CIFAR-IO-P. CIFAR-100-P, and ImageNet-P datasets also modify the original CIFAR and ImageNet datasets. These datasets contain smaller perturbations than CIFAR-C and are used to measure the classifier's prediction stability. Each example in these datasets is a video. For instance, a video with the brightness perturbation shows image getting progressively brighter over time. We should like the network not to give inconsistent or volatile predictions between frames of the video as the brightness increases. Thus these datasets enable the measurement of the "jaggedness" (Azulay & Weiss, 2018) of a network's prediction stream.
Metrics. The Clean Error is the usual classification error on the clean uncorrupted test data. In experiments, corrupted test data appears at five different intensities or severity levels 1 < & S 5. For a given corruption c, the error rate at corruption severity is E..,. We can compute the average error across these severities to create the unnormalized corruption error uCE, = E Ec, On CIFAR-10 10- C Cand CIFAR-100-C we average these values over all 15 corruptions. Meanwhile, on ImageNet we fol- low the convention of normalizing the corruption error by the corruption error of AlexNet (Krizhevsky EAlexNet The average of the 15 corruption et al., 2012). We compute CE, Es 1=1 Ee.s/ 5 Ea=l c.e
errors CEGassiin Noise CEshot Noise CEpiselate CESPEG gives us the Mean Corruption Error (mCE).
Perturbation robustness is not measured by accuracy but whether video frame predictions match. Consequently we compute what called the flip probability. Concretely, for videos such as those with steadily increasing brightness, we determine the probability that two adjacent frames, or two frames with slightly different brightness levels, have "flipped" or mismatched predictions. There are 10 different perturbation types, and the mean across these is the mean Flip Probability (mFP). As with ImageNet-C, we can normalize by AlexNet's flip probabilities and obtain the mean Flip Rate (mFR).
In order to assess a model's uncertainty estimates, we measure its miscalibration. Classifiers capable of reliably forecasting their accuracy are considered "calibrated." For instance, a calibrated classifier should be correct 70% of the time on examples to which it assigns 70% confidence. Let the classifier's s confidence that its prediction Y is correct be written C. Then the idealized RMS Calibration Error is
E ( ýlc c) - c) which is the squared difference between the accuracy at given confidence level and actual the confidence level. In Appendix E, we show how to empirically estimate this quantity and calculate the Brier Score.
4.1 CIFAR- AND CIFAR-100
Training Setup. In the following experiments we show that AUGMIX endows robustness to various architectures including an All Convolutional Network (Springenberg et al., 2014: Salimans & Kingma, 2016), DenseNet-BC (k = 12, d = 100) (Huang et al.. 2017), a 40-2 Wide ResNet (Zagoruyko & Komodakis, 2016), and a ResNeXt-29 (32 x 4) (Xie et al., 2016). All networks use an initial learning rate of 0.1 which decays following a cosine learning rate (Loshchilov & Hutter, 2016 6). All input images are pre-processed with standard random left-right flipping and cropping prior to any augmentations. We do not change AUGMIX parameters across CIFAR-10 and CIFAR- experiments for consistency. The All Convolutional Network and Wide ResNet train for 100 epochs, and the DenseNet and ResNeXt require 200 epochs for convergence. We optimize with stochastic gradient descent using Nesterov momentum. Following Zhang et al. (2017); Guo et al. (2019), we use weight decay of 0.0001 for Mixup and 0.0005 otherwise.
5

---

Published as a conference paper at ICLR 2020
CIFAR-10-C Error Various Methods
30
Standard Clean Error
25
20
Efror
10
Standard
Cutout
Mixup
CutMixx
AutoAugment*
Adv. Training
AugMix
Figure 5: Error rates of various methods on CIFAR- -O-C using ResNeXt backbone. Observe that AUGMIX halves the error rate of prior methods and approaches the clean error rate.
Standard Cutout Mixup CutMix AutoAugment* Adv Training AUGMIX
AlIConvNet
30.8
32.9
24.6
31.3
29.2
28.1
15.0
DenseNet
30.7
32.1
24.6
33.5
26.6
27.6
12.7
CIFAR-1O-C
WideResNet
26.9
26.8
22.3
27.1
23,9
26.2
11.2
ResNeXt
27.5
28.9
22.6
29.5
24.2
27.0
10,9
Mean
29.0
30.2
23.5
30.3
26.0
12.5
AllConvNet
56.4
56.8
53.4
56.0
55.1
56.0
42.7
DenseNet
59.3
59.6
55.4
59.2
53.9
55.2
39.6
CIFAR-100-C
WideResNet
53.5
50.4
52.9
35,9
ResNeXt
53.4
54.6
51.4
54.1
51.3
54.4
34,9
Mean
55.6
56.1
52.6
55.5
52.5
55.2
38.3
Table 1: Average classification error percentages. Across several architectures, AUGMIX obtains CIFAR- 10-C and CIFAR-100-C corruption robustness that exceeds the previous state the art.
Results. Simply mixing random augmentations and using the Jensen-Shannon loss substantially improves robustness and uncertainty estimates. Compared to the "Standard" data augmentation baseline ResNeXt on CIFAR-10-C, AUGMIX achieves 16.6% lower absolute corruption error as shown in Figure 5. In addition to surpassing numerous other data augmentation techniques, Table demonstrates that these gains directly transfer across architectures and CIFAR-100-C with zero additional tuning. Crucially, the robustness gains do not only exist when measured in aggregate. Figure 12 shows that AUGMIX improves corruption robustness across every individual corruption and severity level. Our method additionally achieves the lowest mFP on CIFAR- 11FAR-10-P across three different models all while maintaining accuracy on clean CIFAR-10, shown in Figure 6 (left) and Table 6. Finally, we demonstrate that AUGMIX improves the RMS calibration error on CIFAR-I0 10 and CIFAR-IO-C, as shown in Figure (right) and Table 5, Expanded CIFAR-IO- and calibration results are in Appendix D, and Fourier Sensitivity analysis is in Appendix B.
Robustness and CIFAR-10-P Data
Calibration on Clean and Corrupted Data
20
CIFAR-10
RMS Calibration Error (%)
CIFAR-10-C
4
Mean Flip Probability
15
10
Standard
CutMix
AugMix
Standard
CutMix
AugMix
Figure 6: CIFAR-10-P prediction stability and Root Mean Square Calibration Error values for
Rt.NeXt. AUGMIX simultaneously reduces flip probabilities and calibration error.

---

Published as a conference paper at ICLR 2020
4.2 IMAGENET
Baselines. To demonstrate the utility of AUGMIX on ImageNet, we compare to many techniques designed for large-scale images. While techniques such as Cutout (Devries Taylor, 2017) have not been demonstrated to help on the ImageNet scale, and while few have had success training adversarially robust models on ImageNet (Engstrom et al., 2018), other techniques such as Stylized ImageNet have been demonstrated to help on ImageNet-C. Patch Uniform (Lopes et al., 2019) is similar to Cutout except that randomly chosen regions of the image are injected with uniform noise; the original paper uses Gaussian noise, but that appears in the ImageNet-C test set noise. We tune Patch Uniform over 30 hyperparameter settings. Next, AutoAugment (Cubuk et al., 2018) searches over data augmentation policies find a nigh-performing data augmentation policy. We denote AutoAugment results with Augmnntm since we remove augmentation operations that overlap with ImageNet-C corruptions, as with AUGMIX. We also test with Random AuloAngmett* an augmentation scheme where each image has a randomly sampled augmentation policy using AutoAugment* operations. In contrast to Augmnnn,ent, Random AutoAugment* and AUGMIX require far less computation and provide more augmentation variety, which can offset their lack of optimization. Note that Random AutoAugment* is different from RandAugment introduced recently by Cubuk et al. (2019): RandAugment uses AutoAugment operations and optimizes single distortion magnitude hyperparameter for all operations, while Random AutoAugment* randomly samples magnitudes for each operation and uses the same operations AUGMIX. MaxBlur Pooling (Zhang, 2019) is a recently proposed architectural modification which smooths the results of pooling. Now, Stylized ImageNet (SIN) is technique where models are trained with the original ImageNet images and also ImageNet images with style transfer applied. Whereas the original Stylized ImageNet technique pretrains on ImageNet-C and performs style transfer with a content loss coefficient of O and a style loss coefficient of 1, we find that using 0.5 content and style loss coefficients decreases the mCE by 0.6%. Later, we show that SIN and AUGMIX can be combined. All models are trained from scratch, except MaxBlur Pooling models which has trained models available.
uniform
Training Setup. Methods are trained with ResNet-50 and we follow the standard training scheme of Goyal et al. (2017), in which we linearly scale the learning rate with the batch size, and use learning rate warm-up for the first 5 epochs, and Autonut and AUGMIX train for 180 epochs. All input images are first pre-processed with standard random cropping horizontal mirroring.
Noise
Blur
Weather
Digital Elastic Pixel JPEG mCE
Network
Clean
Gauss. Shol Impulse Detocus Glass Moiisn Zaui
Snow
Frost
Fog
Bright Cuntrast
Standard
23.9
79
80
82
82
90
84
80
86
81
75
65
79
91
77
80
80.6
Parch Uniform
24.5
67
68
83
77
80
75
77
84
71
AutoAugmens (AA)
22.8
69
68
72
77
83
80
81
79
75
64
56
70
88
57
71
72.7
Random AA
23.6
70
71
72
80
86
81
81
77
72
61
75
88
73
76.1
MaxBlur pool
23.0
73
74
76
74
86
77
77
63
56
68
86
71
73.4
SIN
27.2
69
70
70
77
84
76
82
74
75
69
65
69
80
64
77
73.3
AUGMIX
66
75
67
69
AUGMIX+SIN
25.2
61
62
61
69
77
63
72
66
68
63
59
52
74
60
67
64.9
Table 2: Clean Error, Corruption Error (CE), and mCE values for various methods ImageNet-C. C. The mCE value is computed by averaging across all 15 CE values. AUGMIX reduces corruption error while improving clean accuracy, and it can be combined with SIN for greater corruption robustness.
Results. Our method achieves 68.4% mCE as shown in Table 2, down from the baseline 80.6% mCE. Additionally, note that AUGMIX allows straightforward stacking with other methods such as SIN to achieve an even lower corruption error of 64.1 1 % mCF. Other techniques such as Automngtent* require much tuning, while ours does not. Across increasing severities of corruptions, our method also produces much more calibrated predictions measured by both the Brier Score and RMS Calibration Error as shown in Figure 7. As shown in Table 3, AUGMIX also achieves a state-of-the art result on ImageNet-P P at with an mFR of 37. 7..4 down from 57.2 77.2%. We demonstrate that scaling up AUGMIX from CIFAR to ImageNet also leads ofate-oftthe-eee art results in robustness and uncertainty estimation.
4.3 ABLATIONS
We locate the utility of AUGMIX in three factors: training set diversity, our Shannnn di- vergence consistency loss, and mixing. Improving training set diversity via increased variety of augmentations can greatly improve robustness. For instance, augmenting each example with
7

---

Published as a conference paper at ICLR 2020
Noise
Blur
Weather
Digital
Network
Clean
Gaussian Shot
Motion
Zoom
Snow
Bright
Translate
Rotate
Tilt Scale
mFR
Standard
23.9
57
55
62
65
66
65
43
53
57
49
57.2
Patch Uniform
24.5
32
25
50
52
54
57
40
48
49
46
45.3
AutoAugment* (AA)
22.8
50
45
40
44
50
51.7
Random AA*
23.6
53
46
53
63
59
57
42
48
54
47
52.2
SIN
27.2
53
50
57
72
51
62
43
53
57
53
55,0
MaxBlur pool
23.0
52
51
59
63
57
64
34
43
49
40
51.2
AUGMIX
22.4
46
41
32
35
37.4
AUGMIX+SIN
25.2
45
40
54
32
48
27
35
38
39
38.9
Table 3: ImageNet-P results. The mean flipping rate is the average of the flipping rates across all 10 perturbation types. AUGMIX improves perturbation stability by approximately 20%.
ImageNet
ImageNet Call bration Across Severities
ImageNet Calibration Across Severitie
100
80
Standard
Standard
70
AugMix
Standard (Ensemble)
20-
Stardard (Ersemble)
AugMix
AugNix
Standard (Ensemble)
AugMix (Ensemble)
Aughtix
Aughix (Ensembli)
60
Brier Score
70
50
2.0
60
50
30
20
0
0
4
Corruption Severity
Corruption Severity
Corruption Severity
Figure Uncertainty results on ImageNet-C. Observe that under severe data shifts, the RMS calibration enror with ensembles and AUGMIX is remarkably steady. Even though classification emor increases, calibration roughly preserved. Severity zero denotes clean data.
randomly sampled augmentation chain decreases the error rate of Wide ResNet on CIFAR- 10-C from 26.9% 17.0% Table Adding the -nsen--Shannon divergence consistency loss drops error rate further to 14.7%. Mixing random augmentations without the Jenson-Shanne divergence loss gives us an error rate of 13.1%. Finally, re-introducing the Jensen-Shannon divergence gives us AUGMIX with error rate of 11.2%. Note that adding even more mixing is not necessarily beneficial. For error rate 13.3%, possibly due to an
instance, applying AUGMIX on top of Mixup increases increased chance of manifold intrusion (Guo et al., 2019). Hence AUGMIX's 's careful combination of variety, consistency loss, and mixing explain its performance.
Method
CIFAR 10-C Error Rate
C10AR-100-C Error Rate
Standard
26.9
53.3
AutöAAumment
23,99
49.6
Random AutoAnggmnt**
17.0
43,6
Random AutoAugment* JSD Loss
14.7
40.8
AugmentAndMix (No ISD Loss)
13.1
39.8
AUGMIX (Mixing JSD Loss)
11.2
35.9
Table 4: Ablating components of AUGMIX on CIFAR- and CIFAR- Variety through randomness, the Jensen-Shannon divergence (JSD) loss, and augmentation mixing confer robustness.
5 CONCLUSION
AUGMIX data processing technique which mixes randomly generated augmentations and uses a Jensen-Shannon loss to enforce consistency. Our simple-to-mmplemen technique obtains state-of-the-ant performance on CIFAR-10/100-C, ImageNet-C, CIFAR-10/100-1 and ImageNet-P. AUGMIX models achieve state-of-the-ar calibration and can maintain calibration even the distribution shifts. We hope that AUGMIX will enable more reliable models, necessity for models deployed in safety-critical environments.
8

---

Published as a conference paper at ICLR 2020
REFERENCES
Aharon Azulay and Yair Weiss. Why do deep convolutional networks generalize poorly small image transformations? arXiv preprint, 2018.
Sanghyuk Chun, Seong Joon Oh, Sangdoo Yun, Dongyoon Han, Junsuk Choe, and Youngjoon Yoo. empirical evaluation on robustness and uncertainty regularization methods. ICML. Workshop on Uncertainty and Robustness in Deep Learning. 2019.
Ekin Cubuk, Barret Zoph, Jonathon Shlens, and Quoe V Le. Randaugment: Practical automated data augmentation with reduced search space. arXiv preprint arXiv:1909.137/99 2019.
Ekin Dogus Cubuk, Barret Zoph, Dandelion Mané, Vijay Vasudevan, and Quoc V. Le. AutoAugment:
Learning augmentation policies from data. CVPR, 2018.
Jia Deng, Wei Dong, Richard Socher, Li jia Li, Kai Li, and Fei-Fei. ImageNet: A large-scale hierarchical image database. CVPR, 2009.
Terrance Devries and Graham W. Taylor. Improved regularization of convolutional neural networks with Cutout. arXiv preprint arXiv: 0ete:/70.0055555 2017.
Logan Engstrom, Andrew Ilyas, and Anish Athalye. Evaluating and understanding the robustness of adversarial logit pairing. arXiv preprint, 2018.
Robert Geirhos, Carlos R. M. Temme, Jonas Rauber, Heiko H. Schütt, Matthias Bethge, and Felix A. Wichmann. Generalisation in humans and deep neural networks. NeurIPS, 2018.
Justin Gilmer and Dan Hendrycks. A discussion of adversarial examples are not bugs, they are features': : Adversarial example researchers need to expand what is meant by' by'robustsess'. Distill, (8)=e00019-1, 2019.
2019.
Justin Gilmer, Ryan P. Adams, Ian J. Goodfellow, David Andersen, and George E. Dahl. Motivating the rules of the game for adversarial example research. CoRR, hbbs/180..06732 2018.
Priya Goyal, Piotr Dollár, Ross B. Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch SGD: Training ImageNet in hour. CoRR, 70s/0777.,,,77,, 2017.
Keren Gu, Brandon Yang. Jiquan Ngiam, Quoe Le, and Jonathon Shlens. Using videos to evaluate image model robustness, 2019.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. ICML, 2017.
Hongyu Guo, Yongyi Mao, and Richong Zhang. Mixup as locally linear out-of-manifold regulariza- tion. In AAAI, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recogmition. CVPR, 2015.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. ICLR, 2019.
Dan Hendrycks and Kevin Gimpel. baseline for detecting misclassified and out-of-distrriution examples in neural networks. ICLR, 2017.
Dan Hendrycks, Kimin Lee, and Mantas Mazeika. Using pre-training can improve model robustness and uncertainty. In ICML, 2019a
Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. ICLR, 2019b.
Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In CVPR, 2017.

---

Published as a conference paper at ICLR 2020
Daniel Kang, Yi Sun, Dan Hendrycks, Tom Brown, and Jacob Steinhardt. Testing robustness against unforeseen adversaries. arXiv preprint, 2019.
Harini Kannan, Alexey Kurakin, and lan Goodfellow. Adversarial logit pairing. NewrlPS, 2018.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. ImageNet classification with deep convolu- tional neural networks. NeurlPS, 2012.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In NewrIPS, 2017.
Zachary Chase Lipton, Yu-Xiang Wang, and Alexander J. Smola. Detecting and correcting for label shift with black box predictors. ArXiv, abs/1 802.03916, 2018.
Raphael Gontijo Lopes, Dong Yin, Ben Poole, Justin Gilmer, and Ekin Dogus Cubuk. Improv- ing robustness without sacrificing accuracy with patch Gaussian augmentation. arXiv preprint 2019.
Hya Loshchilov and Frank Hutter. SGDR: stochastic gradient descent with warm restarts. ICLR, 2016.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant adversarial attacks. ICLR, 2018.
Khanh Nguyen and Brendan O'Connor. Posterior calibration and exploratory analysis for natural language processing models. EMNLP, 2015.
Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, D Sculley, Sebastian Nowozin, Joshua V Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model's uncertainty? Evaluating predictive uncertainty under dataset shift. NeurIPS, 2019.
Aditi Raghunathan, Sang Michael Xie, Fanny Yang, John C. Duchi, and Percy Liang, Adversarial training can hurt generalization. arXiv preprint 1rXiv:1906.0600 2019.
Tim Salimans and Diederik Kingma. Weight normalization: simple reparameterization to accelerate training of deep neural networks. NeurlPS, 2016
Jost Tobias Springenberg. Alexey Dosovitskiy, Thomas Brox, and Martin A. Riedmiller. Striving for simplicity: The all convolutional net. CoRR, abs/1412.6806, 2014.
Yuji Tokozume, Yoshitaka Ushiku, and Tatsuya Harada. Between-class learning for image classifica- tion. CVPR, 2018.
Antonio Torralba and Alexei A. Efros. Unbiased look at dataset bias. CVPR, 2011.
Igor Vasiljevic, Ayan Chakrabarti, and Gregory Shakhnarovich. Examining the impact of blur on recognition by convolutional networks, 2016.
Qizhe Xie, Zihang Dai, Eduard Hovy, Minh-Thang Luong, and Quoc Le. Unsupervised data augmentation. arXiv preprint arXiv: 1904.12494 2019.
Saining Xie, Ross Girshick, Piotr Dollr, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. CVPR, 2016.
Dong Yin, Raphael Gontijo Lopes, Jonathon Shlens, Ekin D Cubuk, and Justin Gilmer. A Fourier perspective model robustness in computer vision. arXiv preprint arXiv: neee::900000999 2019.
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. ICCV, 2019.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In BMVC, 2016.
Hongyi Zhang, Moustapha Cissé, Yann Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. ICLRR 2017.
10

---

Published as a conference paper at ICLR 2020
Richard Zhang. Making convolutional networks shift-invarian again. In JCML, 2019.
Stephan Zheng, Yang Song, Thomas Leung. and Ian Goodfellow. Improving the robustness of deep neural networks via stability training. CVPR, 2016.
Zhun Zhong, Liang Zheng. Guoliang Kang. Shaozi Li, and Yi Yang. Random erasing data augmenta- tion, arXiv preprint arXiv: 7008 04896, 2017.

---

Published as a conference paper at ICLR 2020
A HYPERPARAMETEE ABLATIONS
In this section demonstrate that AUGMIX's hyperparameters not highly sensitive, that AUGMIX performs reliably without careful tuning. For this set of experiments, the baseline AUGMIX model trains for 90 epochs, has mixing coefficient of 0.5, has examples per Jensen-Shannon Divergence (1 clean image, 2 augmented images), has chain depth stochastically varying from 1 to and has
augmentation chains. Figure 8 shows that the performance of various AUGMIX models with different hyperparameters. Under these hyperparameter changes, the mCE does not change substantially.
AugMix ImageNet-C Ablations
80
ResNet 50 Baseline
MCE (%6
65
90
180 Epochs
0.1
0.5
5
3 Examples in
1
1,2 1,2,3
5
Mixing Coefficient (a)
Chain Depth
Mixture Width (k)
Figure AUGMIX hyperparameter ablations on ImageNet-C. ImageNet-C classificato performance is stable changes to AUGMIX's hyperparameters.
Model Sensitivity Fourier Analysis
Baseline
Cutout
AugMix
100
60
40
20
Figure Fourier Sensitivity Heatmap of the baseline Wide ResNet, Cutout, and AUGMIX CIFAR-10 models. All Fourier basis perturbations are added to clean CIFAR 10 test images. AUGMIX maintains robustness frequencies and is far more robust mid and high frequency modifications. Example perturbed images are shown above, with black pointer lines indicating the Fourier basis vector used to perturb the image. For each basis vector we compute the error rate of the model after perturbing the entire test set.
B FOURIER ANALYSIS
commonly mentioned hypothesis (Gilmer Hendrycks, 2019) for the lack of robustness of deep neural networks is that they readily latch onto spurious high-frequency correlations that exist in the data. In order to better understand the reliance of models to such correlations, we measure model sensitivity additive noise at differing frequencies. We create 32 32 sensitivity heatmap. That is,
12

---

Published as a conference paper at ICLR 2020
add total of 32 32 Fourier basis vectors to the CIFAR-10 test set, the resulting error rate after adding each Fourier basis vector. Each point in the heatmap shows the error rate the CIFAR-10 test set after it has been perturbed by single Fourier basis vector. Points corresponding to low frequency vectors are shown the center of the heatmap, whereas high frequency vectors are farther from the center. For further details Fourier sensitivity analysis, refer the reader to Section 2 of Yin et al. (2019). In Figure 9 we observe that the baseline model is robust low frequency perturbations but severely lacks robustness high frequency perturbations, where error rates exceed 80%. The model trained with Cutout shows similar lack of robustness. In contrast, the model trained with AUGMIX maintains robustness to low frequency perturbations, and on the mid and high frequencies AUGMIX is conspicuously more robust.
time, and record
C AUGMENTATION OPERATIONS
The augmentation operations we use for AUGMIX are shown in Figure 10.
Figure 10: Illustration of augmentation operations applied to the same image. Some operation severities have been increased to show detail.
We do not use augmentations such contrast, color, brightness, sharpness, and Cutout as they may overlap with ImageNet-C test set corruptions. We should note that augmentation choice requires additional care. Guo et al. (2019) show that blithely applying augmentations potentially cause augmented images to take different classes. Figure shows how histogram color swapping augmentation may change bird's class, leading to manifold intrusion.
Manifold Intrusion from Color Augmentation
Original
Augmented
Figure 11: An illustration of manifold intrusion (Guo al., 2019), where histogram color augmenta- tion can change the image's class.
13

---

Published as a conference paper at ICLR 2020
D ADDITIONAL RESULTS
We include various additional results for CIFAR-IO, CIFAR-IO- 10-C and CIFAR-IO-P below. Figure 12 reports accuracy for each corruption, Table 5 reports calibration results for various architectures and Table 6 reports clean error and mFR. We refer to Section 4.1 for details about the architecture and training setup.
CIFAR-10-C Error Rates for Each Corruption
60
Standard
AugMix
Standard
40
Clean Error
Error (%)
30
20
Shot
Nalss
Impuise
Dafocus
Blur
Glass
Blur
Motion
Blur
Zoom BBur
Frost
Fag
Brightiess
Contrast
Elastic
Pivelate
JPEG
Figure 12: AUGMIX improves corruption robustness across all CIFAR-IO-C noise, blur, weather, and
digital corruptions, despite the model never having seen these corruptions during training.
Standard Cutout Mixup
CutMix AutoAugment* Adv Training AUGMIX
NltConvNet
5.4
4.0
12.6
3.1
4.2
11.1
2.2
DenseNet
7.5
6,4
15.6
5.4
6.0
16.2
5.0
CIFAR-10
WideResNet
6.8
3.8
14.0
5.0
4.7
10.7
4.2
ResNeXt
3,0
4.4
13.5
3.5
3.3
5.8
3.0
Mean
5.7
4.7
13.9
4.2
4.6
11.0
3.6
AllConvNet
21.2
21.3
15.4
16.2
10.4
5.2
DenseNet
26.7
27.8
12.9
25.6
21.1
15.0
11.7
CIFAR-IO-C
WideResNet
27.6
19.6
11.1
17.8
17.1
10.6
8.7
ResNeXt
16.4
21.4
11.7
19.6
15.1
11.6
8.3
Mean
23.0
22.5
11.4
19.6
17.4
11.9
8.5
Table 5: RMS Calibration Error of various models and data augmentation methods across CIFAR-10 and CIFAR- 10-C. All values are reported as percentages.
Standard Cutout Mixup CutMix
AutoAugment*
Ady Training AUGMIX
AllConvNet
6.1
6.1
6.3
6.4
6,6
18.9
6.5
DenseNet
5.8
4.8
5.5
5.3
4.8
17.9
4.9
CIFAR-10
WideResNet
5.2
4.4
4.9
4.6
4.8
17.1
4.9
ResNeXt
4.3
4.4
4.2
3.9
15.4
4.2
Mean
5.4
4.9
5.2
5.0
5.0
17.3
5.1
AllConvNet
4.2
5.0
3.9
4.5
4.0
2.0
1.5
DenseNet
5,0
5.7
3.9
6.3
4.8
2.1
1.8
CIFAR-10-P
WideResNet
4.2
4.3
3.4
4.6
4.2
2.2
1.6
ResNeXt
4.0
4.5
3.2
5.2
4.2
2.5
1.5
Mean
4.3
4.9
3.6
5.2
4.3
2.2
1.6
Table CIFAR-10 Clean Error and CIFAR- mean Flip Probability. values are percentages. While adversarial training performs well on CIFAR-I0- 10-P, it induces substantial drop in accuracy (increase in error) on clean CIFAR-10 where AUGMIX does not.
14

---

Published as a conference paper at ICLR 2020
E CALIBRATION METRICS
Due the finite size of empirical test sets, the RMS Calibration Error must be estimated by partitioning all test set examples into b contiguous bins {Bi, .,...., B ordered by prediction confidence. this work we bins which contain 100 predictions, so that we adaptively partition confidence scores on the interval [0. 1] (Nguyen & O'Connor, 2015; Hendrycks et al., 2019b). Other works partition the interval [0, with 15 bins of uniform length (Guo al., 2017). With these bins, we estimate the RMS Calibration Error empirically with the formula
B
1 B
(yk
Σ
Ck
B
kEB,
kEB.
This is separate from classification error because a random classifier with an approximately uniform posterior distribution approximately calibrated. Also note that adding the "refinement" Ec[(P(Y
YIC c) (P(Y- YIc = c))] to the square of the RMS Calibration Error gives us the Brier Score (Nguyen & O'Connor, 2015).
15


Reviews:
[{"review_id": "SJlwxsHKKr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a method called AugMix, which is intended to improve model robustness to data distribution shift. AugMix appears fairly simple to implement. Several new images are created by augmenting an original image through chains of sequentially applied transformations (the \"Aug\" part of AugMix), then the augmented images are combined together, along with the original image, via a weighted sum (the \"Mix\" part of AugMix). Additionally, a Jensen-Shannon Divergence consistency loss is applied during training to encourage the model to make similar predictions for all augmented variations of a single image. This technique is shown to achieve state-of-the-art performance on standard robustness benchmarks without loss of clean test accuracy, and is also shown to improve calibration of model confidence estimates.\n\nOverall, I would tend to vote for accepting this paper. The method is simple yet effective, the paper is very well written and easy to follow, and experiments are extensive and, for the most part, convincing.\n\nQuestions:\n1) The one main concern I have with the training procedure is the amount of time the models were trained for. It is known that model trained with aggressive data augmentation schemes often require much longer training than normal in order to fully benefit from the stronger augmentations. For example, AutoAugment trains ImageNet models for 270 epochs [1], while CutMix trains for 300 epochs [2]. However, the ImageNet experiments in this paper claim to follow the procedure outlined in [3], which only trains for 90 epochs. This is reflected in the clean test accuracy, where AutoAugment only appears to provide a 0.3% gain over standard training, while we might expect 1.3% improvement (according to [2]). The AllConvNet and WideResNet in CIFAR-10 and CIFAR-100 experiments were also trained for only 100 epochs each, where 200 is more conventional. Again this shows in the reported numbers: on WideResNet for CIFAR-10, Mixup only has a 0.3% gain were as we might expect 1% improvement instead [4], and AutoAugment has 0.4% improvement, were as we might expect 1.3% gain if trained longer [1]. My question then is, how much does training time affect results? Do AugMix, and other techniques such as Mixup, CutMix, and AutoAugment, achieve better robustness when models are trained for longer, or do they become more brittle as training time is extended? \n\n2) For the Jensen-Shannon divergence consistency, how much worse does it perform when using JS(Porig;Paugmix1) versus JS(Porig;Paugmix1;Paugmix2)? What might cause this behaviour?\n\n3) Patch Gaussian is changed to Patch Uniform to avoid overlap with corruptions in ImageNet-C. How does Patch Uniform compare to Patch Gaussian in terms of performance for non-Gaussian noise corruptions?\n\n4) How does AugMix perform as an augmentation technique in terms of clean test accuracy compared to other SOTA techniques? Is there a trade-off between clean test accuracy and robustness, or does AugMix improve performance in both domains? Can AugMix be combined with other augmentation techniques or does this destroy robustness properties?\n\nThings to improve the paper that did not impact the score:\n5) It would be nice if the best result in each column could be bolded in Tables 2-4.\n\nReferences:\n[1] Cubuk, Ekin D., Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V. Le. \"Autoaugment: Learning augmentation policies from data.\" CVPR (2019).\n\n[2] Yun, Sangdoo, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. \"Cutmix: Regularization strategy to train strong classifiers with localizable features.\" ICCV (2019).\n\n[3] Goyal, Priya, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. \"Accurate, large minibatch sgd: Training imagenet in 1 hour.\" arXiv preprint arXiv:1706.02677 (2017).\n\n[4] Zhang, Hongyi, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. \"mixup: Beyond empirical risk minimization.\" ICLR (2018).", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "HygBn2yTYS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a novel method called augMix, which creates synthetic samples by mixing multiple augmented images. Coupled with a Jensen-Shannon Divergence consistency loss, the proposed method has been experimentally, using CIFAR10, CIFAR100, and ImageNet, shown to be able to improve over some augmentation methods in terms of robustness and uncertainty.\n \nThe paper is very well written and easy to follow. The idea of the approach is simple, and should be easy to be implemented. The evaluation of the proposed method is currently based on experimental evidence. Nevertheless, the empirical studies in its current form could be further improved. Please see my detailed comments below.\n\n1. The proposed approach relies on a chain of augmented methods. In this sense, experimental studies on the sensitivity for how the augmentation methods in the chain (e.g., augmentation operations) and their chain structure (e.g., length of the chain) impact the performance of the augMix should be provided. This is in particular relevant because the authors did mention that “adding even more mixing is not necessarily beneficial” on page 8.\n\n2. Since the proposed method mixes multiple augmented images, a more appropriate comparison baseline would be a method involving creating synthetic data with multiple images. For example, the n-fold Mixup method as discussed in Guo AAAI2019 (Mixup as Locally Linear Out-Of-Manifold Regularization).\n\n3. Some experimental results/observations deserve further discussions. For example, on page 8, the authors mention that “applying augMix on top of Mixup increases the error rate to 13.3%”. I wonder if the authors could provide any insights or hypothesis on why the proposed model behaviors in this way? \n\n4. Would that be any manifold intrusion issue as discussed in Guo’s AAAI2019 paper? That is, would it be possible to generate images that are very close to some real images but with different labels? For example, by looking at the bottom-center image in the Appendix B, the synthetic image created seems to be close to some birds with other categories. \n\n5. Does the method work for other network architectures such as DenseNet?\n\n\n*********new  comment**********\nDuring the rebuttal period, the paper has been improved with additional experimental results, analyses, and observations. I therefore have adjusted my evaluation score accordingly. \n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "BkloWXh9KB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper discusses a new  data augmentation method which improves the accuracy of the network for several specific shifted domain scenarios. The main goal of the paper is to increase the robustness of the deep model trained on the augmented data to generalize well beyond the data corruption like the  rotation, translation, noise,.... For each input, they apply  $k$ different operation of image shift and make the weighted combination of them. The weight vector is generated randomly from Dirichlet distribution with the parameter $\\alpha$.  The weighted combined images would be added to the original image in convex combination. The convex weights are generated from distribution Beta with parameter $\\beta$. Later they train the network with adding the Jensen-Shannon divergence for the posterior distributions of augmented images as the consistency regularizer.  They show this data augmentation will increase the accuracy of the model for shifted and non-shifted domains and also it leads to more calibrated model for domain shift problem.\n\nPros: \nthe paper is well-written with clear implementation details. The level of experiments are wide and cover different aspects. The experiments shows the significant improvement compared to several baselines. The authors conducted the experiments for a wide range of model-datasets to show the validity of their ideas. \n\nCons: \n1- The title of this work is a strong claim that is not supported in the paper. In this paper, it is mentioned that AugMix is a data augmentation method that generates data to add to the training set and after training with data augmentation, the method would be more robust to other distortions that can be added to the datasets. Generally, the definition of domain shift is wider than just adding perturbation to the dataset.  To support the claim, the paper should also report the results for similar tasks datasets such as CIFAT10-STL10- or MINIST-SVHN for different models and with different domain adaptation methods. The claim about the improvement of uncertainty also is not supported well by the experiments. The method should be tested for many model-datasets specifically, to support improving the  uncertainty under the domain shift idea like the paper [1].  \n\n2- The novelty of the work is limited. The generating method of distorted  images is the combination of previously proposed methods like [2] and [3].  The motivation of why the proposed method is working well is not clear. How this objective function can improve the robustness to the image perturbation but it does not lose the accuracy is not discussed. It would be better if the proposed method were supported by theory and also the intuition and explained why it should get better results than previous data augmentation methods such as AutoAugment [3].\n\n3-  Fine-tuning the parameters like $k$, $\\alpha$ and $\\beta$ is not discussed at all.\n\n4- To show the robustness of the proposed method to domain shift, the paper compares the proposed method to other data augmentation methods that are not designed for domain shift which seems unfair.  \n\nReferences:\n[1] Ovadia, Yaniv, et al. \"Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift.\" arXiv preprint arXiv:1906.02530 (2019).\n[2] Zhang, Hongyi, et al. \"mixup: Beyond empirical risk minimization.\" arXiv preprint arXiv:1710.09412 (2017).\n[3] Cubuk, Ekin D., et al. \"Autoaugment: Learning augmentation policies from data.\" arXiv preprint arXiv:1805.09501 (2018).\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper tackles the problem of learning under data shift, i.e. when the training and testing distributions are different. The authors propose an approach to improve robustness and uncertainty of image classifiers in this situation. The technique uses synthetic samples created by mixing multiple augmented images, in addition to a Jensen-Shannon Divergence consistency loss. Its evaluation is entirely based on experimental evidence.

The method is simple, easy to implement, and effective. Though this is a purely empirical paper, the experiments are extensive and convincing. 

In the end, the reviewers didn't show any objections against this paper. I therefore recommend acceptance.
Decision: Accept (Poster)