AugMax: Adversarial Composition of Random Augmentations for Robust Training
Authors: Haotao Wang
Chaowei Xiao
Jean Kossaifi
Zhiding Yu
Anima Anandkumar
Zhangyang Wang
Venue: NeurIPS 2021 Poster (2021)
Keywords: Model robustness
natural corruptions
data augmentation
adversarial training

Abstract: Data augmentation is a simple yet effective way to improve the robustness of deep neural networks (DNNs). Diversity and hardness are two complementary dimensions of data augmentation to achieve robustness. For example, AugMix explores random compositions of a diverse set of augmentations to enhance broader coverage, while adversarial training generates adversarially hard samples to spot the weakness. Motivated by this, we propose a data augmentation framework, termed AugMax, to unify the two aspects of diversity and hardness. AugMax first randomly samples multiple augmentation operators and then learns an adversarial mixture of the selected operators. Being a stronger form of data augmentation, AugMax leads to a significantly augmented input distribution which makes model training more challenging. To solve this problem, we further design a disentangled normalization module, termed DuBIN (Dual-Batch-and-Instance Normalization), that disentangles the instance-wise feature heterogeneity arising from AugMax. Experiments show that AugMax-DuBIN leads to significantly improved out-of-distribution robustness, outperforming prior arts by 3.03%, 3.49%, 1.82% and 0.71% on CIFAR10-C, CIFAR100-C, Tiny ImageNet-C and ImageNet-C. Codes and pretrained models are available: https://github.com/VITA-Group/AugMax.

AugMax: Adversarial Composition of Random Augmentations for Robust Training
Haotao Wang', Chaowei Xiao Jean Kossaifi Zhiding Yu?
Anima 2 and Zhangyang Wang
Department of Electrical and Computer Engineering, University of Texas at Austin NVIDIA Ariiona State University *California Institute of Technology
[htwang, atlaswang) (rtdswang/uutxxs.eeeee
[chaoweix, jkossaifi, zhidingy, aanandkumar @ nvidia.com
Abstract
Data augmentation is simple yet effective way to improve the robustness of deep neural networks (DNNs). Diversity and hardness are two complementary dimen- sions of data augmentation achieve robustness. For example, AugMix explores random compositions of a diverse set of augmentations to enhance broader cov- erage, while adversarial training generates adversarially hard samples spot the weakness. Motivated by this, we propose a data augmentation framework, termed AugMax, unify the aspects of diversity and hardness. AugMax first randomly samples multiple augmentation operators and then learns an adversarial mixture of the selected operators. Being stronger form of data augmentation, AugMax leads to a significantly augmented input distribution which makes model training more challenging. To solve this problem, we further design a disentangled normalization module, termed DuBIN Hiaal-Bacch-nnd-Ina gles the instance-wise feature heterogeneity arising from AugMax. Experiments show that AugMax-DuBlle DuBIN leads to significantly improved out-of- out-of-distribution robustness, outperforming prior arts by 3.03%, 3.49%, 1. 1.82% and 0.71% on CIFARIO-C, CIFARI00-C, Tiny ImageNet-C and ImageNet-C. Codes and pre- trained models are available: https://github.... / www.o//VVATAT//////o AugMax.
Normalization). that disentan-
Introduction
Out-of-distribution (OOD) samples present a challenge when deploying Al models in the real world. Examples include natural corruptions (e.g., due to camera blurs or noise, snow, rain, or fog in image data), sensory perturbations (e.g., sensor transient error, electromagnetic interference) and domain shifts (e.g., summer + winter). However, deep networks are often trained on limited amounts of data which may not cover sufficient scenarios. As result, they are vulnerable to unforeseen distributional changes despite achieving high performance standard benchmarks [1-3]. This jeopardizes their trustworthiness as well as safe deployment in real-worke environments. Thus critical to develop techniques that improve robustness when training with relatively clean datasets.
Several techniques have been proposed consolidate the robustness against unforeseen corruptions. including robust data augmentation [5, 3, 8, 6), Lipschitz continuity [9-11], stability training [12], pre-training [13-16], and robust network structures [17-19], to name few. Among these techniques, data augmentation is of particular interest due to its empirical effectiveness, ease of implementation, low computational overhead, and plug-and-pe aang-aad nature.
Work partially done during an internship at NVIDIA.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

(b) AugMix (Diversity)
(c) PGD Attack (Hardness)
(d) AugMax (Ours)
(a) Standard
(Diversity & Hardness)
Figure 1: The effects of diversity and hardness in data augmentations. We visualize features of augmented images fed to the network during training. The features are from the penultimate layer of ResNeXt29 trained CIFARIO training data using only standard data augmentation (random flipping and translation) following [4]. For visualization, we randomly selected 300 fixed images from 3 fixed classes (denoted by different colors) which we apply different augmentation methods: (a) standard augmentation (random flipping and translation); (b) AugMix [5]; (c) PGD Attack [6, 7]; (d) AugMax (ours). In order to achieve good model robustness, the augmented training data should both be diverse and also contain enough hard cases, As can be seen, PGD Attack generates hard cases (which the network cannot separate) but are diverse enough (they all clustered together) while AugMix creates diverse but not hard samples (they are well separated). By contrast, our approach (AugMax) achieves unification between hard and diverse samples.
There are mainly two categories of data augmentation approaches:
The first category aims increase diversity of the training data by composing multiple random transformations [20-22, 5, 3J. While standard data augmentation methods (e.g.. random flipping and translation) lead to poor robustness [5], more aggressive combinations of multiple augmentations have shown promise. One such successful example is AugMix [5], which stochastically samples from diverse augmentation operations and randomly mixing them to produce highly diverse augmented images. AugMix can increase the sample diversity coverage compared to standard augmentations, as shown in Figure 1 (a) and (b).
The second category aims to boost the hardness of the training data by sampling from the worst-case augmentations that tries fool the model into misclassite misclassifying the samples. A common technique achieve this goal is adversarial perturbation [6, 23, 8]. Training over such worst-case samples allows model to actively fix its generalization weaknesses [24] and empirically improves its robustness against corruptions [25, 26, 7J. Due to the extra complexity to generate adversarial perturbations, the improved robustness usually comes at the cost of largely increased training time compared with aon-adveraarial methods [6, 27]. An example falling into this category is the PGD attack [6], as is shown in Figure (c).
Previous work has focused on leveraging one of these category to improve robustness. In this paper, unify both approaches in single framework. In particular, are complementary and that a unification between the two is necessary to achieve robustness. We propose strategy to achieve this unification and successfully increase robustness.
show that diversity and hardness
Summary of contributions:
We propose AugMax, novel augmentation framework which achieves robustness through unification between diversity and hardness, by searching for the worst-case mixing strategy.
Being a stronger form of data augmentation, AugMax leads to a significantly augmented and more heterogeneous input distribution, which also makes model training more challenging. To solve this problem, we design new normalization strategy, termed DuBIN, to disentangle the instance-wise feature heterogeneity of AugMax samples.
We show that combination of AugMax and DuBIN (AugMax- achieves otate-of-the-eee the-art robustness against corruptions and improves robustness against other common distribution shifts.
AugMax achieves good unification between sample diversity and hard comer-case generation during data augmentation. Aag is built on top of the AugMix framework [5] which mixes multiple data augmentation operators multi-branch and layered pipeline. However, different from AugMix where augmentation operators and mixing weights are both randomly sampled, operators first

---

adversarial gradient
contrast
at
auto-
translale-x
translate-y
Mi
1 m
contrast
all
dreg
62 dWay
ground-rruts
rotate
wg
adversrrial target
un-modified image
Figure 2: AugMax overview. Black and red arrows represent forward paths and back- pack-propagation paths to generate AugMax images, respectively. In contrast to AugMax, where the mixing parameters m and are adversarially learned, AugMix randomly samples m and w from predefined distributions (and thus no backpropergation on m and w).
randomly sampled, followed by adversarially trained mixture of the selected operators in Augmax. This simple change from AugMix AugMax results in considerable difference in their feature distributions. From the visualizations (in Figure 1), AugMax generates more adversarially "hard samples", while still keeping good amount of diversity compared to AugMix and PGD Attack. Searching for adversarial mixing strategies in AugMax is slightly more expensive since it involves adversarial mixing. To make this efficient, adopt efficient adversarial training strategy [28]. As a result, AugMax adds only a reasonable amount of extra complexity compared to non-adversarial training methods, while improving robustness significantly. For example, AugMax training time is only 1.5 times that of AugMix on ImageNet (see Table 7). Moreover, Augmax is significantly more efficient than traditional adversarial training which has 10 times training time overhead compared with AugMix.
Being stronger form of data augmentation with adversarial sample generation, AugMax leads to a significantly augmented input distribution which makes model training more challenging. This naturally motivates to propose novel and finer-grained normalization scheme termed Dual-Batch- and-Instance Normalization (DuBIN). As illustrated in Figure 3, DuBIN adds an additional instance normalization (IN) in parallel the traditional Dual Batch Normalization (DuBN) [7, 29] used in adversarial training, in order to better model and disentangle the instance-wise feature heterogeneity arising from AugMax. We show that adding the instance normalization is an important knob to promote the capability of AugMax in boosting model robustness.
Our framework, AugMax- DuBINN is illustrated in Figure 2. -uBN trains clean images and achieves state-of-the-an robustness on natural corruption benchmarks [l], and also improves model robustness against other common distribution shifts [30, 2]. In particular, method surpasses state-of-the-an method on CIFAR10-C, CIFAR100-C, Tiny ImageNet-C and ImageNet-C by 3.03%, 3.49%, 1.82% and 0.71% respectively.
Related Work
2.1 Robustness to Distributional Shifts
Hendrycks al. [1] pioneer the study of prediction errors exhibited by machine learning models unseen natural corruptions and perturbations. They introduce two variants of the original ImageNet validation set to evaluate the model's 's robustness: the ImageNet-C for input corruption robustness, and the ImageNet-P for input perturbation robustness. The former applies 15 diverse corruptions drawn from four main categories (noise, blur, weather, and digital) on ImageNet validation images. The latter contains perturbation sequences generated from each image. Recht et al. [30, 2] collect CIFAR10.1 and ImageNet- V2 as reproduction of the original CIFAR10 [31] and ImageNet [32] test sets, and find that minute natural distribution shift caused by the minutiae in dataset collection process leads to a large drop accuracy for broad range of image classifiers. Geirhos el al. [33] observe that deep models trained on ImageNet are biased towards textures, and the model robustness
can be improved by emphasizing more on global shape features [33-36]. Wang al. [37] find that the benchmark test performance does not always translate to the real-world generalizability,
3

---

and propose an efficient method to troubleshoot trained models using real-world unlabeled images. Hendrycks et al. [38] collect ImageNet-A with the new concept of natural adversarial examples: unmodified real-world images that falsify common machine learning models. Other representative benchmark efforts include [39-42, 3]. all demonstrating the brittlness of deep models under various distribution shifts. Readers of interest are referred to recent survey [43].
2.2 Data Augmentation: Random and Adversarial
Data augmentations have been widely used to increase training set diversity and improve model generalization ability [44, 21, 33, 22]. Among those methods, AugMix [5] achieves the outstanding robustness against natural corruptions by randomly mixing multiple diverse augmentations. It establishes high performance bars on many natural corruption benchmarks including CIFARIO-C. CIFAR100-C and ImageNet-C [1]. DeepAugment [3] feeds clean images to image-to-image models with randomly perturbed model weights to generate visually diverse augmented images. Combined with AugMix, it achieves state-of-the-art robustness on ImageNet-CI [1]. Lately, concurrent work MaxUp [45] generates hard training samples by selecting the worst-case weights in the MixUp [20] pipeline, improve both model generalization ability and adversarial robustness. Our work shared similar mindset to Mp. but we build on the more sophisticated AugMix pipeline and handle the resulting higher feature heterogeneity using new normalization module.
Besides, Adversarial training (AT) [6, 46-52] utilizes adversarial samples as special data augmenta- tion method, and has also shown promise in improving model robustness against natural corruptions [23] and domain gaps [25], usually at the cost of increasing training time [27] and degrading the
standard accuracy on clean images [53]. Volpi et al. [25] and Zhao et al. [26] augment the training set with samples from a fictitious adversarial domain to improve domain adaptation performance.
Xie el al. [7] show that adversarial training can improve both standard accuracy and robustness against distribution shifts, with the help of auxiliary batch normalization. Adversarial Noise Training (ANT) [8] uses random Gaussian noise with learned adversarial hyperparameters (mean and variance) as the augmentation. Gowal et al. [54] propose to generate new augmented images by adversarially composing the representations of different original images. Wang al. [55] improve the robustness of pose estimation models by adversarially mixing different corrupted images as data augmentation.
Based on DeepAugment [3], Calian et al. [56] propose AdversarialAugmmett which optimizes the parameters of image-to-ing to-image models to generate adversarially augmented images, achieving promising results against natural corruptions. Robey al. [57] proposed model-based robust learning which generates augmented images with adversarial natural variations to improve model robustness against naturally-occurring conditions (e.g., snow, decolorization, shadow, and many others). The authors further generalized such idea to the domain generalization problem and achieved impressive results [58].
2.3 Normalization
Batch Normalization (BN) [59] has been a cornerstone for training deep networks. Inspired by BN, more task-specific modifications are proposed by exploiting different normalization axes, such as Instance Normalization (IN) [60], Layer Normalization (LNN [61] and Group Normalization (GN) [62]. A few latest works investigate to use multiple normalization layers instead of one BN. Zajkac al. [63] use two separate BNs to handle the domain shift between labeled and unlabeled data in semi-supervised learning. Xie et al. [64, 7] observe the difference between standard and adversarial feature statisties during AT, and craft dual batch normalization (DuBN) disentangle the standard and adversarial feature statistic to improve both the standard accuracy and robustness. While most deep models just use one normalization type throughout the network, IBN-Net [29] unifies IN and BN in one model. It finds that when applied together, IN tends to learn features invariant to appearance (colors, styles, etc.) changes, while BN is essential for preserving high-level semantic contents. IBN-Net shows that combining IN and BN in an appropriate manner can improve model generalization. A similar combination was also explored in the style transfer field [65], learning to selectively normalize only disturbing styles while preserving useful styles.

---

3 Method
3.1 Preliminary: The AugMix Framework
At high level, AugMix creates a new composite image from a clean sample by first applying several simple augmentation operations to it before combining the results through random linear combinations. This augmentation scheme is coupled with Jensen- -Shannn consistency loss that enforces feature similarity among clean and augmented images. Specifically, AugMix consists of multiple augmentation chains (3 by default), each applied in parallel to the same input image. Each chain composes one to three randomly selected augmentation operations. These augmentations are the same as Autmmut [21]. but excluding those that overlap with ImageNet-C corruptions. The augmented images from each augmentation chains are then combined together with the original, clean sample using linear combinations with randomly samples coefficients. The final image thus incorporates several sources of randomness, coming respectively from the choice of operations, the severity level of these operations, the lengths of the augmentation chains and the mixing weights. Despite its simplicity, AugMix achieves state-of-the-ar corruption robustness. In the remaining of this section, build on the previous AugMix work and propose AugMax, augmentation strategy that further improves robustness.
3.2 AugMax: Augmented Training with Unified Diversity and Hardness
In this section, we rigorously introduce Figure 2. At a high level, the main difference between AugMax and AugMix is new optimization procedure used to select the mixing weights and
data augmentation method, AugMax, illustrated in
Given a data distribution D over images Rd and labels y E R our goal train mapping (classifier) f Rd + Ie from images to output softmax probabilities, parameterized by 6, that is robust (unknown) distribution shifts. We minimize the empirical risk
min LE (ayii [(f(x; 0), y),
where -(.,)) -) is the loss function (e.g., cross-entropy). As illustrated in Figure 2, an AugMax image xx is generated by learning set of adversarial mixing parameters m", wn m",w" which maximizes the loss:
x² g(ooiggi m",""""""""" ww":
where g(-) denotes the AugMax augmentation function, Toriy is the original image, and
m , _0 arg max L( (origi m. w);0),y) s.t. m [0, w E [0, 1J" 1-
1
To simplify the optimization problem Eq. (3), we use a re- re-parameterization trick by setting w (p), where (.) is the softmax function, and convert the optimization into a differentiable one:
m p" = arg max
C(f(g (p)): y); w o
Training with Aug augmentation can then be written altogether as minimax optimization:
minE minE (a.)) _ [C(((()) (=C((a)):0)))))) 0).),) [([([))) 0), (x,
(5)
x= g(=; m", w²); w" =ppp)) m", m".p" arg max (())) 0), me|0.11.pERR
where C. is a consistency loss regularizing augmented images to have similar model outputs with the original images and À is the trade-off parameter. Our implementation of L. is adapted from [5]:
(x( JS(f(w; 0), (w;; 0), f(=";
(6)
with JS(-) being the Jensen-Shanman divergence and z being the augmented image generated by AugMix from æ. In order to reduce the training complexity overhead caused by adversarial training,
we employ an accelerated adversarial attack method [28] to solve Eq. (4), which adds only
reasonable amount of extra complexity compared to AugMix baseline. The algorithm solve Eq. (5)
is summarized in Appendix A.
5

---

Visualization of the effects of diversity and hardness To illustrate the motivation behind Aug- Max, we visualize the feature representations induced by different augmentation methods showcase their diversity and hardness in Figure 1. Specifically, we visualize the impact of various data augmen- tation methods on the feature representations obtained with ResNeXt29 that normally trained on CIFAR-10 10. Note that the model is trained using only default standard augmentations (i.e., random flipping and translation), and therefore has never seen augmentation from AugMix, AugMax, or PGD attack. Following [4], we visualize the penultimate layer feature distributions of training samples from three different classes. Three hundred training images are selected from each class for visualization. For AugMix and &o PGD attack, we use the same hyperparameters in the original papers [5, 6].
We observe in Figure that the clean images form separated clusters in the feature space, leaving large regions between those clusters empty. This potentially causes uncertain and unreliable generalization if new test samples (from a shifted distribution) fall into those regions. AugMix enlarges each cluster to cover a broader region, effectively increasing sample diversity. However, after augmentation, very few samples are close to the decision boundaries, revealing an inability to generate hard samples. Samples from the PGD attack, the other hand, collapse into small region around the classification borders, losing feature diversity. We further investigate the method through ablation studies in Section 4.3.
3.3 DuBIN: Disentangled Normalization for Heterogeneous Features
AugMax leads to better coverage of the input space by unifying sample diversity and hardness, as demonstrated in Figure 1. While such more versatile distribution in data augmentation has the potential to increase robust- ness, we do observe that naive incorporation of AugMax leads to relatively marginal improvement over Mix (see Section 4.3). This is due to the comprehensivenes. of AugMax can also lead to a higher feature heterogeneity, which may require larger model capacity to encode. To ad- dress this problem, we design a novel normalization layer, termed Dual Batch-andd-nsstand Instance Normalization (DuBIN), to disentangle the instance-level heterogeneity.
CCAV
DUBIN
DuBN
IN
BN
Audilary BN
chanrel
Spft
C/2 channels
charnals
Figure 3: Illustration of DuBIN. "Split" and "cat" represent splitting and concate- nating on features along the channel di- mension. The input feature with C channels is split into halves (i.e., A and nn each with C/2 channels). Each half goes into different normalization layers (either IN DuBN) and the outputs concatenated back along the channel di- mension.
As illustrated in Figure 3. DuBIN consists of two parallel parts: (1) dual batch normalization (DuBN) [7], which is the by-default normalization layer to use for adversarial training [7, 64], to disentangle the group-level statistics of the clean and comer-case augmented samples?; and (2) additional instance normalization, to account for instance- level feature statistics due to augmentation diversity. At each layer, the input feature is split into halves along the channel dimension: one half is then fed into IN, and the other fed into DuBN. Finally, the outputs of IN and DuBN are concatenated back along the channel dimension for the next layer.
To provide additional insights of DuBIN, we compare the BN statistics of AugMax-DuBN and AugMAx-DUBIN in Table 1. o² and o²
Table 1: BN statistics of different layers in WRN40-2 with DuBN or DuBIN trained on CI- FAR using AugMax training.
represent the average batch normalization vari-
ance over all channels of BN, and BN,2, re- spectively. We observe that both σ² and o² are smaller the DuBIN network than those in the DuBN counterpart. This shows that the IN branch in DuBIN can reduce the feature di- versity that BN otherwise needs to model, by encoding instance-level diversities. With lower feature variation, BNs can converge better with improved performance [66].
Layer
Block
Block 1
Block
layer
layer
layer 2
0.0369
0.0450
0.0301
AugMax-DuBl
0.0469
0.0585
0.0382
0.0306
0.0403
0.0264
AugMa
0.0348
0.0466
0.0292
'DuBN consists of two separate BNs: one BN, for clean images and the other BN, for adversarially augmented (e.g., AugMax) images.

---

4 Experiments
Here, we introduce in detail the experimental setting, and report quantitative results for robustness against natural corruptions and distribution shifts. We show that our method induces better robustness than existing ones. We study the properties of our approach in thorough ablation studies.
4.1 Experimental Setup
Datasets and Models We evaluate our proposed method on CIFARIO, CIFARI00 [31], ImageNet [32] and Tiny ImageNet (TIN)'. For neural architectures, we use ResNet18 [67], WRN40-2 [68] and ResNeX(22 [69] on the CIFAR datasets, and ResNet18 on ImageNet and Tiny ImageNer'. To evaluate the model's robustness against common natural corruptions, we use CIFAR 1O-C, CIFARI00- C, ImageNet-C and Tiny ImageNet-C (TIN-C) [1], which are generated by adding natural corruptions
to the original test set images.
Evaluation Metrics Following established conventions [1, 5], we define robustness accu- racy (RA) as the average classification accuracy over all 15 corruptions, to measure the models' robustness on CIFARIO-C and CIFAR 100-C. On ImageNet-C and Tiny ImageNet-C. we use both RA and mean corruption error (mCE) to evaluate robustness. As defined by [1]. mCE is the weighted average of target model corruption errors normalized by the corruption errors of baseline model across different types of corruptions.
Table 2: Evaluation results on CIFAR1O and
CIFARTO-C. The best metric shown in bold.
Model
Metric
Normal
AugMix
AugMax-DuBIN
ResNet18
SA (%)
95.36
95.79
95.76 (-0.05)
RA (%)
74.75
89.49
90.36 (+0.87)
SA (%)
94.78
95.67
95.68 (-0.01)
WRN40-2
RA (%)
73.71
89.01
90.67 (+1.66)
ResNeX129
(%)
95.60
(-0.14)
RA (%)
71.70
89.08
92.11 (+3.03)
We use the AlexNet provide in [1] as the baseline model for experiments on ImageNet. Since [1] does not provide an off-the-shelff baseline model for Tiny ImageNet, we use the conventionally trained ResNet18 on Tiny ImageNet as the baseline model for experiments on Tiny ImageNet. We use standard accuracy (SA) to denote the classification accuracy on the original clean testing images.
Baseline Methods On CIFARIO-C and CIFAR100-C, compare our method with the state-of-the-art method, Aug [5]. In addition, AugMix has previously been combined with other orthogonal methods, such as DeepAugment [3], further boost performance on ImageNet-C. Specifically, [3] combine Duemengment as an orthogonal com- ponent to AugMix, and achieve state-of oftte-of-the-arr art performance on ImageNet-C with DeepAugment AugMix [3]. To allow comparison with this scenario, we also run the following comparison experiment on ImageNet-C and Tiny ImageNet-C: C: (1) AugMix v.s. AugMax and (2) DeepAugment + AugMix v.s. DeepAugment + AugMax. We show that the advantages of Aug still exist when combined with other orthogonal methods.
Table 3: Evaluation results on CIFARI00 and
CIFAR100-C. The best metric is shown in bold.
Model
Metric
Normal
AugMix
AugMax-Dubii
ResNet18
SA (%)
77.99
78.69 (-0.46)
RA (%)
48.46
62.67
65.75 (+3.08)
SA (%)
76.19
77.03
76.80 (:D.23)
WRN40-2
RA (%)
46.80
64.56
66.35 (+1.79)
SA (%)
79.95
78.58
80.70 (+2.12)
ResNeX129
(%)
47.76
68.86 (+3.49)
Finally, also compare AugMax with ANT [8], which utilizes Gaussian noises with learned distribution parameters for data augmen- tation. Since models tend to generalize bet- ter on test images similar to those seen during training [70], ANT mainly improves robustness against high frequency corruptions (e.g.. addi- tive noises), while achieving comparable robust- ness on other unseen corruptions (e.g., bluring) with AugMix. Following [8], we perform the comparison on the subset of Tiny ImageNet-C with 12 remaining corruptions after removing all additive noises.
Table 4: Evaluation results on ImageNet and ImageNet-C. The best metric is shown in bold.
Method
SA (%, 1)
RA (%,
mCE (%,
Normal
69.83
30.91
87.47
AugMix
68.06
34.58
83.08
67.62
35,01
82,56
AugMax-DuBIN
(-0.44)
(+0.43)
(-0.52)
DeepAugment
AugMix
65.32
45.84
69.29
DeepAugment
64.43
46.55
68.47
AugMax-DUBIN
(-0.89)
(+0.71)
(-0.82)
hhtps://www.kagg: wttp://wwwwgc...oom http://wwww..aglll..o////nnnnnns com/c/tiny-imagene
"Ou WRN40-2 and ResNeX129 models are identical to those used in AugMix [5].
7

---

Implementation Details We follow [5] for hyperparameter settings. Specifically, for all experi- ments on CIAFRIO and CIFARI00, we use SGD optimizer with initial learning rate 0.1 and cosine annealing learning rate scheduler, and train all models for 200 epochs. For all experiments on ImageNet, we use SGD optimizer with initial learning rate 0.1 and batch size 256 to train the model for 90 epochs. We reduce the learning rate by 1/10 at the 30-th and 60-th epoch. For all experiments on Tiny ImageNet, we train for 200 epochs and decay the learning rate at the 100-th and 150-th epoch. Other hyperparameter settings are identical as those in ImageNet experiments. We set batch size to 256 for all experiments. In Eq. (4), and p are uniformly randomly initialized between 0 and 1. We use PGD to update m: first do gradient ascend on m and then project it back into the [0, 1] interval. p is updated using gradient ascend. AugMax follows
Table 5: Evaluation results on TIN and TIN-C. The best metric is shown in bold.
Method
SA (%, 1)
RA (%, 1)
mCE (%, J)
Normal
61.64
23.91
100.00
AugMix
61.79
36.85
83,04
the same rules as Mug [5] to randomly select augmentation operation type, severity level and chain length. Following [6]. we iteratively solve the inner maximization and outer minimization problems in Eq. (5): the inner maximization is updated for n steps, for every step update on the outer minimization. Both m and are updated with step size a = 0.1. We set n = 5 on ImageNet for efficiency and n = 10 on other datasets. We set À in Eq. (5) to be 12 on ImageNet and 10 on all other datasets, except for ResNet18 8 on CIFAR 100 where we find À = leads to better performance. All experiments are conducted on a server with four NVIDIA RTX A6000 GPUs.
62.21
38.67
80.72
AugMax-DuBIN
(40,42)
(+1.82)
(-2.32)
DeepAugment
AugMix
59.59
40.67
78,28
DeepAugment
59.72
40.99
77.83
AugMux-DUBIN
(40.13)
(+0.32)
(-0.45)
4.2 Robustness against Natural Corruptions
Results on CIFARIO-C and CIFAR100-C are shown in Table and respectively, where "normal" means training with default standard augmentations (e-g., random flipping and translation).. From the results, we see that Aug achieves new state-of-the-ar performance on both datasets with different model structures. For example, compared with AugMix, our method improves accuracy by as high as 3.03% and 3.49% on CIFAR I0-C and CIFAR 100-C, respectively. Moreover, we observe that our method benefits larger models more. Specifically, in both CIFARIO and CIFAR 100 experiments, the largest robustness gain is achieved on ResNeXt29, which has the largest capacity among the three models.
Results on ImageNet are shown in Ta- ble 4. AugMax-DuBIN outperforms AugMix by 0.52% in terms of mCE on ImageNet-C. Fur- ther combining AugMax-DuBll with DeepAug- ment outperforms AugMix + DeepAugment by 0.82% in terms of mCE, achieving a new state- of-the-art performance on ImageNet-C. This
Table 6: Evaluation results on TIN and TIN-C (wlo
noise). The best metric is shown in bold.
Method SA (%, 1)
RA (%. -)
mCE (%, J)
Normal 61.64
24.38
100.00
AugMix
61.79
37.63
82.54
ANT
61.26
35.30
85.70
AngMax-DuBIN
62.21
38.66
80.29
shows that AugMax can be used as a more advanced basic building block for model robustness and inspires future researches to build other defense methods on top of it.
Results on Tiny ImageNet-C are shown in Table 5 and Table 6. From Table 5, we see that our method improves the mCE by 2.32% compared with AugMix and 0.45% when combined with Deepnugment. From Table 6, we can see our method outperforms ANT by considerable margin.
Table 7: Training time on ImageNet with ResNet18. 8, reported on a single NVIDIA A6000.
Training Time Since we use an accelerated adversarial train- ing method [28] solve Eq. (4), the worst-case mixing strat- egy can be searched efficiently. As a result, AugMax adds only a modest training overhead over AugMix, and is signif- icantly more efficient than traditional adversarial training (AT). Detailed training time is shown in Table 7.
Method
Time (sec/epoch)
Normal
2669
Aug
3622
AT [6]
37162
AugMax
5264
4.3 Ablation Study
Analysis of Different Normalization Layers In this paragraph, we compare AugMax/AugMix when combined with different normalization schemes. When using DuBN or DuBIN, we route
8

---

original images to the BN and AugMax/AugMix images to B. Since it is the by-default setting to use disentangled normalization layers (c.g.. DuBN) for original and augmentation images in adversarial training [7, 64], we follow this traditional routing and combine AugMax with DuBN or with either AugMix-
DuBIN instead of BN or IBN. fair to compare AugMax-DuBNiNn
BN/IBN or JugMix-DuBN/Dull
Results are shown in Table AugMax-DuB1 -DBBN outperforms both AugMix-BN BN and AugMix- DuBN, and AugMax-Duul DuBNN outperforms both AugMix-IBN and AugMix-DuBIN. Noticeably, when combined with AugMix. DuBIN also helps improve robustness. This shows the poten- tial of applying DuBIN to other diversity aug- mentation methods for general performance im- provement.
Table 8: Ablation results on DuBIN. RA (%) on CIFARI0/100-C with WRN40-2 backbone are re- ported. Mean and standard derivation over three random seeds are shown for each experiment.
Method
CIFARIO-C
CIFAR100-C
AugMix-BN
89.01 (#: 0.03)
64.56 (#: 0.04)
AugMix-IBN
89.17 (#: 0.24)
63,94 (#: 0.28)
AugMix-DuBN
89.11 0.21)
64.07 (#: 0.38)
AugMix DuBBN
89.74 (2: 0.35)
65.02 (#: 0.58)
We also conduct stability analyses on these
AugMax-DuBN
89.60 (#: 0.62)
65.06 (+: 0.28) 66.35 (1 0.21)
experiments, by showing the statistical sig- nificance of the improvements achieved by AugMax-DuBl MaMMaX--DBBIN over the algorithm randomness. 8 using three different random seeds, and report the mean (denoted as (4) and standard derivation (denoted as o)) of accuracy in the form of μ (= σ)) in Table 8. As we can observe, the improvements achieved by AugMax-DuBll DuBBNN are statistically significant and consistent.
AugMax-DuBIN
90.67 (= 0.16)
Specifically, we run all experiments in Table
AugMax Hyperparameters In this paragraph, we check the sensitivity of AugMax with respect to its hyperparam- eters: maximization steps step size a, and consistency loss tradeoff parameter X. The results are shown in Table 9. We first fix A = 10 and try different values for n and k. We fix k- = 1 when adjust- ing n and fix n = 10 when adjusting k. As we can see, AugMax is stable with respect to different values of and k. We use n 10 and A: = 1 as the default values since they empirically yield good results at low training overhead. We then fix n = 10, k = 1 and adjust A. We find robustness peaks at around x 10, which we set as the default value.
Table 9:
Ablation results on AugMax-DuBIN update step number n and, carly-stopping step l and consis- tency loss tradeoff parameter A. Results are reported on CIFARIO/CIFARIO 10-C with WRN40-2.
k
À
15
SA (%) (5)
95.78
95.68
95
95 60
95.51
95.90
95.68
95.88
90.49
90.67
90.67
90.38
90.21
88.89
90.67
90.30
Comparison with Different Diversity and Hardness Strategies In this paragraph, we study different strategies to combine diversity and hardness. The results confirm that diversity and hardness are two complementary dimensions and their proper unification boosts model robustness.
Specifically, we compare AugMax with baseline methods from each strategy group (i.e., diver- sity or hardness), and other possible strategies to combine both. Besides AugMix and Adversar-
Table 10: Results of different augmentation strate- gies on CIFAR C1FAR100(-C) with ResNet18.
Method
Strategy
(%)
RA (%)
ial Training (e.g., PDGAT [6]. FAT [28]), which are the two representative examples for diver- sity and hardness as shown in Figure 1, we also design the following baseline methods. (1) Aug-
Normal
77.99
AugMix [5]
(diversity)
78.23
62.67
PGDAT [6]
60.94
47.71
FAT [28]
61.51
48.70
AdvMax
(hardness))
56.61
38.65
MiX+PGDAT: using both AugMix and adver- sarial images for augmentation. This is a naive baseline to combine diversity with hardness. (2)
AugMix+PGDAT AdvMix
61.68
51.39
(diversity & anvveritt)
72.36
56.89
AugMax-DuRM DuBIN
78.52
64.02
AdvMix: applying adversarial attacks on the augmentation hyperparameters such as the rotation
angles and translation pixel numbers [71, 72]. while randomly selecting the mixing parameters; (3)
AdyMax: applying adversarial attacks on both the augmentation hyperparameters and the mixing weights. Please see Appendix B for their implementation details.
'Note that AdyMax implies full hardness and little diversity, while AdvMix switches the order of randomness and adversarial augmentation in the AugMax pipeline.

---

The results are shown in Table 10. AugMax-DuBIN achieves the best performance, outperforming all methods from either diversity hardness group. This shows diversity and hardness to indeed complementary. On the other hand, the other two naive baselines jointly considering diversity and hardness achieve poorer performance, showing nontrivial design a method achieving good balance between diversity and hardness.
4.4 Robustness against Other Distribution Shifts
Although AugMax mainly aims at improving model robustness against common corruptions, find that it can also gain robustness against other types of distribu- tion shifts. Specifically, we evaluate on the distribution shifts caused by the differences in data collection pro- cess using CIFAR 10. [30], and against Spatial Trans- form ad adversarial Attacks (STA) [71] on CIFARIO-ST We generate CIFAR 10-STA by applying the worst-of-k attack with k = 10 on CIFARIO test set following [71]. Results are reported Table 11 (using the same ResNeXt22 models trained on CIFAR1O as in
Table 11: Robustness against other dis-
tribution shifts. Accuracy (%) on CI- FARI0.1 and CIFARIO-STA are evaluated on ResNeX129 trained on CIFARIO.
Method
CIFARI0.1
CIFARIO-STA
Normal
88,90
30,30
Aug
88.90
54.30
AugMax-DuBIn
90.64
63.20
Section 4.2). AugMax-DuBIN steadily outperforms AugMix against both distributional shifts.
5 Conclusion
In this paper, we propose AugMax, a novel data augmentation strategy, that significantly improves robustness by strategically unifying diversity and hardness. To enable efficient training while facing the resulting heterogeneous features, we design a novel normalization scheme termed DuBIN. Using the combination of AugMax and DuBIN, we consistently demonstrate state-of-the- the-the-art art robustness on several natural corruption benchmarks.
Aag unifies diversity and hardness in a heuristic manner. While we showed in ablation studies that Aug outperforms other heuristic combinations, there may be better trade-offs available. We leave such study for future work and focus in this paper on showing the benefit of unifying diversity and hardness, which were separately considered in previous research.
Acknowledgement
Z.W. is supported by the U.S. Army Research Laboratory Cooperative Research Agreement W911NE 17-2-0196 (IOBT REIGN), and an NVIDIA Applied Research Accelerator Program.
References
[1] Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In International Conference Learning Representations (ICLR), 2019. 3, 4,
[2] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do ImageNet classifiers generalize to ImageNet? arXiv preprint arXiv:1902. 10811, 2019.
[3] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of -it-of-distibuttion generalization. In IEEE International Conference Computer Vision (ICCV), pages 8340-8349, 2021. 2, 4,7
[4] Rafael Müller, Simon Korbblith, and Geoffrey Hinton. When does label smoothing help? arXiv preprint
arXiv: /0w./9906022, 2019. 2, 6
[5] Dan Hendrycks, Norman Mu, Ekin D Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshminarayanan. AugMix: A simple data processing method improve robustness and uncertainty. In International Conference on Learning Representations (ICLR), 2020. 1, 2, 4. 5, 6, 7, 8, 15
[6] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. To- wards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations (ICLR), 2018. 1, 4, 8.9
10

---

[7] Cihang Xie, Mingxing Tan, Boqing Gong, Jiang Wang, Alan Yuille, and Quoe V Le. Adversarial examples improve image recognition. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), pages 819-828, 2020. 2, 2,3,4,6, 6,
[8] Evgenia Rusak, Lukas Schott, Roland $ Zimmermann, Julian Bitterwolf, Oliver Bringmann, Matthias Bethge, and Wieland Brendel. simple way to make neural networks robust against diverse image corruptions. In European Conference on Computer Vision (ECCV), pages 53-69, 2020. 1, 7,7
[9] Matthias Hein and Maksym Andriushchenko. Formal guarantees on the robustness of a classifier against adversarial manipulation. In Advances in Neural Information Processing Systems (NewrIPS), pages 2266 2276, 2017.
[10] Tsui-Wei Weng. Huan Zhang, Hongge Chen, Zhao Song, Cho-Jui Hsich, Duane S. Boning, Inderjit S. Dhillon, and Luca Daniel. Towards fast computation of certified robustness for ReLU networks. In International Conference on Machine Learning (ICLR), pages 5273-5282, 2018.
[11] Tsui-Wei Weng, Huan Zhang, Pin- Yu Chen, Jinfeng Yi, Dong Su, Yupeng Gao, Cho-Jui Hsich, and Luca Daniel. Evaluating the robustness neural networks: An extreme value theory approach. International Conference on Learning Representations (ICLR), 2018. 1
[12] Stephan Zheng. Yang Song, Thomas Leung, and Ian Goodfellow. Improving the robustness of deep neural networks via stability training. In IEEE Conference on Computer Vision and Puttern Recognition (CVPR). pages 1480-4488. 2016. 1
[13] Dan Hendrycks, Kimin Lee, and Mantas Mazeika. Using pre-training can improve model robustness and uncertainty. In International Conference on Machine Learning (ICMI.), pages 2712-2721, 2019.
[14] Tianlong Chen, Sijia Liu, Shiyu Chang. Yu Cheng, Lisa Amini, and Zhangyang Wang. Adversarial robustness: From self- supervised pre-training to fine-tuning. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 699-708, 708, 2020.
[15] Ziyu Jiang, Tianlong Chen, Ting Chen, and Zhangyang Wang. Robust pre-traiin by adversarial contrastive learning. In Advances in Neural Information Processing Systems (NeurIPS), pages 16199-16210, 2020.
[16] Jiachen Sun, Yulong Cao, Christopher Choy. Zhiding Yu, Chaowei Xiao, Anima Anandkumar, and Z Morley Mao. Improving adversarial robustness in 3D point cloud classification via self-supervisions. In International Conference Machine Learning Workshop (ICMLW), 2021.
[17] Richard Zhang, Making convolutional networks shift-invariant again. In International Conference on Machine Learning (ICML), pages 7324-7334, 2019.
[18] Cristina Vasconcelos, Hugo Larochelle, Vincent Dumoulin, Nicolas Le Roux, and Ross Goroshin. An effective anti-aliasing approach for residual networks. arXiv preprint arXiv:201 1.60655, 2020.
[19] Jungkyu Lee, Taeryun Won, and Kiho Hong. Compounding the performance improvements of assembled techniques in convolutional neural network. arXiv preprint arXiv:2001.06266 2020.
[20] Hongyi Zhang, Moustapha Cisse, Yann Dauphin, and David Lopez- Paz. MixUp: Beyond empirical risk minimization. International Conference on learning Representations (ICIR), 2018. 2,
[21] Ekin D Cubuk, Barret Zoph, Dandelion Mane. Vijay Vasudevan, and Quoc Le. AutoAugment: Learning augmentation policies from data. arXiv preprint LrXiv::805 095009 2018. 5,
[22] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. CutMix: Regularization strategy to train strong classifiers with localizable features. In IEEE International Conference on Computer Vision (ICCV), pages 6023-6033 2019. 2,
[23] Justin Gilmer, Nicolas Ford, Nicholas Carlini, and Ekin Cubuk. Adversarial examples are natural consequence of test error in noise. In International Conference Machine Learning (ICML), pages 2280 2289, 2019. 2,
[24] Tianlong Chen, Yu Cheng, Zhe Gan, Jianfeng Wang. Lijuan Wang, Zhangyang Wang, and Jingjing Liu. Ad- versarial feature augmentation and normalization for visual recognition. arXiv preprint arXiv: 12171, 2021. 2
[25] Riccardo Volpi, Hongseok Namkoong, Ozan Sener, John C. Duchi, Vittorio Murino, and Silvio Savarese. Generalizing to unseen domains via adversarial data augmentation. In Advances in Neural Information Processing Systems (NeurIPS), pages 2018. 2,4

---

[26] Long Zhao, Ting Liu, Xi Peng, and Dimitris Metaxas. Maximum-entrope adversarial data augmentation for improved generalization and robustness. In Advances in Neural Information P Processing Systems (NeurIPS), 2020. 2, 4
[27] Dinghuai Zhang, Tianyuan Zhang, Yiping Lu, Zhanxing Zhu, and Bin Dong. You only propagate once: Accelcrating adversarial training via maximal principle. In Advances in Neural Information Processing Systems (NewrIPS), pages 227-238, 2019. 2, 4
[28] Jingfeng Zhang. Xilie Xu, Bo Han, Gang Niu, Lizhen Cui, Masashi Sugiyama, and Mohan Kankanhalli. Attacks which do not kill training make adversarial learning stronger. In International Conference on Machine Learning (ICML), pages 11278 11287, 2020. 3, 5, 8, 9, 15
[29] Xingang Pan, Ping Luo, Jianping Shi, and Xiaoou Tang. Two at once: Enhancing learning and generaliza- tion capacities via ibn-net. In European Conference on Computer Vision (ECCV), pages 464-479, 2018. 3.
[30] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do CIFAR-10 classifiers generalize to CIFAR-10? arXiv preprint arXiv:1800 0005,0051, 2018. 3, 10
[31] Alex Krizhevsky. Learning multiple layers of features from tiny images. Master's thesis, University of Toronto, 2009. 7,7
[32] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In IEEF Conference on Computer Vision and Pattern Recognition (CVPR), pages 248-255, 2009. 7,7
[33] Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Imagenet-trained CNNs biased towards texture, increasing shape bias improves accuracy and robustness. In International Conference on Learning Representations (ICLR), 2019. 3,4
[34] Katherine Hermann, Ting Chen, and Simon Kornlith. The origins and prevalence of texture bias in convolutional neural networks. In Advances in Neural Information Processing Systems (NeurIPS), pages 19000-19015, 2020.
[35] Yingwei Li, Qihang Yu, Mingxing Tan, Jieru Mei, Peng Tang, Wei Shen, Alan Yuille, and Cihang Xie. Shape-texture debiased neural network training. In International Conference on Learning Representations (ICLR), 2021.
[36] Mingjie Sun, Zichao Li, Chaowei Xiao, Haonan Qiu, Bhavya Kailkhura, Mingyan Liu, and Bo Li. Can shape structure features improve model robustness under diverse adversarial settings? In IEEE International Conference on Computer Vision (ICCV), pages 7526-7535, 2021.
[37] Haotao Wang. Tianlong Chen, Zhangyang Wang, and Kede Ma. am going MAD: Maximum discrepancy competition for comparing classifiers adaptively. In International Conference on Machine Learning (ICLR), 2020. 3
[38] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. arXiv preprint arXiv:1907.07174, 2019. 4
[39] Haohan Wang. Songwei Ge, Zachary Lipton, and Fric P Xing. Learning robust global representations by penalizing local predictive power. In Advances in Neural Information Processing Systems (NeurIPS), pages 10506-10518, 2019. 4
[40] Keren Gu, Brandon Yang, Jiquan Ngiam, Quoe Le, and Jonathon Shlens. Using videos to evaluate image model robustness. arXiv preprint arXiv: 10076, 2019.
[41] Vaishaal Shankar, Achal Dave, Rebecca Roelofs, Deva Ramanan, Benjamin Recht, and Ludwig Schmidt. Do image classifiers generalize across time? arXiv preprint arXiv:1906.021c. 2019.
[42] Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, and Ludwig Schmidt. Mea- suring robustness to natural distribution shifts in image classification. In Advances in Neural Information Processing Systems (NeurIPS). pages 18583-1888 18599, 2020. 4
[43] Sina Mohseni, Haotao Wang, Zhiding Yu, Chaowei Xiao, Zhangyang Wang, and Jay Yadawa. Practical machine learning safety: A survey and primer. arXiv preprint arXiv:2106.0448 2021. 4
[44] Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data augmentation. arXiv preprint arXiv:1708.00 04896, 24176 4
12

---

[45] Chengyue Gong, Tongzheng Ren, Mao Ye, and Qiang Liu. MaxUp: A simple way to improve generalization of neural network training. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pages 2474-2483, 2021. 4
[46] Shupeng Gui, Haotao Wang, Haichuan Yang, Chen Yu, Zhangyang Wang, and Ji Liu. Model compression with adversarial rohustness: A unified optimization framework. In Advances in Neural Information Processing Systems (NeurlPS), pages 1283-1294, 2019. 4
[47] Ting-Kuci Kuci Hu, Tianlong Chen, Haotao Wang, and Zhangyang Wang. Triple wins: Boosting accuracy, robustness and efficiency together by enabling input- adaatid inference. In International Conference on Learning Representations (ICLR), 2020.
[48] Haotao Wang. Tianlong Chen, Shupeng Gui, Ting- Kuei Hu, Ji Liu, and Zhangyang Wang. Once-for-all adversarial training: In-situ tradeoff between robustness and accuracy for free. In Advances in Neural Information Procexsing Systems (NeurIPS), pages 7449-7461, 2020.
[49] Tianlong Chen, Zhenyu Zhang, Sijia Liu, Shiyu Chang, and Zhangyang Wang. Robust overfitting may be mitigated by properly learned smoothening. In International Conference on Learning Representations (ICLR), 2020.
[50] Dongxian Wu, Shu-Tao Xia, and Yisen Wang. Adversarial weight perturbation helps robust generalization. In Advances in Neurul Information Processing Systems (NeurlPS), pages 2958 2955-9969,,60 2020.
[51] Ruize Gao, Feng Liu, Jingfeng Zhang, Bo Han, Tongliang Liu, Gang Niu, and Masashi Sugiyama. Maximum mean discrepancy test is aware of adversarial attacks. In International Conference on Machine Learning (ICMI.). pages 3564-3575, 2021.
[52] Jingfeng Zhang, Jianing Zhu, Gang Niu, Bo Han, Masashi Sugiyama, and Mohan Kankanhalli. Geometry- aware eestancerreweigheed adversarial training. International Conference on Learning Representations (ICLR), 2020. 4
[53] Dimitris Tsipras, Shihani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Robust- ness may be at odds with accuracy. In International Conference on Learning Reprexentations (ICLR), 2019. 4
[54] Sven Gowal, Chongli Qin, Po-Sen Huang, Taylan Cemgil, Krishnamurthy Dvijotham, Timothy Mann, and Pushmeet Kohli. Achieving robustness in the wild via adversarial mixing with disentangled representations. In IEFE Conference on Computer Vision and Pattern Recognition (CVPR), pages 121 1-1220, 2020.
[55] Jiahang Wang, Sheng Jin. Wentao Liu, Weizhong Liu, Chen Qian, and Ping Luo. When human pose estimation meets robustness: Adversarial algorithms and benchmarks. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages (1855-11864 2021. 4
[56] Dan Calian, Florian Stimberg, Olivia Wiles, Sylvestre-Alvis Rebuffi, Andras Gyorgy, Timothy Mann, and Sven Gowal. Defending against image corruptions through adversarial augmentations. arXiv preprint arXiv:2104.01086. 2021.
[57] Alexander Robey, Hamed Hassani, and George J Pappas. Model- based robust deep learning: Generalizing to natural, out-of-distribution data, arXiv preprint 2005 10247, 2020. 4
[58] Alexander Robey, George Pappas, and Hamed Hassani. Model-based domain generalization. In Advances in Neural Information Processing Systems (NeurIPS), 2021. 4
[59] Sergey loffe and Christian Szegedy. Batch Normalization: Accelerating deep network training by reducing internal covariate shift, In International Conference on Machine Learning (ICML), pages 448 456, 2015,
[60] Dmitry Ulyanov, Andrea Vedaldi, and Vietor Lempitsky. Instance normalization: The missing ingredient for fast stylization. arXiv preprint arXiiv:607.000022 2016. 4
[61] Jimmy Lei Ba. Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint
(14.:16007004555 2016. 4
[62] Yuxin Wu and Kaiming He. Group normalization. In European Conference on Computer Vision (ECCV), pages 3-19, 2018. 4
[63] Michal Zajac, Konrad Zoina, and Stanislaw Jastrzębski. Split batch normalization: Improving semi- supervised learning under domain shift. arXiv preprint 0335551904.0355,, 2019. 4
13

---

[64] Cihang Xie and Alan Yuille. Intriguing properties of adversarial training, In International Conference Learning Reprexentations (ICLR), 2020. 4, 6.
[65] Hyeonseob Nam and Hyo-Eun Kim. Batch-instance normalization for adaptively style-invariant neural networks. In Advances in Neural Information Processing Systems (NeurIPS), pages 2538-2567, 2018. 4
[66] Xiangru Lian and Ji Liu. Revisit batch normalization: New understanding and refinement via composition optimization. In International Conference Artificial Intelligence and Statistics (AISTATS), pages 3254 3263, 2019.
[67] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016.
[68] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In British Machine Vision Conference
(BMVC), pages 87.1 87.12, 2016. 7
[69] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transforma- tions for deep neural networks. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1492-1500, 2017.
[70] Aharon Azulay and Yair Weiss. Why deep convolutional networks generalize so poorly small image transformations? arXiv preprint arXiv:1805.12177 121777777.177.... 2018.
[71] Logan Engstrom, Brandon Tran, Dimitris Tsipras, Ludwig Schmidt, and Aleksander Madry. Exploring the landscape of spatial robustness. In International Conference on Machine Learning (ICML), pages
2019. 9, 10, 15
[72] Chaowei Xiao, Jun- Yan Zhu, Bo Li, Warren He, Mingyan Liu, and Dawn Song. Spatially transformed adversarial examples. In International Conference Leurning Representations (ICLR), 2018. 9, 15
14


Reviews:
[{"review_id": "rEtXeOLg0P", "reviewer": "Reviewer_fdjn", "summary": "This paper proposes a novel data augmentation method AugMax which adversarially learns the parameters for integrating randomly selected augmentation operators to achieve a unification of diversity and hardness. To solve the issue of feature heterogeneity incurred by AugMax, this paper proposes DuBIN which simultaneously uses DuBN and IN instead of only DuBN. AugMax-DuBIN can improve OOD robustness while almost maintaining generalization.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n1.\tThis paper is well organized. The motivation of AugMax-DuBIN is clear. AugMax utilizes the strengths of both AugMix and PGD attack to gain a unification of diversity and hardness. DuBIN combines IN with DuBN to reduce the feature diversity which is validated by smaller batch normalization variance brought by BuBIN.\n2.\tObserving from the experimental results, AugMax-DuBIN can significantly improve robust accuracy and decrease mean corruption error.\n\nCons:\n1.\tPlease illustrate the differences between robustness accuracy (RA) and mCE Since RA is also evaluated as the average accuracy on corruptions. In addition, please provide more details about the corruption methods.\n2.\tIt is better to provide the ablation study on the effect of early-stopping step $k$ in Algorithm 1. I suppose it is a critical hyper-parameter for the performance of AugMax-DuBIN. In addition, please illustrate the differences between AugMax with $k=10$ and PGD with 10 iteration steps. Also, it is better to conduct a fair comparison between FAT[1] which utilizes early-stepped PGD to improve robustness.\n3.\tThis paper lacks some theoretical analyses to further validate the efficacy of AugMax.\n4.\tIt is a bit wired to claim achieve a unification of diversity and hardness. Figure 1 illustrates AugMax gains a trade-off between diversity and hardness. It seems difficult to obtain enough diversity and enough hardness simultaneously. Please provide some discussions about this concern.\n\n[1] Attacks which do not kill training make adversarial learning stronger. ICML 2020\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations: Please carefully refer to \"Cons\" in Main Review.\nNo potential negative societal impact.", "review_text": "", "extra_scores": {"ethical_concerns": "No ethical issues.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "QgNm0qWTxGz", "reviewer": "Reviewer_iAMx", "summary": "This paper presents an interesting new data augmentation called AugMax. It is motivated by diversity-hardness tradeoff of data augmentations. The authors suggest they are two complementary dimensions to achieve out-of-distribution robustness, and a disentangled normalization module termed DuBIN (Dual-Batch-and-Instance Normalization) could alleviate the resultant training difficulty. ", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n-\tStrong and clear motivation. The authors took a unified view towards two existing categories of data augmentation: random and adversarial. Either of the two was shown to improve domain shift robustness, but their complementariness was not previously revealed. The authors are the first to show a unification between the two is necessary to achieve robustness\n-\tSolid technical approach. The authors first proposed AugMax to reconcile diversity with hardness by learning an adversarially weighted combination of multiple random augmentations. Since this stronger augmentation leads to more diverse input distribution that is harder to fit, they further designed a new normalization strategy called DuBIN to disentangle the instance-wise feature heterogeneity of AugMax samples.\n-\tState of the art experimental results. Their simple approach yields significantly improved OoD robustness, outperforming strong previous methods by 2.01%, 2.52%, 1.84% and 15 2.13% on CIFAR10-C, CIFAR100-C, Tiny ImageNet-C and ImageNet-C.\n-\tConvincing ablation experiments. The authors compared different Diversity and Hardness Strategies The results confirm that diversity and hardness are two complementary dimensions, and their proper unification strategy is superior to alternative. \n-\tWriting is very clear and easy to follow. \n\nCons: \n-\tI am not fully convinced why AdvMix is not as good as AugMax in Table 10. Both seem to be sensible options to me to combine diversity and hardness. The author also did not provide an insight why.\n-\tIn Table 8, DuBIN seems to boost both AugMix and AugMax, at similar margins over DuBN. That sets doubts on whether the authors’ motivation of adopting DuBIN is really the case, e.g., because AugMax introduces a particularly heterogenous input distribution. \n-\tAugMax looks computationally heavy, with default n = 10. There was also no mention of training overhead. Please address this\n-\t(minor) The overall innovation is reasonable, but not super strong. It is mainly composedly of existing techniques: adversarial training + AugMix; and BN + IN. To be fair, packing them together is still well-motivated and smart.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No concern was found. This paper should have positive social impact.", "review_text": "", "extra_scores": {"ethical_concerns": "No concern", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "3P0oFrXJw-S", "reviewer": "Reviewer_Gx5n", "summary": "This paper proposed AugMax, which increases both the diversity and the hardness. To make the training easier, the paper design a normalization strategy DuBIN.  Experimental results show that the combination of AugMax and DuBIN exhibits good performance.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper can inspire future AT methods to deal with OOD samples. The idea is intuitive and easy to follow.\n\nMy concerns:\n1. I am confused by the DuBIN layer and not sure how it works, can you show me the formulations of the 2 normalizations?\n2. \n2. In Table 10, you reported the performance of AT on CIFAR100. Can your method be extended to the AT settings, i.e., can your method defend the adversarial examples.\n3. You state that \"AugMax can also lead to a higher feature heterogeneity, which may require larger model capacity to encode.\" Have you tried Different Normalization Layers on different models (e.g., resnet, wrn, etc.) to show how model capacity affects the performance.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Please refer to Main Review.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "17uQ8gVbces", "reviewer": "Reviewer_eujA", "summary": "This paper proposes to fuse multiple data augmentations in an adversarial manner. It achieves good robustness compared with AugMix on various natural distortions.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pos:\n1. The idea is novel and effective.\n2. The paper is well written and easy to understand.\n\nNeg:\n1. I would like to see whether the AugMax could be combined with other data augmentations like CutMix.\n2. The performance on SA seems to be comparable with AugMix. Could the authors provide results on more data augmentations?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "None.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper focuses on an interesting new data augmentation called AugMax. The proposal is to reconcile diversity with hardness by learning an adversarially weighted combination of multiple random augmentations. Since this stronger augmentation leads to more diverse input distribution that is harder to fit, they further designed a new normalization strategy called DuBIN to disentangle the instance-wise feature heterogeneity of AugMax samples. The philosophy behind sounds quite interesting to me, namely, diversity-hardness tradeoff of data augmentations. This philosophy leads to a novel algorithm design I have never seen.

The clarity and novelty are clearly above the bar of NeurIPS. While the reviewers had some concerns on the significance, the authors did a particularly good job in their rebuttal. Thus, all of us have agreed to accept this paper for publication! Please include the additional experimental results in the next version.
Decision: Accept (Poster)