Data Augmentation Can Improve Robustness
Authors: Sylvestre-Alvise Rebuffi
Sven Gowal
Dan Andrei Calian
Florian Stimberg
Olivia Wiles
Timothy Mann
Venue: NeurIPS 2021 Poster (2021)
Keywords: robustness
adversarial
augmentation

Abstract: Adversarial training suffers from robust overfitting, a phenomenon where the robust test accuracy starts to decrease during training. In this paper, we focus on reducing robust overfitting by using common data augmentation schemes. We demonstrate that, contrary to previous findings, when combined with model weight averaging, data augmentation can significantly boost robust accuracy. Furthermore, we compare various augmentations techniques and observe that spatial composition techniques work the best for adversarial training. Finally, we evaluate our approach on CIFAR-10 against $\ell_\infty$ and $\ell_2$ norm-bounded perturbations of size $\epsilon = 8/255$ and $\epsilon = 128/255$, respectively. We show large absolute improvements of +2.93% and +2.16% in robust accuracy compared to previous state-of-the-art methods. In particular, against $\ell_\infty$ norm-bounded perturbations of size $\epsilon = 8/255$, our model reaches 60.07% robust accuracy without using any external data.  We also achieve a significant performance boost with this approach while using other architectures and datasets such as CIFAR-100, SVHN and TinyImageNet.

Data Augmentation Can Improve Robustness
Sylvestre-Alvise Rebuffi*, Sven Gowal Dan Calian, Florian Stimberg. Olivia Wiles and Timothy Mann DeepMind, London
Abstract
Adversarial training suffers from robust overfitting, a phenomenon where the robust test accuracy starts to decrease during training. In this paper, focus reducing robust overfitting by using common data augmentation schemes. We demonstrate that, contrary previous findings, when combined with model weight averaging, data augmentation can significantly boost robust accuracy. Furthermore, we com- pare various data augmentations techniques and observe that spatial composition techniques work best for adversarial training. Finally, we evaluate our approach on CIFAR- 10 against fx and l2 norm-bounded perturbations of size 8/255 and c = 128/288 /255, respectively. We show large absolute improvements of +2.93% and +2.16% in robust accuracy compared previous sfae-off-the-ar methods. particular, against for norm- comn-bounded perturbations of size e 8/255, our model reaches 60.07% robust accuracy without using any external data. We also achieve significant performance boost with this approach while using other architectures and datasets such as CIFAR-100 SVHN and TINYIMAGENET.
Introduction
Despite their success, neural networks are not intrinsically robust. In particular, has been shown that the addition of imperceptible deviations to the input, called adversarial perturbations, can cause neural networks to make incorrect predictions with high confidence [ 18 32, 49). Starting with Szegedy
et al. 49 there has been a lot of work understanding and generating adversarial perturbations
2.66. and on building defenses that are robust to such perturbations [18
29
34 41. Unfortunately,
many of the defenses proposed in the literature target failure cases found through specific adversaries, and as such they are easily broken by different adversaries 3 [53] Among successful defenses are robust optimization techniques like the one by Madry al. (34 that learns robust models by finding worst-case adversarial perturbations at each training step before adding them to the training data.
In fact, adversarial training as proposed by Madry et al. is so effective (20) that it is the de facto
standard for training ad adversarially robust neural networks. Indeed, since Madry et al. (34) various modifications to their original implementation have been proposed 20
Notably, Carmon et al. 77 Hendrycks et al. (25] Najafi et al. [36], Uesato et al. [54] Zhai et al. 660 showed that using additional data improves adversarial robustness, while Gowal et al. (20), Rice et al. (44, Wu et al. (56) found that data augmentation techniques did not boost robustness. This dichotomy motivates this paper. In particular, procedure such that data augmentation becomes useful in the setting without additional data. By making the observation that model weight averaging (WA) (8 helps robust generalization to wider extent when robust overfitting is minimized, we propose to combine model weight averaging with data augmentation techniques. Overall, we make the following contributions:
explore whether it is possible to fix the training
We demonstrate that, when combined with model weight averaging, data augmentation techniques such Cutout [5], CutMix (58) and MixUp (62) can improve robustness.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Figure 1: Robust accuracy of various mod- els submitted to RobustBench [1] against AUTOATTACK [10] CIFAR-10 with
Ours (60.07%)
(Aug mentations)
Gowal et al. (57 20%
Robust test accura Cy
60%
Zhang et (53 .0%) (TRADES)
Muning)
to perturbations of size 8/255 displayed
in publication order. Our method builds on Gowal et al. [20] (shown above with 57.20%) and explores how augmented data can be used to improve robust accuracy by +2. +2.87% without using any additional exter- nal data.
50%
Madry et (44.04%)
Ricg al. (53 42%)
Early stopping)
40%
without additional data
2018
2019
2020
2021
To the contrary of Gowal et al. (20), Rice al. 44
Wu al. (56) which all tried data augmenta- tion techniques without success, we are able to use any of these three aforemennioned techniques obtain new state-of-the-an robust accuracies (see Figure 1}. We find CutMix to be the most effective method by reaching robust accuracy of 60.07% on CIFAR- 10 against &oo perturbations of size c = 8/255 (an improvement of +2.93% upon the state-of-the-e -fftthe-att)..
We conduct thorough experiments to show that our approach generalizes across architectures, datasets and threat models. We also investigate the trade-off between robust overfitting and underfitting to explain why MixUp performs worse than spatial composition techniques.
Finally, we provide empirical evidence that weight averaging exploits data augmentation by ensembling model snapshots which have the same total accuracy but differ at the individual prediction level.
2 Related Work
Adversarial f--norm attacks. Since Szegedy et al. 44 observed that neural networks which achieve high accuracy on test data are highly vulnerable to adversarial examples, the art of crafting increasingly sophisticated adversarial examples has received lot of attention. Goodfellow et [[8] proposed the Faatt Gradient Sign Method (FGSM) which generates adversarial examples with single normalized gradient step. It was followed by R+ FGSM (2), which adds a randonization step, and the Basic Iterative Method (BIM) 32] which takes multiple smaller gradient steps.
Adversarial training as a defense. The adversarial training procedure (34] feeds adversarially perturbed examples back into the training data. It is widely regarded as one of the most successful method to train robust deep neural networks. It has been augmented different ways - with changes in the attack procedure (e.g., by incorporating momentum (16)), loss function (e.g., logit pairing (35)) or model architecture (c.g.. feature denoising [57]). Another notable work by Zhang et al. [63 proposed TRADES, which balances the trade-off between standard and robust accuracy. and achieved norm-bounden bounded perturbations on CIFAR- 10. More recently, the work from Rice al. [44 studied robust overfitting and demonstrated that improvements similar to TRADES could be obtained more easily using classical adversarial training with early stopping. This later study revealed that early stopping
slat-off-the-at performance against E norm-
competitive with many other regularization techniques and demonstrated that data augmentation schemes beyond the typical random padding- -adel-and--ca crod-coopping were ineffective on CIFAR- 10. Finally, Gowal et al. (20) highlighted how different hyper-paramette (such as network size and model weight averaging) affect robustness. They were able to obtain models that significantly improved upon the tfute-ofthhe-aee data augmentation schemes. Similarly to Rice et al. [44), they also make the conclusion that data augmentations beyond random padding- and and-cropping do not improve robustness.
but lacked a thorough investigation on
Data augmentation. Data augmentation has been shown to improve the generalisation of standard (non-robust) training. For image classification tasks, random flips, rotations and crops are commonly used 23 More sophisticated techniques such as Cutour [15] (which produces random occlusions), (which replaces parts of an image with another) and MixUp (62 (which linearly
CurMix 58
interpolates between two images) all demonstrate extremely compelling results. As such, it is rather surprising that they remain ineffective when training adversarially robust networks this work, we revisit these common augmentation techniques in the context of adversarial training.
(20 4M 56) In

---

3 Preliminaries and hypothesis
The rest of this manuscript is organized as follows. In this section, we provide an overview of adversarial training and introduce the hypothesis that model weight averaging works better when robust overfitting is reduced. In (section 4 we discuss that data augmentations can be used to verify this hypothesis. Finally, we provide thorough experimental results in section 6
3.1
Adversarial training
Madry al. 34 formulate saddle point problem to find model parameters that minimize the adversarial risk:
arg min (a
max (( + 6:0),
(1)
8E3
where D is a data distribution over pairs of examples x and corresponding labels y. f(; 0) a model parametrized by 0, / is a suitable loss function (such the 0 loss in the context of classification tasks), and defines the set of allowed perturbations. For l, norm-bounded perturbations of size the adversarial set is defined as S, = {ō ||0||p I ej. In the rest of this manuscript, we will norm-bounden bounded perturbations of size (e.g.. 6x- = 8/255). To solve the inner optimization problem, Madry et al. [34] use Projected Gradient Descent (PGD), which replaces the non-differentiable 0 - loss with the cross-entropy loss Loo and computes an adversarial perturbation = (K) in K gradient ascent steps of size as
use Ep to denote lg norm-le
(k++)) projs (8(h) + sign (Vousleeef(a ((h); 0), 0).)))
(2)
where (0) is chosen at random within S, and where proja (a) projects point back onto set A.,
proja (a) = argmin lla a'12. We refer to this inner optimization with K steps as PGD
3.2 Robust overfitting
To the contrary of standard training, which often shows no overfitting in practice [6]], adversarial training suffers from robust overfitring [44). Robust overfitting the phenomenon by which robust accuracy on the test set quickly degrades while it continues to rise on the train set (clean accuracy on both sets continues improve as well). Rice et al. 44 propose use early stopping as the main contingency against robust overfitting, and demonstrate that it also allows to train models that are more robust than those trained with other regularization techniques (such as data augmentation or increased fg-reuularization). They observed that some of these other regularization techniques could reduce the impact of overfitting at the cost of producing models that are over-regularized and lack overall robustness and accuracy. There is one notable exception which is the addition of external data 7 54). Figure ( shows how the robust accuracy (evaluated on the test set) evolves as training progresses CIFAR-10 10 against 6 8/255. Without external data, robust overfitting is clearly visible and appears shortly after the learning rate is dropped (the learning rate is decayed by 10x two-thirds through training in schedule similar to 44 and commonly used since (34). Robust overfitting completely disappears when an additional set of 500K eseudo-labeled images (see Carmon et al. is introduced.
3.3 Model weight averaging
Model weight averaging (WA) (8 can be implemented using exponential moving average O' of the model parameters 0 with a decay rate r (i.e., 0' <- - - + (1 (1- at each training step). During evaluation, the weighted parameters 0' are used instead of the trained parameters 0. Chen et al. [8) Gowal et al. (20) discovered that model weight averaging can significantly improve robustness on wide range of models and datasets. Chen et al. [8] argue (similarly to (56)) that WA leads to flatter adversarial loss landscape, and thus smaller robust generalization gap. Gowal et al. [20] also explain that, in addition to improved robustness, WA reduces sensitivity early stopping. While this is true, it is important to note that WA is still prone to robust overfitting. This is not surprising, since the exponential moving average "forgets" older model parameters as training goes on. Figure 2 shows how the robust accuracy evolves training progresses when using WA. We observe that, after the change of learning rate, the averaged weights are increasingly affected by overfitting, thus resulting in worse robust accuracy for the averaged model.
3

---

63%
55%
60%
62%
extarral
27%
25%
Stp
(a) Adversarial training with and without additional data from 80M-Ti (without WA)
(b) Effect of WA without extemaa data
(c) Effect of WA with external data
We compare the robust accuracy against 6oe = 8/255 on CIFAR- 10 of an adversarially trained Wide ResNet (WRN)-28-10. Panel (a) shows the impact of using additional external data from 80M-Ti [51] and illustrates robust overfitting. Panel (b) shows the benefit of model weight averaging (WA) despite robust overfitting. (c) that WA remains effective and useful even when robust overfitting disappears. The graphs show the evolution of the robust accuracy as training progresses (against PGD4"). The jump two-thirds through training is due to drop in learning rate.
Figure 2:
3.4 Hypothesis
As WA results in flatter, wider solutions compared to the steep decrease in robust accuracy observed for SSochasticc Gradient Descent (SGD) [8) is natural to ask ourselves whether WA remains useful in cases that do not exhibit robust overfitting. Fiiuure 2(c) shows how the robust accuracy evolves as training progresses when using WA and additional external data (for which standard SGD does not show signs of overfitting). We notice that the robust performance in this setting is not only preserved but even boosted when using WA. Hence, we formulate the hypothesis that model weight averaging helps robustness maintained. This hypothesis is also motivated by the observation that WA acts as a temporal ensemble akin to Fast Geometric Ensembling by Garipov al. [17] who show that efficient ensembling can be obtained by aggregating multiple checkpoint parameters at different training times. As such, to improve robustness, it is important to ensemble suite of equally strong and diverse models. Although mildly successful, we note that ensembling has received some attention in the context of adversarial training [3 48 In particular, Grefenstette et al. [22], Tramèr et al. [22 found that ensembling could reduce the risk of gradient obfuscation caused by locally non-linear loss surfaces.
greater extent when robust accuracy between model iterations can be
4 Data augmentations
Limiting robust overfitting without external data. Rice et al. 44 show that combining data augmentation methods such as Cutout MixUp with early stopping does not improve robustness upon early stopping alone. While, these methods do not improve upon the "best" robust accuracy, they reduce the extent of robust overfitting, thus resulting in a slower decrease in robust accuracy compared to classical adversarial training (which uses random crops and weight decay). This can be seen in Figure 3here where MixUp without WA exhibits no decrease in robust accuracy, whereas the robust accuracy of the standard combination of random padding mudding-and-croppen and-cropping without WA (Pad & Crop) decreases immediately after the change of learning rate.
Testing the hypothesis. Since MixUp preserves robust accuracy while Pad & Crop does not, this comparison can be used to evaluate the hypothesis that WA is more beneficial when the performance between model iterations is maintained. Therefore, we compare in Figure 3(b) the effect of WA on robustness when using MixUp. We observe that, when using WA, the performance of MixUp surpasses the performance of Pad & Crop. Indeed, the robust accuracy obtained by the averaged weights of Pad & Crop (in blue) slowly decreases after the change of learning rate, while the one obtained by MixUp (in green) increases throughout training") Ultimately. MixUp with WA obtains a higher robust accuracy despite the fact that the non-averaged MixUp model has significantly lower "best" robust accuracy than the non-averaged Pad & Crop model. This finding is notable as it 'The accuracy drop just after the change of learning rate stems from averaging very different weights.

---

Robust tot acounoo
100%
twithout Mixup (without
Fad twith WA]
Cutouut (with WIA)
MuUUp (with WAI
Caikiix (withrut WVl
CutMix
20%
Slop
step
(a) Without WA
(h) With WA
Figure 3: Accuracy against 6oc. 8/255 on CIFAR-10 with and without using model weight
averaging (WA) for different data augmentation schemes. The model is a WRN-28-10 and both panels show the evolution of the robust accuracy as training progresses (against PGD40). The jump in robust accuracy two-thirds through training is due to a drop in learning rate.
demonstrates for the first time the benefits of data augmentation schemes for adversarial training (this contradicts the findings from three recent publications: 220 44 (66).
Exploring data augmentations. After verifying our hypothesis for MixUp, we investigate if other augmentations can help maintain robust accuracy and also be combined with WA to improve robust- ness. We concentrate on the following image patching techniques: Cutout [5 which inserts empty image patches and CutMix (58) which replaces part of an image with another. In (section 6 also evaluate RICAP (50) and SmoothMix (33]. We describe more thoroughly these augmentations in Appendix B where we also study additional augmentations with AutoAugment [12] and RandAug- ment 13 Similarly to the analysis done for MixUp, we report in Figure 3) the robust accuracy obtained by Cutout and CutMix with and without WA throughout training. First, we note that these two techniques achieve a higher "best" robust accuracy than MixUp, as shown in in/Firuee 3(a) The "best" robust accuracy obtained by Cutout and CuMMix is roughly identical to the one obtained by Pad & Crop, which is consistent with the results from Rice et al. [4] Second, while Cutout suffers from robust overfitting, CutMix does not. Hence, as demonstrated in the previous sections, we expect WA to be more useful with CutMix. Indeed, we observe in Figure 3(b) that the robust accuracy of the averaged model trained with CutMix keeps increasing throughout training and that its maximum value is significantly above the best accuracy reached by the other augmentation methods. In inssectin we conduct thorough evaluations of these methods against stronger attacks.
5
Experimental setup
Architecture.
We use WRNS 23 59 as backbone network. This is consistent with prior work
which use diverse variants of this network family. Furthermore, we adopt the same architecture details as Gowal al. (20) with Swish/SiLU 24 activation functions. Most the experiments are conducted on a WRN-28-10 10 model which has a depth of 28, a width multiplier of 10 and contains 36M parameters. To evaluate the effect of data augmentations on wider and deeper networks, we also run several experiments using WRN-70-16, which contains 267M parameters.
20
34
44
63
Outer minimization. We use TRADES (63] optimized using SGD with Nesterov momentum 37 42] and global weight decay of 5 x 10 We train for 400 epochs with a batch size of 512 split over 32 Google Cloud TPU v3 cores 4. and the learning rate initially set 0.1 and decayed by a factor 10 two-thirds-of-the oo-dd-----hh-aay way through training. We scale the learning rates using the linear scaling rule of Goyal et al. (2] (i.e., effective LR max max(LR x batch size /56, LR)). The decay rate of WA is set to 0.999. With these settings, training WRN-28-10 10 takes on average 2.5 hours.
Inner minimization. Adversarial examples are obtained by maximizing the Kullback-Leibler divergence between the predictions made on clean inputs and those made on adversarial inputs [ This optimization procedure done using the Adam optimizer (30) for 10 PGD steps. We take an initial step-size of 00.1 which is then decreased to 0.01 after 5 steps.
5

---

58%
Fad & Crop
Cutout
Mixtip
98%
RICAP
Robust test accuracy
50%
Smooth Mix
CutMix
54%
With WA
52%
(53.66%)
0.1 (55.613))
50%
1.4 (53.75%)
81%
84%
87%
20%
Clean tast accuracy
stop
Figure 4: Clean (without adversarial attacks) ac- euracy and robust accuracy (against A for WRN-28 10 trained against
Figure 5: The graph shows the robust test accu- racy against PGD with ecc 8/255 on CIFAR- 10 without using WA as we vary the mixing rate a of MixUp. We report the legend the robust ac- curacy (against A AA+MT) after applying weight averaging the corresponding runs.
8/255 on CIFAR-10 for different data augmentation tech- niques, The lines from circles to squares represent the performance change obtained when using WA.
Evaluation. We follow the evaluation protocol designed by Gowal et al. [20]. Specifically, we train two (and only two) models for each hyperparameter setting. perform early stopping for each model on separate validation set of 1024 samples using PGD similarly to Rice et al. [44] and pick the best model by evaluating the robust accuracy on the same validation set Finally, we report the robust test accuracy against mixture of AUTOATTACK [I0) and MULTITARGETED [19], which is denoted by AA+MT. This mixture consists in completing the following sequence of attacks: AUTOPGD on the cross-entropy loss with 5 restarts and 100 steps, AUTOPGD on the difference of logits ratio loss with restarts and 100 steps and finally MULTITARGETED on the margin loss with 10 restarts and 200 steps. The training curves, such as those visible in Figure 2, are always computed using PGD with steps and the Adam optimizer (with step-size decayed by 10x at step and 30).
Experimental results
First, we will investigate which augmentation techniques benefit the most from WA and why. Then, will generalize our approach to other architecture, threat model and datasets. Finally, we provide empirical evidence showing that WA exploits data augmentation by ensembling model snapshots which differ at the individual prediction level.
6.1 Comparing data augmentations
Here, we compare data augmentations with and without WA. We consider as baseline the Pad & Crop augmentation which reproduces the current state-of-the-an set by Gowal et al. [20]. This augmentation consists in first padding the image by pixels on each side and then taking a random 32 32 crop. In Figure 4 we compare this baseline with various data augmentations, MixUp, Cutout, CurMix, RICAP and SmoothMix. A first cluster (the four top squares), containing RICAP, Cutout, SmoothMix and CutMix, includes the four methods that occlude local information with patching and provide a significant boost upon the baseline with +3.06% in robust accuracy for CutMix and an average improvement of +1 1.54% in clean accuracy. The other cluster, with MixUp, only improves the robust accuracy upon the baseline by a small margin of +0.91 %. Furthermore, we also point out that Pad & Crop and Cutout, which were the two augmentations suffering from robust overfitting in Figure 3(a) benefit the least of WA in Figure 4 (smaller vertical gains). This is consistent with our hypothesis sectiin that WA is the most beneficial when robust overfitting is reduced.
MixUp. A possible explanation to the worse performance of MixUp lies in the fact that MixUp, which samples the image mixing weight with a beta distribution Beta(a, at), tends to either produce images that are far from the original data distribution (when is large) or too close to the original samples (when a is small). In fact, Figure 5] which shows the robust accuracy when training without WA, illustrates the trade-off between robust overfitting and underfitting increasing
lead

---

Robure ree setutwyy
wilh MVA
54%
526
54%
S48
536
22%
withot MV
526
529
0.1
0.3
0.5
5.7
0.0
1-
3.3
3.5
14
25
30
10
25 22
30
window srigth JOUECLET
window lrngth (CUPMic))
(a)
(b)
(c)
Figure 6: Robust test accuracy against AA+MT with Exo = 8/255 on CIFAR- as we vary (a) the mixing rate a of MixUp, (b) the window length when using Cutout and (c) the window length when using CutMix. The model is a WRN-28-10 and we compare the settings without and with WA. As reference when training only with Pad Crop. the same model with WA and without WA reaches 54.44% and 53.66% robust accuracy, respectively. Similarly, without any augmentation, the models with WA and without WA achieve 49.74% and 42.27%, respectively.
robust underfitting (red curve) while an too close would lead robust overfitting. More specifically, we show in Figure 6(a) that MixUp's pss robust accuracy against AA+MT keeps decreasing as a increases and the best performance with WA is reached at a = 0.2, corresponding to blended images close to the original images.
Spatial composition techniques. Figure 6(b,c) show the robust test accuracy as we vary the window length of the patches when using Cutout and CutMix We observe that these two techniques are the most beneficial when using large window lengths with a peak reached at a length of 20. Hence, contrary to MixUp, they work best with patched images which greatly differ from the original images. This performance gap between MixUp and Cutout/CuuMa illustrates a notable difference in the use of data augmentation for adversarial training compared to nominal training. Indeed, adversarial training can lead to underfitting. This leads to some augmentation techniques working better than others in the context of adversarial training. This is the case for spatial composition techniques which outperform blending techniques like MixUp. A possible explanation is that low-level features tend to be destroyed by MixUp, whereas composition techniques locally maintain these low-level features. Hence, we hy pothesize that augmentations designed for robustness need to preserve low-level features. We provide further evidence to support this hypothesis Appendix B By showing that some components of RandAugment such as Posterize or Invert are detrimental to adversarial robustness.
6.2 Generalizing other architectures, threat model and datasets
Generalizing to other architectures. Table shows the performance of CuMMix and the Pad & Crop baseline when varying the model architecture and size. We experiment with different variants of WideResNet and ResNet. We use WA for both CulMix and the Pad & Crop baseline. We observe that CutMix consistently outperforms the baseline by least +2.90% robust accuracy across all model sizes for WideResNet and by at least + +1.76% for ResNet models.
Generalizing to another threat model. We extend our evaluation to fz-norm bounded perturba- tions. Table 2 shows the performance of data augmentation CIFAR- 10 against 6xc 8/255 and 62 = 1288 128/255. /55.. We observe that using CunMix provides a significant boost in robust accuracy for both threat models with up +2.93% (in the foc setting) and +2.16% (in the lg setting).
Generalizing to other datasets. To evaluate the generality of our approach, we evaluate it on CIFAR-100 BI) SVHN (88 and TINYIMAGENET [45] and we report the results Table First, on CIFAR- 100, our best model reaches 32.43% against AUTOATTACK and improves noticeably on the state-of-the-art by +2.40% (in the setting that does not use any external data). Second, on TINYIMAGENET with a WRN-28-11 10 we obtain significant +2.00% boost for robust accuracy against
For all CutMix results except those reported in Fig 6(c)), the window length is randomly sampled such that the patch area ratio follows beta distribution with parameters 1 and = 1.
7

---

CRRP
CUTMIX
SETUP
CLEAN
ROBUST
CLEAN
ROBUST
SETUP
CLEAN
ROBUST
CLEAN
ROBUST
VARYING
ARCHITICTURR
W8N-28-1
ResNet- 18
83.12%
50.52%
80.57%
52.28%
R4
84,685
52.52%
83.35%
$4,80%
Gowal 20 (trained by us)
$4.32%
54.44%
88.60%
Ours (CutMix)
86.22%
57.50%
91.35%
76.12%
WRN-28-10
WRN-34-10
84.32%
84,89%
54.44%
86.09%
18.18
58,09 %
57.50%
WRN-70-16
WRN-34-20
Gowal et 120 (trained by us)
85.29 25.29%%
57. 1.1%%
90.90%
74.50%
WRN-70-16
86.02%
57.17%
87.25%
60.07%
Ours (CutMix)
87. $7.25%
60.07%
92.43%
76.66%
Table 1: Robust test accuracy (against AA+MT) against 6ge = 8/255 on CIFAR- 10 for different architectures. In all cases, we use weight averaging and we compare Pad & Crop and CutMix.
Table 2: Clean (without adversarial attacks) accuracy and robust accuracy (against AA+MT) on CIFAR- 10 we both test against 6x 8/255 and é2 128/255.
MODEL
CLEAN
AA+MT
Table 3: Clean and robust accuracy (AA+MT and AUTOATTACK for select models) on CIFAR-100, SVHN and TINY- IMAGENET against 6:c = 8 8/255 obtained by different models (with WA). The 're- trained' indication means that the models have been retrained according to Gowal et al. [20]'s methodology.
CHBRR-100
et (WRx-34-10)
60.64%
29.33%
WRN-28-11 (nttainne)
59.05%
28.75%
WaN-28-10 (CuMiss)
62.97%
30.50%
29.80%
Gowal
20 (WiN--0-166
60.86%
30.67%
30.03%
WRN-70-16 (netrained)
59.65%
30.62%
WaN-70-16 (CurMii)
65.764
33.24%
32.43%
WaN-28-H 28-028-10000 (sctriine)
92.87%
56 83%
WaN-28-10 (CuMMix)
94.52%
57.32%
TINyIsAGENET
WRN-28-10 (retrained)
53.27%
21.83%
WRN-28-10 (CutMix)
53.69%
23.83%
AA+MT with coo 8/255. Finally, on SVHN our best model reaches 57.32% against AA+MT and improves on the bascline by a smaller margin than on CIFAR-1O, CIFAR- 11FAR-100-0 or TINYIMAGENET. This smaller improvement is expected as CutMix is not suited to SVHN because images of SVHN contain multiple digits per image.
6.3 Empirical elements on how weight averaging exploits data augmentation
Motivating model ensembling. First, we show that model ensembling can be used to improve robust accuracy. To do so, we evaluate ensembling carly-stopped models which have been trained from scratch independently. We ensemble two early-stopped WRN-28-10 10 models trained on CIFAR- 10 with Pad & Crop by taking the average of the two independent models at the prediction level. In spite of this naive ensembling approach, we observe a significant boost robust performance as this ensemble reaches 55.69% robust accuracy against AA+MT compared to 54.44% with a single model. Hence, even a simple ensemble of two independent runs can exploit the variance in individual robust predictions. Actually, the boost in robust accuracy is even stronger when ensembling two early-stopped WRN-28-10 models trained with CutMix as the ensemble reaches 56.35% robust accuracy which is +3.82% better compared to an individual model. Augmentation techniques such as CutMix promote more diversity between runs than Pad Crop, leading thereby to better robust performance when ensembling. This is further evidence that ensembling by its ability of exploiting the diversity of the models is mainly responsible for robustness improvements.
Model ensembling by weight averaging. We would like to ensemble more than two models but it would be inefficient computationally and memory wise to average the predictions of many independently trained models. To circumvent this issue, the naive ensembling approach replaced that models trained
by model weight averaging as we exploit the commonly known fact [8 19 43 with adversarial training tend to be locally linear. Indeed, under the assumption of linearity, weight averaging becomes equivalent to model ensembling. Hence, instead of ensembling independently trained models, weight averaging ensembles model iterations obtained during one training run. As discussed in the previous paragraph, model ensembling improves robustness by exploiting the diversity of equally performing models so we need the model iterations used with weight averaging to have similar robust performance but also some diversity in individual robust predictions. As
8

---

58%
Robus: best scturncy
Comdly casifled
58%
Vode
54%
Foo & Crop CutMix
(32.5%)
6.9
0.95
0.99
0.9925 0.005 0.9975 0.999
Test somole Index
model weight averaging decay rate
Figure The bar plots show the outcome each individual robust prediction for different snapshots same training run of a WRN-28- 10 against 6 8/255 CIFAR- 10 without model weight averaging. The test sample indices have been re-ordered such as to show contiguous blocks. The plots show significant variation in individual robust predictions snapshots while the total robust accuracy (i.e. the number in parenthesis) remains stable.
Figure Robust test accuracy against AA+MT with Ex 8/255 on CIFAR-10 decay rate of the model weight averaging. The model is a WRN-28-10, which is trained either with CutMix or Pad Crop.
vary the
different
have previously seen in Figure 3(a) CutMix without weight averaging prevents robust overfitting and leads to flat robust accuracy after the change of learning rate. Hence, these model snapshots share the same total accuracy but we would like to know if they differ at the individual prediction level. Figure represents the individual robust predictions on test samples for three snapshots taken during training. While these three snapshots roughly have the same number of correctly classified samples, we see that there are on average 888 errors (out of 10k samples) per snapshot which are not made in the other two snapshots. This shows that augmentations which avoid robust overfitting such as CutMix produce diverse and equally performing model iterations, which can be ensembled effectively by model weight averaging, leading thereby to improved robust performance.
The limits when exploiting the diversity between model iterations. When robust overfitting occurs, weight averaging is still helpful but to lesser extent. In fact, compromise must be found between the performance boost from ensembling diverse model iterations and the performance loss from incorporating model iterations with degraded performance in the ensemble. We illustrate this point by running an ablation study in Figure measuring the robust accuracy obtained when varying the decay rate of model weight averaging and using either Pud & Crop or CuiMix. While for CuMMix increasing the weight averaging decay rate (i.e. ensembling more model iterations) always results in better robust performance, we observe that for Pad & Crop the maximum robust performance is obtained at т = 0.9925. When the weight averaging decay rate becomes too large (F 0.9925), too many model iterations with degraded performance are incorporated in the ensemble, thus hurting the robust performance the ensemble. Hence, the diversity between model iterations can only compensate up to certain point for the decrease in robust performance due to robust overfitting.
7 Conclusion
which have tried data augmentation techniques to train adversarially robust models without success, we demonstrate that combining data augmentations with model weight averaging can significantly improve robustness. We also provide insights on why weight averaging works better with data augmentations which reduce robust overfitting. We show in fact that model snapshots of a same run have the same total robust accuracy but they greatly differ at the individual prediction level, thus allowing a performance boost when ensembling these snapshots. Code and models are available online at wittttt:///. h/tp:::////ttbb wwt///epain////eeeemnddd.d -eneeeaainddreeerchhh cree/master: adversar robustness
Contrary previous works 20
44 [6],

---

References
[1] M. Andriushchenko, F. Croce, N. Flammarion, and M. Hein. Square Attack: a query-efficient black-box adversarial attack via random search. Eur. Conf. Comput. Vis., 2020.
[2] A. Athalye and I. Sutskever. Synthesizing robust adversarial examples. Int. Conf. Mach. Learn., 2018.
[3] A. Athalye, N. Carlini, and D. Wagner. Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples. Int. Conf. Mach. Learn., 2018.
[4] J. Bradhury, R. Frostig, P. Hawkins, M. J. Johnson, C. Leary, D. Maclaurin, and S. Wanderman- Milne. JAX: composable transformations of Python+NumPy programs, 2018. URL http!
//githb.. wwtchuu..om/ooggaa/ee com/google/jax
[5] N. Carlini and D. Wagner. Adversarial examples are not easily detected: Bypassing ten detection methods. In Proceedings of the 10th ACM Workshop Artificial Intelligence and Security, pages 3-14. ACM, 2017.
[6] N. Carlini and D. Wagner. Towards evaluating the robustness of neural networks. In 2017 IEEE Symposium on Security and Privacy, 2017.
[7] Y. Carmon, A. Raghunathan, L. Schmidt, J. C. Duchi, and P. S. Liang. Unlabeled data improves adversarial robustness. In Adv. Neural Inform. Process. Syst., 2019.
[8] T. Chen, Z. Zhang, S. Liu, S. Chang, and Z. Wang. Robust overfitting may be mitigated by properly learned smoothening, In International Conference Learning Representations, volume 1, 2021.
[9] F. Croce and M. Hein. Minimally distorted adversarial examples with a fast adaptive boundary at- tack, arXiv preprint arXiv: 1907.900770044 2020. URI. hhttps://aaxiiy http:::///awwwa.o/o////oo org/pdf //9f//190. 02044
[10] F. Croce and M. Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-Fe free attacks. arXiv preprint 2xXx:20003.016%0 2020.
[11] F. Croce, M. Andriushchenko, V. Sehwag, N. Flammarion, M. Chiang, P. Mittal, and M. Hein. Robustbench: a standardized adversarial robustness benchmark. arXiv preprint arXiv:2010.09670, 2020.
[12] E. D. Cubuk, B. Zoph, D. Mane, V. Vasudevan, and Q V. Le. Autoaugment: Learning augmentation policies from data. IEEE Conf. Comput. Vis. Pattern Recog., 2019.
[13] E. D. Cubuk, B. Zoph, J. Shlens, and Q. V. Le. Randaugment: Practical automated data augmentation with a reduced search space. IEEE Conf. Comput. Vis. Pattern Recog.. 2020.
[14] J. Cui, S. Liu, L. Wang, and J. Jia. Learnable boundary guided adversarial training. arXiv preprint arXiv:2011.11164 2020. http:://arrivi..rrr wwtp://o//ppppppp/..//// 11164
[15] T. DeVries and G. W. Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint
045522, 2017.
[16] Y. Dong, F. Liao, T. Pang, Su, J. Zhu, X. Hu, and Li. Boosting Adversarial Attacks with Momentum. IEEE Conf. Comput. Vis. Pattern Recog., 2018.
[17] Т. Garipov, P. Izmailov, D. Podoprikhin, D. Vetrov, and A. G. Wilson. Loss surfaces, mode connectivity, and fast ensembling of dnns. arXiv preprint arXiv:1802.10022 2018. URL https://arxiv. wrt. httg://ff/880..10000
[18) J. Goodfellow, J. Shlens, and C. Szegedy. Explaining and harnessing adversarial examples. Int. Conf. Learn. Represent., 2015.
[19] Gowal, Uesato, C. Qin, P.-. Huang, Mann, and P. Kohli. An Alternative Surrogate Loss for PGD-based Adversarial Testing, arXiv preprint arxiv: 19100099, 2019.
10

---

[20] S. Gowal, C. Qin, J. Uesato, T. Mann, and P. Kohli. Uncovering the limits of adversarial training against norm nounddddded adversarial examples. arXiv preprint arXiv:2010.03593, 2020. URL attpps//arxiss wrg/pdf//// p200 03593
[21] Goyal, P. Dollár, R. Girshick, P. Noordhuis, L. Wesolowski, A. Kyrola, A. Tulloch, Y. Jia, and K. He. Accurate, large minibatch sgd: Training imagenet in hour. arXiv preprint 1rXiv:/706.0200 2017.
[22] E. Grefenstette, Stanforth, B. O'Donoguue, J. Uesato, G. Swirszez, and Kohli. Strength in numbers: Trading-off robustness and computation via adversariillly-traine ensembles. arXiv preprint arXiv: 2018. URL httpp: :/apsii////xiiv/////a//a:a/a wwt.ootg///////oooooo//oo/oo pttp//1d//11....//11 09300
[23] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. IEEE Conf. Comput. Vis. Pattern Recog., 2016.
[24] D. Hendrycks and K. Gimpel. Gaussian error linear units (gelus). arXiv preprint arXtv:1606.08415, 2016.
[25] D. Hendrycks, K. Lee, and M. Mazeika. Using Pre- Training Can Improve Model Robustness and Uncertainty. Int. Conf. Mach. Learn., 2019.
[26] T. Hennigan, T. Cai, T. Norman, and I. Babuschkin. Haiku: Sonnet for JAX, 2020. URL http://gtt /ttt/ubgttubbbt./ com/ /wtp//depmnnddd//e dmminddddmmaaaai
[27] I.. Huang, C. Zhang, and H. Zhang. Self- Self-Adaptive Training: beyond Empirical Risk Minimiza- tion. arXiv preprint arXiv:2002. 10319, 2020.
[28] P. Izmailov, D. Podoprikhin, T. Garipov, D. Vetrov, and A. G. Wilson, Averaging Weights Leads Wider Optima and Better Generalization. Uncertainty in Artificial Intelligence, 2018.
[29] H. Kannan, A. Kurakin, and I. Goodfellow. Adversarial Logit Pairing. arXiv preprint urXiv:1803.06377 2018.
[30] D. P. Kingma and Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6680, 2014.
[31] A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[32] A. Kurakin, Goodfellow, and S. Bengio. Adversarial examples in the physical world. ICLR workshop, 2016.
[33] J.-H. Lee, M. Z. Zaheer, M. Astrid, and S.-L Lee. Smoothmix: simple yet effective data augmentation to train robust classifiers. IEEE Conf. Comput. Vis. Pattern Recog. Worksh., 2020.
[34] A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu. Towards deep learning models resistant to adversarial attacks. Int. Conf. Learn. Represent. 2018.
[35] M. Mosbach, M. Andriushchenko, T. Trost, M. Hein, and D. Klakow. Logit Pairing Methods Can Fool Gradient-Basee Attacks. arXiv preprint arXiv: 12042, 2018.
[36] A. Najafi, S.-i. Maeda, M. Koyama, and T. Miyato. Robustness to adversarial perturbations in learning from incomplete data. Adv. Neural Inform. Process. Syst., 2019.
[37] Y. Nesterov. A method of solving convex programming problem with convergence rate o1/k2). In Sov. Math. Dokl, 1983.
[38] Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Y. Ng. Reading digits in natural images with unsupervised feature learning. 2011.
[39] T. Pang, K. Xu, C. Du, N. Chen, and J. Zhu. Improving adversarial robustness via promoting ensemble diversity. Int. Conf. Mach. Learn. 2019.
[40] T. Pang, X. Yang, Dong, Xu, H. Su, and Zhu. Boosting Adversarial Training with Hypersphere Embedding. Adv. Neural Inform. Process. Syst., 2020.
11

---

[41] N. Papernot, P. McDaniel, X. Wu, S. Jha, and A. Swami. Distillation defense to adversarial perturbations against deep neural networks. IEEE Symposium on Security and Privacy, 2016.
[42] B. T. Polyak. Some methods of speeding up the convergence of iteration methods. USSR Computational Mathematics and Mathematical Physics, 1964.
[43] C. Qin, J. Martens, S. Gowal, D. Krishnan, K. Dvijotham, A. Fawzi, S. De, R. Stanforth, and Kohli. Adversarial Robustness through Local Linearization. Adv. Neural Inform. Process. Syst., 2019.
[44] L. Rice, E. Wong, and J. Z. Kolter. Overfitting in adversarially robust deep learning. Int. Conf. Mach. Learn., 2020.
[45] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy,
A. Khosla, M. Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 2015.
[46] S. Santurkar, D. Tsipras, B. Tran, A. Ilyas, L. Engstrom, and A. Madry. Image synthesis with single (robust) classifier. arXiv preprint (90i::99000095552 2019.
[47] L. Song, R. Shokri, and P. Mittal. Privacy risks of securing machine learning models against adversarial examples. In Proceedings of the 2019 ACM SIGSAC Conference Computer and Communications Security, 2019.
[48] T. Strauss, M. Hanselmann, A. Junginger, and Ulmer. Ensemble methods as defense to adversarial perturbations against deep neural networks. arXiv preprint arXiv:1709.03444 2017.
[49] C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus. Intriguing properties of neural networks. Int. Conf. Learn. Represent., 2014.
[50] R. Takahashi, T. Matsubara, and K. Uehara. Ricap: Random image cropping and patching data augmentation for deep cnns. Asian Conf. Mach. Learn., 2018
[51] A. Torralba, R. Fergus, and W. T. Freeman. 80 million tiny images: a large dataset for non- parametric object and scene recognition. IEEE Trans. Puttern Anal. Mach. Intell., 2008
[52] F. Tramèr, A. Kurakin, N. Papernot, I. Goodfellow, D. Boneh, and P. McDaniel. Ensemble Adversarial Training: Attacks and Defenses. arXiv preprint arXiv: 1705.0774 2017. URL
mttts://arxiv.cor www.aawwww.orgp///////// 07204
[53] J. Uesato, B. O' O'Donoguee, A. v. d. Oord, and P. Kohli. Adversarial Risk and the Dangers of Evaluating Against Weak Attacks. Int. Conf. Mach. Learn., 2018
[54] J. Uesato, J.-B. Alayrac, P.-S. Huang, R. Stanforth, A. Fawzi, and P. Kohli. Are labels required for improving adversarial robustness? Adv. Neural Inform. Process, Syst., 2019.
[55] R.
Wightman. pytorch- image-models 2019.
Pytorch image models.
https:://githu..com//
[56] D. Wu, S.-t. Xia, and Y. Wang. Adversarial weight perturbation helps robust generalization. Adv. Neural Inform. Process. Syst., 2020.
[57] C. Xie, Y. Wu, L. van der Maaten, A. Yuille, and K. He. Feature denoising for improving adversarial robustness. IEEE Conf. Comput. Vis. Pattern Recog., 2019.
[58] S. Yun, D. Han, S. J. Oh, S. Chun, J. Choe, and Y. Yoo. Cutmix: Regularization strategy train strong classifiers with localizable features. Int. Conf. Comput. Vis., 2019.
[59] S. Zagoruyko and N. Komodakis. Wide residual networks. Brit. Mach. Vis. Conf., 2016.
[60] R. Zhai, T. Cai, D. He, C. Dan, K. He, J. Hoperoft, and L. Wang. Adversarially Robust 2019.
Generalization Just Requires More Unlabeled Data. arXiv preprint
[61] C. Zhang, S. Bengio, M. Hardt, B. Recht, and O. Vinyals. Understanding deep learning requires rethinking generalization. Int. Conf. Learn. Represent., 2017. URL https:/ttt::/// tttt:p::///perrevies net/paffFid-SygddB3
12

---

[62] H. Zhang, M. Cisse. Y. N. Dauphin, and D. Lopez-Paz. mixup: Beyond empirical risk mini- mization, Int. Conf. Learn. Represent. 2018.
[63] H. Zhang, Y. Yu, J. Jiao, E. P. Xing, L. E. Ghaoui, and M. I. Jordan. Theoretically Principled Trade-off between Robustness and Accuracy. Int. Conf. Mach. Learn., 2019.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] We claim that data augmentation can improve adver- sarial robustness when combined with model weight averaging. Experiments show significant improvements and provide empirical evidence on how weight averaging exploits data augmentation to improve robustness.
(b) Did you describe the limitations of your work? [Yes] We discuss the limitations of data augmentation in the context of adversarial training. In particular, we show in section 6/iith the example of MixUp that there is trade-off between robust overfitting and underfitting.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] We discuss potential negative societal impacts in the conclusion.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed reproduce the main ex- perimental results (either the supplemental material or as a URL)? [Yes] The code written in JAX [4] and Haiku (26) is available online htttps://giia deepmind/
com/ wwwasteer/dveerriiarrobbbeee
deepmind deepminnd/deeppinnde
research
(b) Did you specify all the training details (e.g., data splits. hyperparameters, how they were chosen)? [Yes] All training details are in section 5]
(c) Did you report error bars (c.g., with respect to the random seed after running exper- iments multiple times)? [No] As doing adversarial training with 10 PGD steps is roughly ten times more computationally expensive than nominal training, we do not re- port error bars. Nevertheless, as a comparison point, we trained WRN-28- 10 models on CIFAR- 10 with Pad & Crop and with CutMix. The resulting robust test accuracies on CIFAR-10 against Exc = 8/255 are respectively 54.44 and 57.50+0.224 thus showing a relatively low variance in the results. Furthermore, our best models are well clear of the threshold for statistical significance.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] All details are in /scciion 5
4. If you using existing assets (e.g., code, data, models) or curating/releasing
assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] We use CIFAR-10, 100, SVHN and TINYIMAGENET.
(b) Did you mention the license of the assets? [No] Please refer to citations for details on licensing. All datasets are available for non-commercial use.
(c) Did you include any new assets either in the supplemental material or as a URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
13

---

(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "c5RMtsRJ46U", "reviewer": "Reviewer_ircd", "summary": "The main result of this paper is that some kinds of data augmentation can boost adversarial robustness when combined with model weight averaging. Prior work investigating data augmentation does not perform weight averaging, and reported that data augmentation wasn't useful. This paper reports empirical gains on common benchmarks with CutMix augmentation, and run some experiments to see where the gains in model weight averaging are coming from, the effect of unlabeled data, generalization across different architectures etc. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Clarity: Overall the paper was very well written: claims were clear and well motivated. I had a few questions about the results/writing below: \n\nl131-134: it is unclear to me why \"model weight averaging helps robustness to a GREATER extent when the accuracy across iterations can be maintained\". The plot only shows that even when there is no robust overfitting, weight averaging (WA) could be helpful\n\nl161-175: How did you decide which augmentations to try to use? In particular, why do you only study cutout, cutmix and mixup compared to other things like rotations, autoaugment etc? It does seem like the kind of data augmentation to use is quite important empirically, and it would be nice to see a more substantial discussion about this in the paper. \n\nl161-175: Relatedly, do augmentations that help standard training also help robust training (with WA)? What's a good heuristic to decide which augmentations to use in robust training vs standard training?\n\nl194: It says that you train two models for each hyperparam setting so are the reported numbers in the table the average across these models? What's the variance like? From personal experience, SVHN at \\eps = 8/255 is quite high variance across training runs so it would be good to check \n\nl221-229: The paper tries to explain why MixUp doesn't work as well - by saying that the blended images might be very far from original images. However, this could be true even for CutMix where you literally replace parts of an image with another. It would be nice to have some sample illustrations of what the augmentations look like if you believe there is a significant qualitative difference between these kinds of augmentations that lead to the result. \n\nl235-238: I didn't follow the argument here: how does adv training's underfitting determine which data augmentations work and which do not?\n\nl249-257: The gains of data augmentation when using extra data are quite small- does this improve when you train the models for longer, if underfitting is the issue?\n\nl46-48/ Sec 6.3: The empirical evidence about why data augmentation helps is not really substantiated because the current paper doesn't present results for what the model predictions look like when not using data augmentation. \n\nOther questions: \n1. What happens if we ensemble different models trained from scratch without data augmentation with early stopping? This is expensive but as per the intuition in the paper, this should also work just as well or better to improve the robust accuracy?\n2. How did you make sure that the attack evaluation was not misleading? It's great that the authors use AutoAttack which tunes hyperparameters of the attack directly, but it would be good to know what other checks the authors used to make sure that the improvements are not simply because of bad tuning of the attacks. This is a real concern because the gains are generally a bit small \n\nSignificance: This paper makes a nice observation that data augmentation with WA improves robustness. This makes the story more consistent with standard training where data augmentation typically helps. However, the gains are not big enough to have a substantial empirical advantage. Also, the paper doesn't add much new insights into why some data augmentation reduces overfitting, why do some augmentations work better with standard training vs adversarial training? So it's unclear what future works could build on this paper. \n\n\n\n\nOriginality: The paper takes two well-studied concepts (data augmentation and model weight averaging) and puts them together. This work is also very close to the work of Rice et al. that study robust overfitting. They also reported the observation that data augmentation leads to lesser overfitting i.e. drop upon training (but same best robust accuracy). This paper builds on that observation to use weight averaging to get more gains from data augmentation. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "UnJPoFEAmum", "reviewer": "Reviewer_NzdM", "summary": "The authors combine adversarial training with popular data augmentation techniques. They discuss how model averaging and data augmentation can be combined to prevent robust overfitting. The authors evaluate their method on several datasets and show state-of-the-art results on CIFAR10.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Quick justification of rating:\nI enjoyed reading this paper and think the authors did a good job first explaining, isolating and motivating the problem, then reviewing related work and possible solutions and finally, presenting their own solution along with the necessary experiments and ablation studies.\n\nI second the authors’ sentiment in line 76: “As such, it is rather surprising that they remain ineffective when training adversarially robust networks [15, 38, 49].” Data augmentation techniques are popular in e.g. domain adaptation and corruption robustness. It is an interesting idea to try to understand under which conditions data augmentation can be beneficial for adversarial robustness. Therefore, I think this paper will be relevant to the adversarial community.\n\nThe authors report a new state of the art on CIFAR10 for adversarial robustness with their method.\n\nI would appreciate it if the authors answered my two comments below in the rebuttal and also clarified them in the main paper.\n\n1) Figure 6c: Model selection. The authors show here how robust accuracy changes depending on the window length. 1) Did the authors then use the determined best value for the window length (20) to report their main results in Table 3? 2) Was this hyperparameter chosen on a separate validation set or on the CIFAR10 test set? If it was chosen on the test set, is this model selection process legit in this case? (Hyperparameters should be chosen on a separate validation set.) 3) Was the same value used to obtain the results on the other datasets in Table 2? The same questions apply to all other used data augmentation techniques, e.g. which values were used for Cutout. Maybe, it would be helpful if the authors considered subscripting the hyperparameter for the used technique, e.g. writing Cutmix_{20} if this value is different between datasets.\n\n2) No adaptive attacks were considered. Please see [1] and [2] for why adaptive attacks should always be performed when proposing a new defense. \n\n### Paper writing: very good\n\nThe paper is very well written, easy to follow, and I really enjoyed reading it. The authors motivate their method well and also provide a good context for their paper in the related work section. The methodology in section 3 is explained well, e.g. all variables are explained, the definitions are precise. The concept of “robust overfitting” and possible solutions are explained well and are easy to understand.\n\nI greatly appreciate the statement in Fig. 1: “Our method builds on Gowal et al. [15] (shown above with 57.20%) and explores how augmented data can be used to improve robust accuracy by +2.87% without using any additional external data.” This statement allows to easily place this paper in context with related work.\n\n### General comments:\n\nLine 114: Model Averaging is a popular technique in teacher-student training in domain-adaptation where the teacher model uses the EMA weights of the student model, e.g. [1]. I wonder whether there is a connection / interpretation here, similar to how MA is used in domain adaptation?\n\nLine 150: minor comment: Writing “verifying the hypothesis” reads like the authors pursue “an agenda” and design the experiment to confirm their hypothesis. I would suggest rephrasing the statement as “Testing the hypothesis”.\n\nParagraph in line 150: I think it would be important to explain how “Mixup” and “Cutmix” are implemented. For Mixup, are the authors interpolating between clean and adversarial images? Same for Cutmix, are authors replacing a part of a clean image with an adversarial image? Does it make a difference how the techniques are applied? E.g. does it make a difference for Mixup whether the mixing images are of the same kind, e.g. we interpolate only between clean images and only between adversarial images?\n\nFig3b: Could the authors explain the drop in robust accuracy after the learning rate drop with WA?\n\nLine 196: “pick the best model by evaluating the robust accuracy on the same validation set.” I realize that the authors follow the procedure in Gowal et al, but this choice seems strange to me. In [2], the authors argue that in adversarial robustness, we are interested in the worst-case robustness and this is the value that should be reported. I am therefore surprised by the unintuitive choice to pick the best model here.\n[2] Nicholas Carlini, Anish Athalye, Nicolas Papernot, Wieland Brendel, Jonas Rauber, Dimitris Tsipras, Ian Goodfellow, Aleksander Madry, Alexey Kurakin, “On evaluating adversarial robustness”\n\nI like Fig. 4 since it shows clearly that using WA is better than not using WA and the color code makes it easy to quickly find the different models. \n\nLine 221: lesser -> worse\n\nFigure 6a: Please include the number for alpha=0. Similar for 6b and 6c, please include values for not using the data augmentation techniques for comparison.\n\nSection 6.3. I found this section a bit hard to understand. I would suggest to the authors to look into [3] where the authors quantify the error consistency between different DNNs (and humans). Maybe the metrics used in [3] would be valuable for this study, and easier to understand than the current setting.\n[3] Robert Geirhos, Kristof Meding, Felix A. Wichmann, „Beyond accuracy: quantifying trial-by-trial behaviour of CNNs and humans by measuring error consistency”\n\nFigure 7 is mentioned before Figure 8. The ablation mentioned in Figure 7 does not really fit into subsection 6.3.\n\nTable 2 is mentioned after Table 3, please fix this.\n\nThe authors should write what they have put into the Supplement. Currently, the supplement is not even mentioned in the paper. Please include a paragraph detailing the additional experiments and studies.\n\nI appreciate showing results for black-box attacks in Table 4 in the Supplement. Could the authors compare to Gowal et al here as well as they have done in all other evaluations? \n\n[1] Tarvainen and Valpola: “Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results”\n[2] Florian Tramer, Nicholas Carlini, Wieland Brendel, Aleksander Madry: \"On adaptive attacks to adversarial example defenses\"\n\n### Review updates post rebuttal\n\nI read the other reviews and the authors' responses.\n\nI think the authors did a good job addressing my comments in their rebuttal. In particular, I had two main comments about hyperparameter selection and adaptive attacks. Considering my first comment: The authors wrote in their rebuttal that they used default hyperparameters from the papers where the augmentation techniques were introduced. I think this is a justified choice which means that they definitely did not \"cheat\" during model selection by choosing hyperparameters on the test set. Additionally, this choice makes their models comparable to models trained by other researchers trained with the used techniques when using the default hyperparameters. Considering adaptive attacks, the authors have a point that those attacks are primarily needed for cases when gradient obfuscation is a concern or when customized elaborate defenses are implemented which make the use of standard attacks impossible. Since they use standard training here and merely add data augmentation and EMA of the weights, gradient obfuscation isn't a concern.\n\nI think this submission is publication-worthy because:\n1.\tData augmentation is the standard \"go-to\" tool to increase robustness towards common corruptions. In self-supervised learning, for example in contrastive learning, choosing the appropriate data augmentation is key. It is peculiar that data augmentation (being such a vital ingredient in other areas of deep learning) has not worked at all so far for adversarial robustness. The authors are the first ones to show what needs to be done to make data augmentation work for adversarial robustness. This is (for me) the main novelty of this paper: The authors take a very popular technique that hasn't worked at all so far and make it work.\n2.\tThe authors report a new state of the art in adversarial robustness on CIFAR10.\n\nThus, I still think it is a good paper and should be accepted.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, the authors discussed the limitations and potential negative societal impacts of their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "DzeMbNtdUB", "reviewer": "Reviewer_Bm6o", "summary": "This paper has demonstrated that data augmentation can be one key factor for preventing robust overfitting. This is an interesting observation as it contradicts the previous studies. The author further combined augmentations with weight averaging to improve the robustness. The experiments are extensive and show impressive results.", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "----------------------------------------------------------------------\n**Pros**\n1. **The finding is interesting that data augmentation can prevent robust overfitting**\n\n2. **The experiments are extensive and show significant improvement**\n\n3. **The writing and presentation are clear**\n----------------------------------------------------------------------\n**Cons**\n1. **Details are missing: cannot find the training objective or algorithm table**\\\nDid the author utilize CutMix after generating the adversarial example, or first apply CutMix then generate the adversarial example; an ablation study is needed (which one is better and why?)\\\nHow did the author modify the TRADES [8] objective with CutMix...? I believe this part is not trivial as it contains KL divergence in the loss formula.\n\n2. **Concerns about the analysis of data augmentation(DA): Why is CutMix the most effective method**?\\\nIs it related to the clean accuracy of standard ERM training?\\\nIf so, did the author have tried AutoAugment [4] or RandAugment [5] (which shows the state-of-the-arts performance)? Since these techniques do not need other instances to define the augmentation (unlike CutMix), hence, it is more simple to analyze\n\n3. **Concerns about the analysis of data augmentation(DA):  Which is more critical for improving the robustness, DA, or label mixing**?\\\nSince the mixing methods are a combination of augmentation and label mixing [3], maybe decomposing the effect might help for the analysis.\\\nE.g., in Figure 6, the CutMix is even effective under a window length of 30, which is extremely interesting. The window length of 30 will not contain an important foreground feature of another instance. This result might indicate the label mixing (or smoothing) is essential.\n\n4. **Concerns about the novelty**\\\nThe method itself is not specified for adversarial robustness, where it can be applied to various classification problems.\\\nAlso, It seems that weight averaging (WA) and DA is an orthogonal contribution, as the DA prevents the overfitting and WA is for improving the generalization. Note that [2,3] already has shown that loss landscape smoothing is effective for robust overfitting and generalization (as mentioned in line 118).\n\n----------------------------------------------------------------------\n**Questions**\n1. Is it necessary to use WA rather than Sharpness-aware-minimization (SAM) [6]? Since SAM is known to be an advanced version of WA, it might show better performance. Note that SAM and AWP [1] share the same idea.\n\n2. The author used Adam optimizer for the inner maximization, which is not common (I only have seen such optimization in local linear regularization [7]). Is there a reason that the authors used Adam rather than the common PGD?\n\n3. Why is the combination of AutoAttack and Multi-target attack is less effective than the AutoAttack…? \n\n4. In Figure 1, which model architecture is used for the comparison?\n\n5. What does \"(retrained)\" indicate in Table 2? Especially for SVHN and TinyImageNet. Is it the retrained version of Cui et al., or Gowal et al.?\n\n----------------------------------------------------------------------\n**To sum up, I recommend (weak) rejection for this conference**.\\\nI believe the author needs more analysis on the data augmentation as it is the main contribution of this paper. Also, building a method that is specified for adversarial robustness will be great. \n\nAlso, some details are missing or questionable. I did not consider this as a negative factor for the initial review, however, the author needs to some parts for better clarification.\n\n----------------------------------------------------------------------\n**You can find the references here**\\\n[1] “Adversarial Weight Perturbation Helps Robust Generalization”, Wu et al., NeurIPS 2020\\\n[2] “Robust Overfitting may be mitigated by properly learned smoothening”, Chen et al., ICLR 2021\\\n [3] “On the Generalization Effects of Linear Transformations in Data Augmentation”, Wu et al., ICML 2020\\\n[4] “AutoAugment: Learning Augmentation Strategies from Data”, Cubuk et al., CVPR 2019\\\n[5] “RandAugment: Practical automated data augmentation with a reduced search space”, Cubuk et al., NeurIPS 2020\\\n[6] “Sharpness-Aware Minimization for Efficiently Improving Generalization”, Foret et al., ICLR 2021\\\n [7] “Adversarial Robustness through Local Linearization”, Qin et al., NeurIPS 2019\\\n[8] “Theoretically Principled Trade-off between Robustness and Accuracy”, Zhang et al., ICML 2019", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, they have discussed the limitations and societal impacts.", "review_text": "", "extra_scores": {"ethical_concerns": "No ethical concerns.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "AdIWwtz5Ubw", "reviewer": "Reviewer_GNo2", "summary": "This paper demonstrates that data augmentation with a weight average can mitigate the adversarial overfitting problem and also can boost the robustness with a large margin. Especially, the author found that cutmix and weight average work the best for adversarial training with large performance gain which is state-of-the-art.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: \nThis method is not a novel method combination of cutmix and weight average, however, demonstrates the adversarial training also can get benefit from data augmentation and weight average technique for the first time.  Moreover, the technique can also prevent the robust overfitting problem.\n\nQuality: \nThe submission is technically sound. However, it is a bit unclear that whether a weight average boost the performance of robustness or data augmentation does. Also, does weight average help in mitigating overfitting problems or augmentation. Cutmix itself already did not suffer any overfitting while with weight average it can gain large performance gain. However, in pad&crop, weight average seems to mitigate the overfitting problem but it did not boost the performance that large enough.\n\nAs I understand, the main message is augmentation helps in robustness. However, the paper mainly figures out that model weight averaging helps robustness to a greater extent when robust accuracy between model iterations can be maintained which seems different content from the title and abstract.\n\nClarity: Submission is easy to understand. However, it was hard to follow what is the main claim of the paper as I aforementioned.\n\nSignificance: Results are interesting that most of the previous work claim that augmentation does not give any benefits to robustness which was wrong in some sense. They provide thorough experiments on different sizes of architecture and diverse datasets. However, since the proposed method is just combining existing methods and didn't demonstrate any plausible reason why those two techniques help in overfitting and performance, I am not sure this work is significant enough.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Despite the different scores, all four reviewers agree on two main points:

A) The empirical results, in particular the non-trivial robustness gains, are interesting.
B) The explanations given for the robustness gains are only preliminary and do not account for all observed phenomena (e.g., why certain augmentations work better than others).

The reviews then arrive at different scores based on how much importance they assign to each point.

Overall I agree with both points and see this as a borderline paper. The authors could have written a stronger paper by avoiding tenuous explanations and instead focusing on a thorough experimental evaluation. A paper with strong empirical results does not have to offer a comprehensive explanation for each observed phenomenon. Instead, it suffices to discuss possible hypotheses as directions for future work. If the authors want to provide explanations, they should be rigorously investigated with experiments designed specifically to test the explanations. Presenting explanations without corresponding experiments undermines the validity of the paper.

Nevertheless, the experimental results still outweigh the lack of rigor in the provided explanations for me and I recommend accepting the paper. For the final version, I suggest that the authors clearly separate experimental findings from potential explanations. The paper could benefit from de-emphasizing the latter, e.g., by moving some of the explanations to the appendix and instead moving more of the adversarial evaluation details to the main text.
Decision: Accept (Poster)