Data augmentation alone can improve adversarial training
Authors: Lin Li
Michael W. Spratling
Venue: ICLR 2023 poster (2023)
Keywords: deep learning
adversarial training
data augmentation
adversarial robustness

Abstract: Adversarial training suffers from the issue of robust overfitting, which seriously impairs its generalization performance. Data augmentation, which is effective at preventing overfitting in standard training, has been observed by many previous works to be ineffective in mitigating overfitting in adversarial training. This work proves that, contrary to previous findings, data augmentation alone can significantly boost accuracy and robustness in adversarial training. We find that the hardness and the diversity of data augmentation are important factors in combating robust overfitting. In general, diversity can improve both accuracy and robustness, while hardness can boost robustness at the cost of accuracy within a certain limit and degrade them both over that limit. To mitigate robust overfitting, we first propose a new crop transformation Cropshift with improved  diversity compared to the conventional one (Padcrop). We then propose a new data augmentation scheme, based on Cropshift, with much improved diversity and well-balanced hardness. Empirically, our augmentation method achieves the state-of-the-art accuracy and robustness for data augmentations in adversarial training. Furthermore, it matches, or even exceeds when combined with weight averaging, the performance of the best contemporary regularization methods for alleviating robust overfitting.

Published as a conference paper at ICLR 2023
DATA AUGMENTATION ALONE CAN IMPROVE ADVER- SARIAL TRAINING
Lin Li & Michael Spratling Department of Informatics King's College London 30 Aldwych, London, WC2B 4BG, UK
1lin.3.1i. michael str.lllspaaannngg (eccnggggkkkkecco .ck.uk
ABSTRACT
Adversarial training suffers from the issue of robust overfitting, which seriously impairs generalization performance. Data augmentation, which effective at preventing overfitting in standard training, has been observed by many pre- vious works to be ineffective in mitigating overfitting in adversarial training. This work proves that, contrary to previous findings, data augmentation alone can significantly boost accuracy and robustness adversarial training. We find that the hardness and the diversity of data augmentation are important factors in combating robust overfitting. In general, diversity can improve both accu- racy and robustness, while hardness can boost robustness the cost of accu- racy within certain limit and degrade them both over that limit. To miti- gate robust overfitting, we first propose a new crop transformation, Cropshift, which has improved diversity compared to the conventional one (Padcrop). We then propose a new data augmentation scheme, based on Cropshift, with much improved diversity and well-balanced hardness. Empirically, our augmentation method achieves the state-of-the-ar accuracy and robustness for data augmen- tations in adversarial training. Furthermore, when combined with weight aver- aging matches, or even exceeds, the performance the best contemporary regularization methods for alleviating robust overfitting. Code is available at: www.:/cew...ceccc.eaeoooo. Alone Improves-AT
INTRODUCTION
Adversarial training, despite its effectiveness in defending against adversarial attack, is prone to overfitting, Specifically, while performance on classifying training adversarial examples improves during the later stages of training, test adversarial robustness degenerates. This phenomenon is called robust overfitting (Rice et al. 2020p. To alleviate overfitting, Rice al. (2020) propose to track the model's 's robustness on a reserved validation data and select the checkpoint with the best validation robustness instead of the one the end of training. This simple technique, named early- stopping (ES), matches the performance of contemporary state-of-the-arr methods, suggesting that overfitting in adversarial training impairs its performance significantly. Preventing robust overfitting is, therefore, important for improving adversarial training.
Data augmentation is an effective technique alleviate overfitting in standard training, but it seems to not work well in adversarial training. Almost all previous attempts (Rice et al., 2020) Wu et al. 2020 Gowal al. 2021a Rebuffi et al. 202T, Carmon et al., 2019) to prevent robust overfitting by data augmentation have failed. Specifically. this previous work found that several advanced data augmentation methods like Cutout Vrevs & Taylor 2017). Mixup (Zhang et al., 2018) and Cut- mix (Yun et 2019) failed to improve the robustness of adversarially-tranee models to match that of the simple augmentation Flip-Padcrop with ES, as shown in Fig. [ Thus the method of using ES with Paid-Pac has been widely accepted as the "baseline" for combating robust overfitting. Even with ES, Cutout still fails to improve the robustness over the baseline, while Mixup boosts the robustness marginally (< 0.4%) RRiee et al. 2020 Wu et al. 2020). This contrasts with their excellent performance in standard training. Recently, Tack et al. (2022) observed that AutoAugment (Cubuk al. 2019) eliminate robust overfitting and boost robustness greatly. This, however,

---

Published as a conference paper at ICLR 2023
53
Our method is the only one that sig- nificantly improves both accuracy and robust- ness over the baseline (Flip-Padcrop with early- stopping). Cutout and Cutmix fail to beat the baseline regarding robustness. AutoAugment achieves only a small improvement on robust- ness over the bascline. Robustness is evaluated against AutoAttack. See Section S for details of training and evaluation settings.
Figure 1:
51
Plip-PPadcoop
Mllii-Padorop+ES
Robustness
49
A Cutout
47
Cutmix
AutoAugment
45
baseline
ours
43
X ours+SWA+ES
41
82
83
84
Accuracy
contradicts the result Gowal et al. (2021a); Carmon et al. (2019) where the baseline was found to outperform AutoAugment terms of robustness. Overall, to date, there has been no uncontrover- sial evidence showing that robust generalization can be further improved over the baseline by data augmentation alone, and no convincing explanation about this ineffectiveness.
This work focuses improving the robust generalization ability of adversarial training by data augmentation. We first demonstrate that the superior robustness AutoAAggentt claimed by TTck et al. (2022) is actually a false security since its robustness against the more reliable AutoAAtack (AA) (Croce & Hein 2020) (48.71%) 71))
1 is just slightly higher than the baseline's ' (48.21% 21% as shown in Fig. (see Appendix A a detailed discussion). We then investigate the impact of the hardness and diversity of data augmentation on the performance of adversarial training. It is found that, in general, hard augmentation can alleviate robust overfitting and improve the robustness but at the expense of clean accuracy within certain limit of hardness. Over that limit, both robustness and accuracy decline, even though robust overfitting is mitigated more with the increase in hardness. On the other hand, diverse augmentation generally can alleviate robust overfitting and boost both accuracy and robustness. These results give us the insight that the optimal data augmentation for adversarial training should have much diversity as possible and well-balanced hardness.
To improve robust generalization, we propose new image transformation, Cropshift, a more di- verse replacement for the conventional crop operation, Padcrop. Cropshift is used as component in a new data augmentation scheme that we call Improved Diversity and Balanced Hardness (IDBH). Empirically, IDBH achieves the state-of-the-art robustness and accuracy among data augmentation methods in adversarial training. improves the end robustness to be significantly higher than the robustness of the baseline augmentation with early-stopping (Fig. [ which all previous attempts failed to achieve. Furthermore, it matches the performance of the state-of-the-ar regularization methods for improving adversarial training and, when combined with weight averaging, consider- ably outperforms almost all of them in terms of robustness.
2 RELATED WORKS
Robust overfitting can be successfully mitigated by smoothing labels, using Knowledge Distilla- tion (KD) (Chen et al. 2021) and Temporal Ensembling (TE) (Dong al. 2022), and/or smooth- ing weights using Stochastic Weight Averaging (SWA) (Chen et al. 2021) and Adversarial Weight Perturbation (AWP) (Wu et al., 2020)). Moreover, [Singla al. 2021) found that using activation functions with low curvature improved the generalization of both accuracy and robustness. Alterna- tively, Yu et al. (022)) attributed robust overfitting the training examples with small loss value, and showed that enlarging the loss of those examples during training, called Minimum Loss Con- strained Adversarial Training (MLCAT), can alleviate robust overfitting. Our work prevents robust overfitting by data augmentation, and hence complements the above methods.
To date, is still unclear if more training data benefits generalization in adversarial training. Schmidt et al. (2018) showed that adversarial training requires more data, compared to its standard training counterpart, to achieve the same level of generalization. In contrast, Min et al. (2021): Chen et al. (200)) proved that more training data can hurt the generalization in some particular adversarial training regimes on some simplified models and tasks. Empirically. a considerable improvement has been observed in both clean and robust accuracy when the training set is dramatically expanded, in a remi-supervisee way, with unlabeled data (Carmon et al.] 2019 Allayrac et al., 2019). e.g., using Robust Training (RST) (Carmon et al. 2019) with synthetic data generated by generative
2

---

Published as a conference paper at ICLR 2023
model (Gowal et al. 2021b). Although data augmentation alone doesn't work well, it observed to improve robustness to a large degree when combined with SWA (Rebuffi et al., (021) or Consis- tency (CONS) regularization (Tack et al. 2022). In contrast, our work doesn I require any additional data or regularization: it improves robust generalization by data augmentation alone.
Common augmentations (He al. 2016a) used in image classification tasks include Padcrop (padding the image at each edge and then cropping back to the original size) and Horizontal Flip. Many more complicated augmentations have been proposed to further boost generalization. Cutout (DeVries & Taylor 2017) and Random Erasing (Zhong et al.] 2020) randomly drop region in the input space. Mixup (Zhang et al., 2018)) and Cutmix (Yun al. 20199 randomly interpolate two images, as well as their labels, into new one. AutoAugment (Cubuk et al. 2019) employs a combi- nation of multiple basic image transformations like Color and Rotation and automatically searches for the optimal composition of them. TrivialAugment (Müller & Hutter 2021) matches the perfor- mance of AutoAugment with similar schedule yet without any explicit search, suggesting that this computationally expensive process may be unnecessary. The method proposed here improves on the above methods by specifically considering the diversity and hardness of the augmentations. The dif- ference between data augmentation in standard and adversarial training is discussed in Appendix B
How DATA AUGMENTATION ALLEVIATES ROBUST OVERFITTING
This section describes an investigation into how the hardness and the diversity of data augmentation effects overfitting in adversarial training. During training, the model's robustness was tracked at each epoch using PGDIO applied to the test set. The checkpoint with the highest robustness was selected as the "best" checkpoint. Best (end) robustness/accuraccy refers the robustness/accuracy of the best (last) checkpoint. In this section, the terms accuracy and robustness refer to the end accuracy and robustness unless specified otherwise. The severity of robust overfitting was measured using the best robustness minus the end robustness. Hence, the more positive this gap in robustness the more severe the robust overfitting. The training setup described in Appendix C
3.1 HARDNESS
Hardness was measured by the Affinity metric (Gontijo-Lopes et al. 202T) adapted from standard training:
hardness Robustness( M, Dreat)
(1)
Robustness D'est)
where M is an arbitrary model adversarially trained on the unaugmented training data. Dtest refers to the original test data set and D'est is Drest with the augmentation (to be evaluated) applied. Robustness( M, D) is the robust accuracy of M evaluated using PGI550 D. Hardness is model-specific measure. It increases as the augmentation causes the data to become easier to attack, i.e., as the perturbed, augmented, data becomes more difficult to be correctly classified.
We found that moderate levels hardness can alleviate robust overfitting and improve the robustness but at the price of accuracy. Further increasing hardness causes both accuracy and robustness to decline, even though robust overfitting is alleviated further. The value of hardness where this occurs very sensitive to the capacity of the model. Therefore, to maximize robustness, hardness should be carefully balanced, for each model, between alleviating robust overfitting and impairing accuracy.
Experimental design. We investigated the effects of hardness in adversarial training for individual and composed augmentations. For the individual augmentations the following 12 image transforma- tions were choosen: ShearX, Shear Y. TranslateX, Translate Y, Rotate, Color, Sharpness. Brightness, Contrast, Solarize, Cutout and Cropshift (a variant of Padcrop introduced in Section 4 For each Eq. (T) was used to calibrate the strength of the augmentation (e.g. angle for Rotation) onto one of 7 levels of hardness (see Appendix C.2 for specific values), except for Color and Sharpness which were applied at 3 strengths. For simplicity, the integers to 7 are used to represent these 7 degrees of hardness. Standard Cutout is allowed to cut partially outside the image, and thus the hardness is not always directly related to the nominal size of the cut-out region (strength). To ensure alignment between strength and hardness, we force Cutout to cut only inside the image and refer this variant as Cutout-i. To control for diversity, each augmentation was applied with one degree of diversity. As
3

---

Published as a conference paper at ICLR 2023
None Rolate
ShearX ShearY
Cropshift-1
Contrast Solarize
Brightness
Cutout-i-1
Sharpness
79
43
11
Gap of Robustness
End Rabustness
End Accuracy
75
39
7
73
37
71
35
0
2
4
5
6
7
0
1
2
3
4
7
8
0
2
3
4
5
6
7
8
Hardness
Hardness
Hardness
(a)
(b)
(c)
11
Gap of Robustness
End Acouracy
78
End Robustness
43
76
74
39
72
37
-1
70
None FP
CW
CS AuA
TA
None
None
CW
CS AuA
TA
(d)
(e)
(f)
Figure 2: The performance of models trained with different individual transformations (top row) and
composed augmentations (bottom row). None refers to no data augmentation applied. Robustness is evaluated against PGD50.
result the effects of applying Cutout and Cropshift with certain hardness is deterministic through- training. Specifically, Cutout always cuts the fixed location (sampled once the beginning of training). Similarly, CropShift crops a fixed region and shifts it to the fixed location (both sam- pled once at the beginning of training). We name these two variants as Cutout-i-1 and CropShift-1 respectively. Models were trained with each transformation at each hardness.
To investigate composed augmentations, models were trained with various multi-layered data augmentations: Flip-Padcrop (FP), FP-Cutoot[Weak] (CW), FF-CCutou[[Stoong] (CS), FP- AutoAugment-Cunr Cutout (AuA) and TggirivialAugmmett -aralll-Auumeettnuuututttt (TA). All of them shared the same parameters for Flip and Paderop. CW and CS used 8x8 and 10x10 Cutout respectively. AuA and TA used 16x 16 Cutout as in their original settings (Cubuk et al. 2019: Muller & Hutter (02II). Different from the default experimental setting, augmentations here robustness was evaluated against AA since AuA was observed to fool the PGD attack (Appendix (A).
always applied during training, and
Hardness increases from FP TA more layers stack up and/or the strength of individual compo- nents increases. Hence, this experiment can, as for the experiments with individual augmentations, be seen an investigation into the effects of increasing hardness. Here, we are not controlling for diversity, which also roughly increases from FP to TA. However, this does not affect our conclusions as diversity boosts accuracy and robustness (see Section(.22), and hence, the decrease in these values that we observe with increased hardness cannot be explained by the effects of increased diversity.
Observations. It can be seen that the gap between best and end robustness drops, i.e., robust overfitting turns milder with the increase in hardness in Figs, 2a and 2d The gap of robustness for AuA is negative in Fig. 2d because the PGDI0 attack was fooled to select vulnerable checkpoint as the best: see Appendix A for more discussion. For accuracy and robustness, there are roughly three phases. First, both accuracy (Fig. (b) and robustness (Fig. 2c) increase with hardness. This is only observed for some transformations like Cropshift at hardness and 2. In this stage, the underlying model has sufficient capacity to fit the augmented data so it benefits from the growth of data complexity.
Second, accuracy (Fig. 2b) starts to drop while robustness (Fig. 2ch continuously increases. As the intensity of the transformation increases, the distribution of the transformed data generally deviates more from the distribution of the original data causing the mixture of them to be harder fit for
4

---

Published as a conference paper at ICLR 2023
44
78
Gap of Robustness
h=1
End Robustness
9-
42
End Accuracy
77
7
h=2
76
h=3
36
75
10
6
8
10
0
2
8
10
No. of Transforaatioss
No. Transformations
No. of Transformations
(a)
(b)
(c)
13
47
79
Gap of Robustness
11
Cutout-i-1
End Robusiness
45
Ind Accuracy
78
9-
43
77
7-
41
5
Cutout
Cropshiff-1
39
3
37
Cropshift
35
74
0
2
3
4
5
6
7
3
8
6
Hardness
Hardness
Hardness
(d)
(e)
(f)
None Rotate
ShearX
TranslateX * Cropshift-1
Contrast
Brightness
ShearY
TranslateY
Cutout-i-1
Solarize
11
44
78
Gap Robustness
End Robustness
9
42
Accuracy
77
7-
40
76
5-
38
75
74
5
Length of Hardness Range
Length of Hardness Range
Length of Hardness Range
(h)
Figure 3: Performance of the models trained using augmentations with different type diversity (top row), spatial diversity (middle row) and strength diversity (bottom row). Spatial diversity rises from restricted variant (solid lines) to the unrestricted variant (dashed lines) for the same transforma- tion (line color) in Figs. 3d 3e and 31 Robustness is evaluated against PGD50.
standard training, Adversarial training can be considered as standard training plus gradient regular- ization (Li & Spratling 2022) Roughly speaking, accuracy drops in this stage because the model's capacity is insufficient to fit the increasingly hard examples for the optimization of the clean loss (standard training) under the constraint of gradient regularization. Nevertheless, robustness could still increase due to the benefit of increasing robust generalization, i.e., smaller adversarial vulner- ability. Third, accuracy (Fig. 2eb and robustness (Fig- 2 drop together. Accuracy continues to decline due to the severer (standard) underfitting. Mcanwhile, the harm of decreasing accuracy outweighs the benefit of reduced robust overfitting, which results in the degeneration of robustness.
The graphs of Color, TranslateX and Translate are omitted from Figs. 2a 2b
and because
they exhibit exceptional behavior at some values of hardness. Nevertheless, these results generally show that robust overfitting is reduced and robustness is improved. These results presented and discussed in Appendix [D.]] Appendix D.2 provides figure showing best accuracy as function of hardness for individual augmentations. more obvious downward trend with increasing hardness can be seen in this graph compared to the graph for end accuracy shown in Fig. 2b
3.2 DIVERSITY
To investigate the effects of augmentation diversity, variation in the augmentations was produced in three ways: (1) using varied types of transformations ("type diversity""); (2) varying the spatial area

---

Published as a conference paper at ICLR 2023
to augment ("spatial diversity"); and (3) increasing the variance of the strength while keeping the mean strength constant (" ("strength diversity").
Type diversity. We uniformly drew M transformations, with fixed and roughly equivalent hard- ness, from the same pool of transformations as the individual experiment in Section 3.1 but excluding Cutout and Cropshift. During training, one of these M transformations was randomly sampled, with uniform probability, to be applied to each sample separately in batch. Diversity increases M increases from O (no augmentation applied) to 10 (one augmentation from a pool of 10 applied). The experiments were repeated for three levels of hardness {1, 2, 3). For all levels of hardness, the gap of robustness (Fig. Ba) reduces, and the end robustness (Fig. 3b) and accuracy (Fig. 3cb increases, as M increases. These trends are more pronounced for higher hardness levels.
Spatial diversity. Transformations like Cutout and Cropshift (described in Section 4 have large inherent diversity due to the large number of possible crop and/or shift operations that can be per- formed on the same image. For example, there are 28x28 possible 4x4 pixel cutouts that could be taken from a 32x32 pixel image, all with the same strength of 4x4. In contrast, transformations like Shear and Rotation have only one, or if sign is counted, two variations at the same strength, and hence, have much lower inherent diversity. To investigate the impact of this rich spatial di- versity, we run the experiments to compare the performance of Cutout-i-l with Cutout-i, and to compare Cropshift-1 and Cropshift, at various levels of hardness. In both cases the former variant of the augmentation method is less diverse than the latter. We observed that the rich spatial diversity in Cutout-i and Cropshift helps dramatically shrink the gap between the best and end robustness (Fig. 3d). and boost the end robustness (Fig. 3eb and accuracy (Fig. 3f at virtually all hardness.
Strength diversity. Diversity in the strength was generated by defining four ranges of hardness: (4), (3,,,5), (2,3,4,5, 3, 6} and (1, 21, 4,5, 7]. During training each image was augmented using a strength uniformly sampled at random from the given range. Hence, for each range the hard- ness of the augmentation, on average, was the same, but the diversity of the augmentations increased with increasing length of the hardness range. Models trained with differing diversity were trained with each of the individual transformations defined in Section 3.1 excluding Color and Sharpness. Strength diversity for Cutout-1, Cropshift-1 I and Rotate can be seen to significantly mitigate robust overfitting (Fig. (3g) and boost robustness (Fig. 3h), whereas for the other transformations seems have no significant impact on these two metrics. Nevertheless, clear increase in accuracy (Fig. 3i) is observed when increasing strength diversity for almost all transformations.
4 DIVERSE AND HARDNESS- -BLLNSEE DATA AUGMENTATION
This section first describes Cropshift, our proposed version of Padcrop with enhanced diversity and disentangled hardness. Cropshift (Fig. + Algorithm I first randomly crops a region in the image and then shifts it around to a random location in the input space. The cropped region can be either square or rectangular. The strength of Cropshift is parameterized by the total number, N, of cropped rows and columns. For example, with strength 8, Cropshift removes 1, f, b lines from the left, right, top and bottom borders respectively, such that + r + t + b b=8 8. Cropshift significantly diversifies the augmented data in terms of both the content being cropped and the localization of the cropped content in the final input space. Furthermore, Cropshift offers a more fine-grained control on the hardness. In contrast, for Paderop hardness is not directly related the size of the padding, as for example, using 4 pixel padding can results in cropped images with variety of total image content that is trimmed (from rows and columns trimmed, to no rows and no columns trimmed).
To mitigate robust overfitting, we propose a new data augmentation scheme with Improved Diver- sity and Balanced Hardness (IDBH). Inspired by Müller & Hutter (202T), we design the high-level framework of our augmentation as 4-layer sequence: flip, crop, color/shape and dropout. Each has distinct semantic meaning and is applied with its own probability. Specifically, we implement flip using Horizontal Flip, crop using Cropshift, dropout using Random Erasing, and color/shape using a set of Color, Sharpness, Brightness, Contrast, Autocontrast, Equalize, Shear (X and Y) and Rotate. The color/shape layer, when applied to augment an image, first samples transformation cording to a probability distribution and then samples a strength from the transformation' 's strength range. This distribution and the strength range of each component transformation are all theoret-
6

---

Published as a conference paper at ICLR 2023
Algorithm 1. Pscudo code of Cropshift.
Note: randi(N) uniformly samples an integer be- tween [0, N).
shift
w, h = get image size(img)
/1 crop region of orig. image cropa, cropy randi(N), N crope topz, topy = randi randi(cropy) w', h'
pad
cropæ, h cropy cropped = crop(img, topæ, topy, w', h') // shift the cropped region I, r randi(cropz). randi(cropy) aug zeros like (img) angle x x + w', y h²) = cropped
Figure 4: Illustration Cropshift (top) and Pad- crop (bottom) with equivalent hardness.
ically available optimize. Pseudo-code for the proposed augmentation procedure can be found in Appendix
The probability and the strength of each layer was jointly optimized by heuristic search max- imize the robustness. It is important to optimize all layers together, rather than individually. First, this enables more extensive and more fine-grained search for hardness so that a potentially better hardness balance can be attained. Moreover, it allows certain hardness to be achieved with greater diversity. For example, raising hardness through Cropshift also improves diversity, while doing so through the color/shape layer hardly increases diversity. However, optimizing the parameters of all layers jointly adds significantly to the computational burden. To tackle this issue, the search space was reduced based insights gained from the preliminary experiments and other work, and grid search was performed only over this smaller search space. Full details are given in Appendix D A better augmentation schedule might be possible if, like AuA, more advanced automatic search was applied. However, automatically searching data augmentation in adversarial training is extremely expensive, and was beyond the resources available us.
IDBH improves diversity through the probabilistic multi-layered structure which results in a very diverse mixture of augmentations including individual transformations and their compositions. We further diversify our augmentation by replacing the conventional crop and dropout methods, Padcrop and Cutout, in AuA and TA with their dliversityenhhanee respectively. IDBH enables balanced hardness, the structure design and optimization strategy produce much larger search space of hardness, so that we are able find an augmentation that achieves a better trade-off between accuracy and robustness.
variants Cropshift and Random Erasing
RESULTS
We adopt the following setup for training and evaluation (fuller details in Appendix C The model architectures used were Wide ResNet34 with widening factor of I (WRN34-1), its 10x widened version WRN34-10 (Zagorykoo & Komodakis) (016) and PreAct ResNet18 (PRN18) (He et al. 2016b). For PRN18, report the result of two variants, weak and strong, of method with slightly different parameters (hardness), because we observed a considerable degradation of best robustness the strong variant when combined with SWA.
5.1 STATE-OF-THE-ART DATA AUGMENTATION FOR ADVERSARIAL TRAINING
From Tab. it can be seen that our method, IDBH, achieves the state-of-the-art best and end robust- ness for data augmentations, and its improvement over the previous best method is significant. The robust performance is further boosted when combined with SWA. Moreover, our method is the only one WRN34-1 that successfully improves the end robustness to be higher than the best robustness achieved by the baseline. On PRN18. IDBH[strong] improves the end robustness over the baseline's s best robustness by -1.8%% which much larger than the existing best record (+0.5%) achieved by AuA. This suggests that data augmentation alone, contrary to the previous failed attempts, can sig- nificantly beat the baseline augmentation with ES. More importantly, method also presents the

---

Published as a conference paper at ICLR 2023
Table 1: Performance various data augmentation methods (w.o. SWA) and their weight-averaged variants (w. SWA) for WRN34-1 and PRN18 on CIFARIO. The best record is highlighted for each metric in each block. All methods were trained the same way, except that Cutmix was trained longer for convergence as in Rebuffi et al. (022). Robustness is evaluated against A.A.
w.o. SWA
SWA
Augmentation
Accuracy (%)
Robustness (%)
Accuracy (%)
Robustness
best
end
diff.
best
end
diff.
best
end
diff.
best
end
Wide 1
baseline
78.37
78.96
-0.58
45.11
43.84
1.26
77.76
79.47
-1.71
45.71
44,69
1.02
Cutout
77.65
78.41
-0.76
45.22
44.43
0.78
76.86
79.09
-2.23
45.74
45.37
0.37
Cutmix
74.12
75.52
-1.40
45.10
44.49
77.95
78.06
-0.11
45.27
45.32
-0.05
AuA
74.59
74.93
-0.34
42.62
43.28
-0,66
75.30
75.36
-0.06
43.44
43.52
-0.08
TA
73.19
73.37
-0.18
42.06
0.14
73.41
73.53
-0.12
42.79
42.74
0.06
IDBII (ours)
79.07
79.20
-0.13
46.15
45.65
0.50
77.82
79.83
-2.01
46.70
46.26
0.44
PreAct ResNer18
baseline
82.50
83,99
-1.49
48.21
42.46
5.74
79.22
84.67
-5.45
49.18
42.93
6.25
Cutout
83.35
84,14
-0.79
49.18
47.75
1.43
81.98
84.37
-2.39
50.18
48.76
1.42
Cutmix
82.47
81.51
0.96
49.73
48.09
1.65
84,09
85.70
-1.62
50.57
47.50
3,07
AuA
83.41
84.04
-0.62
49.15
48.71
0.44
82.08
84.20
-2.12
49.53
49.59
-0.07
TA
81.68
82.26
-0.58
48.83
48.62
0.21
81.63
82.73
-1.11
49.25
49.42
-0.17
weak] (ours)
84.98
85.82
-0.84
50.34
48.94
1.40
84.18
86.45
-2.27
51.73
49.88
1.85
[DBil[strong] (ours)
83.96
84.92
-0.97
50.74
49.99
0.75
82.98
85.49
-2.51
51.49
50.77
0.72
highest best and end accuracy on these architectures, both w. and w.o. SWA, except for WRN34-1 with SWA where our method is very close to the best. Overall, our method improves both accuracy and robustness achieving a much better trade-off between them.
Data augmentation was found to be sensitive to the capacity of the underlying model. As shown in Tab. [ augmentations such as baseline, AuA and TA perform dramatically different on two ar- chitectures because they use the same configuration across the architectures. Meanwhile, augmen- tations like Cutout and ours achieve relatively better performance on both architectures but with different settings for hardness. For example, the optimal strength of Cutout is 8x8 on WRN34-1, but 20x20 on PRN18. Therefore, is vital for augmentation design to allow optimization with wide enough range of hardness in order to generalize across models with different capacity. A further discussion about generalization to alternative architectures like transformers is in Appendix
5.2 BENCHMARKING SAATE-OF-THE-TTE ROBUSTNESS WITHOUT EXTRA DATA
Tab. 2 shows the robustness of recent robust training and regularization approaches. IDBH matches the best robustness achieved by these methods on PRNI8, and outperforms them considerably in terms of best robustness on WRN34-10. This is despite IDBH not being optimised for WRN34-10. addition, our method also produces an end accuracy comparable to the best achieved by others suggesting better trade-off between accuracy and robustness. More importantly. the robustness can be further improved by combining SWA and/or AWP with IDBH. This suggests that IDBH improves adversarial robustness in way complementary to other regularization techniques. We highlight that our method when combined with both AWP and SWA achieves state-of-thh-ard robustness without extra data. We compare our method with those relying on extra data in Appendix D.3
5.3 GENERALIZATION TO OTHER DATASETS
Our augmentation generalizes well to other datasets like SVHN and TIN (Tab. B It greatly reduces the severity of robust overfitting and improves both accuracy and robustness over the baseline aug- mentation on both datasets. The robustness on SVHN has been dramatically improved by +7.12% for best and +13.231 for end. The robustness improvement on TIN is less significant than that on the other datasets because we simply use the augmentation schedule of CIFAR10 without further optimization. A detailed comparison with those regularization methods on these two datasets be found in Appendix [D.4] Please refer to Appendix F22 for generalization analysis to larger datasets like ImageNet (Deng al. 2009) and Appendix for the training and evaluation settings.
8

---

Published as a conference paper at ICLR 2023
Table 2: Performance of methods for alleviating robust overfitting for PRNI8 and WRN34-10 on CIFARIO. Robustness is evaluated against AA.
PreAct ResNer18
Wide ResNet34-10
Method
Accuracy (%)
Robustness (%)
Accuracy (%)
Robustness (%)
best
end
diff.
best
end
diff.
best
end
ditf.
hest
end
diff.
AT (Madry ul., 2018
82.50
83.99 -1.49 48.21 42.46
5.75
85, 86.63
-0.74
53.42
48.22 5.20
TRADE [Zhang cl al.
2019
81.19
82.48
-1.29
49.03
46.80
2.23
84.65
53.08
Pretraining (Hendrycks ct
2019
87.89
54.92
AWP (W 2020
83.33
84.39
-1.06
50.57
49.95
0.62
85.57
54.04
MLCAT Yu al.
2022
84.10 84.77 -0.67
50.70
50.32
0.38
54.65
54.56
0.09
CONS (Tack el al.
2022
85.25 86.45 -1.20 49.05
48.57
0.48
89.93
89.82
0.11
54.08
52.36
1.72
CChen al. 202T
84.51 85.40 -0.89 49.87
49.72
0.15
86.81
87.06
-0.25
55.50"
55.34*
0.16
TE Dong ct al. 2022)
82.35 82.79 -0.44
50.59
49.62
0.97
[DBH[strong] (ours)
83,96 84.92 -0.97 50.74
49.99
0.75
88.61
89.12
-0.52
55.83
54.01
1.82
[DBH[weak]+SW] (ours)
84.18 86.45 -2.27 51.73
49.88
1.85
89.04
89.93
-0.89
57.70
54.10
3.61
weak (ours)
82.98 83.03 -0.05
52.27
52.21
0.06
88.47
88.94 -0.47
57.88
57.68
0.20
weak] (ours)
83.42 83.45 -0.03 52.46
52.52
-0.06
89.00 89.08 -0.08
58.16
58.13
0.04
The source of the results in this table are described Appendix C Results marked are evaluated against PGD20 the AA robustness expected he lower.
Table 3: Performance our method for PRNI8 Table 4: Performance of variants of IDBH for on SVHN and TIN. Robustness is evaluated by AA (AutoPGD) for SVHN (TIN).
PRNI8 on CIFARIO. Robustness is evaluated by AA.
Data
Augmentation
Accuracy (%)
Robustness (5)
SWA
Variant
Accuracy (%)
Robustness (%)
best
end
best
end
best
end
best
end
bascline
90.55
90.18
47.48
40,86
padcrop cropshift
83.74
84.82
50.15
49.25
SVHN
IDBH (ours)
93.70
93.92
54.56
54,09
w.O.
83,96
84.92
50.74
49.99
baseline
46.94
46.60
20.19
13.82
padcrop
83.11
85.21
51.14
49.40
TIN
IDBH (ours)
50.91
51.21
21.29
19.22
cropshift
84.18
86.45
51.73
49.88
5.4 ABLATION TEST
We find that Cropshift outperforms Padcrop in our augmentation framework. To compare them, replaced Cropshift with Padcrop in IDBH and kept the remaining layers unchanged. The strength of Padcrop was then optimized for the best robustness separately for w. and w.o. SWA. As shown in Tab. 4 changing Cropshift to Pademop in our augmentation observably degrades both accuracy and robustness both and w.o. SWA.
CONCLUSION
This work has investigated data augmentation a solution to robust overfitting We found that improving robust generalization for adversarial training requires data augmentation to be as diverse possible while having appropriate hardness for the task and network architecture. The optimal hardness of data augmentation is very sensitive to the capacity of the model. To mitigate robust overfitting, we propose new image transformation Cropshift and new data augmentation scheme IDBH incorporating Cropshift. Cropshift significantly boosts the diversity and improves both accu- racy and robustness compared to the conventional crop transformation. IDBH improves the diver- sity and allows the hardness to be better balanced compared to alternative augmentation methods. Empirically, IDBH achieves the state-of-the-art accuracy and robustness for data augmentations in adversarial training. This proves that, contrary to previous findings, data augmentation alone can significantly improve robustness and beat the robustness achieved with baseline augmentation and carly-stopping. The limit of our work is that we did not have sufficient computational resources to perform more advanced, and more expensive, automatic augmentation search like AutoAuument, which implies that the final augmentation schedule have described may be suboptimal. Never- theless, the proposed augmentation method still significantly improves both accuracy and robustness compared to the previous best practice. We discuss the potential plans improve in Appendix

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
The authors acknowledge the use of the research computing facility at King's College London, King's Computational Research, Engineering and Technology Environment (CREATE), and the Joint Academic Data science Endeavour (JADE) facility. This research was funded by the King's China Scholarship Council (K-CSC).
Reproducibility Statement. Our methods including both Cropshift and IDBH can be easily im-
plemented using the popular machine learning development frameworks like PyTorch (Paszke et al. 2019). These two algorithms Cropshift and IDBH are illustrated in pseudo codes in Algorithm [] andd Algorithm Srorithm@rsspectivvvee
The procedure of optimizing IDBH is described in detail in Appendix E The full parameters of the optimal augmentation schedules we found are described in Tab. [2 and
Tab. 11 The training and evaluation settings are described in Section and Appendix [C. To further facilitate the reproducibility, we are going to share our code and the pre-trained models with the reviewers and area chairs once the discussion forum is open, and will publish them alongside the paper accepted.
REFERENCES
Jean-Baptiste Alayrac, Jonathan Uesato, Po-Sen Huang, Alhussein Fawzi, Robert Stanforth, and Pushmeet Kohli. Are Labels Required for Improving Adversarial Robustness? In 33rd Conference on Neural Information Processing Systems (NeurIPS 2019). pp. 10, 2019.
Maksym Andriushchenko and Nicolas Flammarion. Understanding and Improving Fast Adversarial Training. In Advances in Neural Information Processing Systems, pp. 12, 2020.
Yair Carmon, Aditi Raghunathan, Ludwig Schmidt, John C Duchi, and Percy Liang. Unlabeled Data Improves Adversarial Robustness. In Advances in Neural Information Processing Systems, pp. 12, 2019.
Lin Chen, Yifei Min, Mingrui Zhang, and Amin Karbasi. More Data Can Expand The General- ization Gap Between Adversarially Robust and Standard Models. In Proceedings of the 37th International Conference on Machine Learning, pp. 1670-1680. PMLR, November 2020. URL
www.m.rpeee. www..prressiill. wttppp/eeew././eceeeeo...ggo.oe html
ISSN: 2640-3498.
Tianlong Chen, Zhenyu Zhang. Sijia Liu, Shiyu Chang, and Zhangyang Wang. Robust Overfitting may be mitigated by properly learned smoothening. In International Conference on Learning Representations, 2021. URL https: httpa//////peeeeee.eeeeeeee net wttttt/frrumiiiiieeee q2zy5ur2w>
/
Francesco Croce and Matthias Hein. Reliable Evaluation of Adversarial Robustness with an Ensem- ble of Diverse Parameter-free Attacks. In Proceedings of the 37th International Conference on Machine Learning, pp. 11, 2020.
Ekin D. Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V. Le. AutoAugment: Learning Augmentation Strategies From Data. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 113-123, Long Beach, CA, USA, June 2019. IEEE. htttps://ieeexplorss
ISBN 978-1-72813-200
doi: N0.119//VVR..201.0000 Seee..org//ccmmmn////333311
Ekin D. Cubuk, Barret Zoph, Jonathon Shlens, and Quoc Le. Randaugment: Practical automated data augmentation with a reduced search space. In Advances in Neural Information Processing Systems, 2020.
Deng, W. Dong, R. Socher, L. Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In 2009 IEEE Conference Computer Vision and Pattern Recognition, pp- 248-255 255, June 2009. doi: 00.1109/CVPR.2200552222...
ISSN: 1063-6919.
Terrance Vries and Graham W. Taylor. Improved Regularization of Convolutional Neural Net- works with Cutout. Ttl::1708.0555 [cs], November 2017. URL http:://axx: http://.xxx.oo/////////// abs/ 2os/1108 04552 arXiv: 778.04552.
10

---

Published as a conference paper at ICLR 2023
Yinpeng Dong, Ke Xu, Xiao Yang, Tianyu Pang, Zhijie Deng, Hang Su, and Jun Zhu. Exploring Memorization in Adversarial Training. In International Conference on Learning Representations, 2022. URL https: http://wwww..........oo.ooaoaaaa
Raphael Lontijo-Lopes, Sylvia Smullin, Ekin Dogus Cubuk, and Ethan Dyer. Tradeoff's in Data
Augmentation: An Empirical Study. In International Conference Learning Representations,
2021. URL httpl::t/////..cc. http::::///peeeevieeee net ftre/foruee forum?id ZcKPWahG6wy)
Sven Gowal, Chongli Qin, Jonathan Uesato, Timothy Mann, and Pushmeet Kohli. Un- covering Limits of Adversarial Training against Norm-Bounded Adversarial Examples. arXiv:2010.083 201..00000 [cs, stat], March 2021a. (RLttttt:///ax: orggabs/////aa//// 035933
arXiv: 2010.03593.
Sven Gowal, Sylvestre-Alvise Rebuffi, Olivia Wiles, Florian Stimberg, Dan Calian, and Timothy Mann. Improving Robustness using Generated Data. In Thirty-Fifih Conference on Neural Infor- mation Processing Systems, pp. 16, 2021b.
Kai Han, Yunhe Wang, Hanting Chen, Xinghao Chen, Jianyuan Guo, Zhenhua Liu, Yehui Tang, An Xiao, Chunjing Xu, Yixing Xu, Zhaohui Yang, Yiman Zhang, and Dacheng Tao. A Survey on Vision Transformer. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45 (1):87-110, January 2023. ISSN 1939-3539, doi: 10.1109/TPAML2222.3113333
Conference
Name: IEEE Transactions on Pattern Analysis and Machine Intelligence.
Ryuichiro Hataya, Jan Zdenek, Kazuki Yoshizoe, and Hideki Nakayama. Faster AutoAugment: Learning Augmentation Strategies Using Backpropagation. Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (eds.), Computer Vision ECCV 2020, Lecture Notes in Computer Science, pp. 1-16, Cham, 2020. Springer International Publishing. ISBN 978-3-030- 58595-2. doi: 10. 3-1090888--000--98-----8
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In 2016 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 770-778, Las Vegas, NV. USA, June 2016a. IEEE. ISBN 978-1-4673-8881-11 doi: 10.1109/ CVPR.2016.90. URL
leee http://wwpp...........oooooooeee
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity Mappings in Deep Residual Networks. In Proceedings of the European Conference on Computer Vision (ECCV), 2016b.
URL
org/ htcp:/bbbs1111... 05027
Dan Hendrycks, Kimin Lee, and Mantas Mazeika. Using Pre-Training Can Improve Model Robust- ness and Uncertainty. In Proceedings of the 36th International Conference on Machine Learn- ing. pp. 2712-2721. PMLR, May 2019. wtRLhttss::/ hattps::/proceeciingg wwt.mm............ ISSN: 2640-3498.
hendrycksi9a.o html
Hoki Kim. Torchattacks: A PyTorch Repository for Adversarial Attacks, February 2021. URI.
http://arxiv. abs////0b///00 21122011110000 01950 arXiv:2010.01950 [cs].
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N. 7(7):3, 2015.
Lin Li and Michael Spratling. Understanding and Combating Robust Overfitting via Input Loss Landscape Analysis and Regularization. submitted, pp. 29, August 2022.
Aoming Liu, Zehao Huang, Zhiwu Huang, and Naiyan Wang. Direct Differentiable Aug- mentation Search. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 12219-12228, 2021. URL
thecvf. com/ ICCV2021/Has 1t.002//hhmm..m /Liu Def.Drrrret...feeeeetaabaeee htw.olleeeAmmeeetttoooo_sso.ee ICCV_ 202122221222 _aapll_pppe____ [tmpl
wtw.cc.cco//cooeettt. content/
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards Deep Learning Models Resistant to Adversarial Attacks. In International Conference on Learning Representations, 2018. URL http:///wwww...o////////////////
11

---

Published as a conference paper at ICLR 2023
Yifei Min, Lin Chen, and Amin Karbasi. The curious case of adversarially robust models: More data can help, double descend, or hurt generalization. In Proceedings of the Thirty-Seveni Conference
Uncertainty in Artificial Intelligence. pp. 129-139. PMLR, December 2021. URL https:
//proceedi ngs. ml prwss / 61 /minnmmall html
ISSN: 2640-3498.
Yichuan Mo, Dongxian Wu, Yifei Wang. Yiwen Guo, and Yisen Wang. When Adversarial Training
Meets Vision Transformers: Recipes from Training to Architecture. October 2022. URL https:
//openreview.. www. wwweteoooommeeeeeeeeeee dVV2V/ QQJ
Samuel G. Müler and Frank Hutter. TrivialAugment: Tuning-Free Yet State-of-the-Art Data Augmentation. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 774-782, 2021. httpttttppp///poaapptpttttttttte
wwweteeeeeeeee wwwccvffccomcoooo www.o///ooe//////ccccccccccc IICC//021/ html Mww.aeerrriiiaalaaaa Tawplaigmmeeettttnnnenaa Tuning Free Yet_State-of the Art__
Data Awwmentatiinnncccc..c.oo... pat_ipppperhhtmmm
Yuval Netzer, Tao Wang. Adam Coates, Alessandro Bíssacco, Bo Wu, and Andrew Y. Ng. Read- ing Digits in Natural Images with Unsupervised Feature Learning. NIPS Workshop on Deep Learning and Unsupervised Feature Learning 2011, 2011. edd/
hwwsenumbers...ooomo pdf
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Ed- ward Yang. Zachary Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An Imperative Style. High- Performance Deep Learning Library. In Advances in Neural Information Processing Systems, vol- ume 32, pp. 9926-8037 2019. httpptts::///aa Abstractttttttaaae.......aaace.. .tml
Sylvestre-Alvise Rebuffi, Sven Gowal, Dan Andrei Calian, Florian Stimberg, Olivia Wiles, and Timothy Mann. Data Augmentation Can Improve Robustness. In Neural Information Processing Systems, May 2021. URL https: htttps://opprreeeeeeet /pwnreveeeeeeeeeee
net. www.://o/fformmttttm/mmmmm hit.srrnmmmn..kkkkkksnsshsnnn
Leslie Rice, Eric Wong, and J Zico Kolter. Overfitting in adversarially robust deep learning. In Proceedings of the 37th International Conference on Machine Learning, pp. 12, 2020.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei- Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer doi: 10.1007/s1 10007//11233-005-088666 URL /tRL/ttts:://oo..rrg0 http:/110100001111111111 015 0816-y
Vision, 115(3):211-252, December 2015. ISSN 1573-1405
Ludwig Schmidt, Shibani Santurkar, Dimitris Tsipras, Kunal Talwar, and Aleksander Madry. Adver- sarially Robust Generalization Requires More Data. In Advances in Neural Information Process- ing Systems, volume 31. Curran Associates, Inc., 2018. URL htttttt:/pappprr// nips. cc/ baper/2018/hashr /ttp:///eeeeEEEEEEEEEEEaaaaaeee2 oara Abstract. .html
Vikash Sehwag, Saeed Mahloujifar, Tinashe Handina, Sihui Dai, Chong Xiang, Mung Chiang, and Prateek Mittal. Robust Learning Meets Generative Models: Can Proxy Distributions Improve Adversarial Robustness? March 2022. URL https: http:///pppreviiieee net forum?ick
WVXONNWBBKV
Ali Shafahi, Mahyar Najibi, Amin Ghiasi, Zheng Xu, John Dickerson, Christoph Studer, Larry S. Davis, Gavin Taylor, and Tom Goldstein. Adversarial Training for Free! In Advances in Neural Information Processing Systems, 2019. URL.Atttp///axx...rrr//bbbb/
12843]
Rulin Shao, Zhouxing Shi, Jinfeng Yi, Yu Chen, and Cho-Jui Hsieh. On the Adversarial Robust- ness of Vision Transformers. Transactions on Machine Learning Research, January 2023. ISSN 2835-8856. URL
Vasu Singla, Sahil Singla, Soheil Feizi, and David Jacobs. Low Curvature Activations Reduce Overfitting in Adversarial Training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.ionnppp.iiie 16423-16433, 2021.
www.eeesssstt....eeeeeeeeeeee thecvf.com..
Sorteent/Lccv222//ttmnii Lttp:///ttnnnnnnaaaaaaaaaaaaaaa www_wuureauureeeeooeooaaeaaeeeee Everfittting_in_Aveeee htt.aarriiaa__rraineee_eeeeeoeeo
http:///D.....oooooooooooooooonn Cttp:ttuuurecitttttiiiottot Reduce
12

---

Published as a conference paper at ICLR 2023
Jihoon Tack, Sihyun Yu, Jongheon Jeong, Minseon Kim, Sung Ju Hwang, and Jinwoo Shin. Con-
sistency Regularization for Adversarial Robustness. Proceedings of the AAAI Conference on
Artificial Intelligence, 36(8):8414 884444442222 June 2022. ISSN 2374-3468, 2159-5399. doi: 10. 1609/aai..v66in
URL. ht /tt.://///////o.......oooooooaoa
Flew/20811
Haotao Wang. Chaowei Xiao, Jean Kossaifi, Zhiding Yu, Anima Anandkumar, and Zhangyang Wang. AugMax: Adversarial Composition of Random Augmentations for Robust Training. May
2021. URL B021..tRRtttteeee.. htttps://pperrvvieee wwwp::/tttteetteetteeet/ P
Eric Wong, Leslie Rice, and J. Zico Kolter. Fast is better than free: Revisiting adversarial training.
In International Conference Learning Representations, 2020. URL http::/azxxx org/
2cs/2201.03994
Dongxian Wu, Shu-Tao Xia, and Yisen Wang. Adversarial Weight Perturbation Helps Robust Generalization. In Advances in Neural Information Processing Systems, vol- ume 33, pp. 2958 2969, 2020. URL http:: 1ef9cc21233ee145eee 2et.2ee000ee02222522522222222222 Abstracct. html
Chaojian Yu, Bo Han, Shen, Jun Yu, Chen Gong, Mingming Gong, and Tongliang Liu. Un- derstanding Robust Overfitting of Adversarial Training and Beyond. In Proceedings of the 39th International Conference on Machine Learning, pp. 25595-25610.. PMLR, June 2022. URL
mlr prwssprsasaae..... www2//yuuu w2tpp2/y222bbp html ISSN: 2640-3498.
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon
Yoo. CutMix: Regularization Strategy to Train Strong Classifiers With Localizable Features. In
Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6023-6032, wwtp:/ew.ccooccoooooooooac.eeeee content ICCV 201 ICCV__C1//VV__m1111111 Cwwsssfiers.......
2019. URL beg
Yur. CutMix Regularization wtwp:oonnnttttttttttttotee_ee Trainnn__rii_....
With Localizable, htt.iiiiabbeeaaaeeeeeeeeeee ICCV_ (CCCc__01______eee
html
Sergey Zagoruyko and Nikos Komodakis. Wide Residual Networks. In Procedings of the British
Machine Vision Conference 2016, pp. 87.1-87.12 York, UK, 2016. British Machine Vision As- sociation. ISBN 978-1-901725-599 doi: 10.5244/C.30.87. URL http::/wwee bmva org/
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling Vision Transform-
ers. pp. 12104-12113, 2022. URL [stpss:/ thecvf com/ content/
SVPR2022/htmi. /
Zhai Scaling Vision Transformers CVPR 2022 paper.
html
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. Theoretically Principled Trade-off between Robustness and Accuracy. In International Confer- ence on Machine Learning, pp. 7472-7482, PMLR, May 2019. URUhttt::/pproeedaii ISSN: 2640-3498.
mlr. wtt.ppresss/////// wttpppres//////hagg11ggg999 9p...p9pppppmm html
Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond Em-
pirical Risk Minimization. February 2018. URL
r1Dopl- Perappl-bb;::: moteee-rldpipppp-
Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yang. Random Erasing Data Aug- mentation. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pp- 13001-13008, April 2020. doi: 10. 11.69//aai.v34iiii 7000. URL httpss:/ojss.aat .org/
index.
Number: 07.
13

---

Published as a conference paper at ICLR 2023
60
55
AA (AuA)
Robustness
50
PGD (AuA)
45
AA (highest on AuA)
40
PGD (highest on AuA)
35
AA (baseline)
30
PGD (baseline)
25
50
100
150
200
Epoch
(a) PreAct ResNet18
55
50
AA (AuA)
Robustness
45
PGD (AuA)
AA (highest on AuA)
40
PGD (highest on AuA)
35
AA (baseline)
30
PGD (baseline)
25
50
100
150
200
Epoch
(h) Wide ResNet34-1
Figure 5: The test robustness measured using PGD (blue lines) and AA (red lines) every 10 epochs during training. Results are shown, with solid lines, for PreAct ResNet18 (top) and Wide ResNet34- (bottom) trained with AutoAugment (AuA). The dashed lines represent the best robustness of the same models trained with the baseline data augmentation method, Flip-Padcrop, rather than AuA.
A RECONCILING THE CONTRADICTORY RESULTS FOR AUTOAUGMENT
Experimental Setting. AutoAugment was implemented as in the original paper (Cubuk et al., 2019] Horizontal Flip half chance, Paderop with pixels padding, Color/shape layer searched for CIFARIO, 10, 16x 16 Cutout. We adopted the implementation of the color/shape layer from PyTorch Paszke et al. 2019). The model architectures were Wide ResNet34-1 and PreAct ResNet18, We trained the models using the same settings as Tack et al. (2022), which is actually our default training setting Section Robustness was evaluated against both Projected Gradient Descent (PGD)
(Madry et al. 2018 with 10 steps and AutoAttack (AA) (Croce & Hein 2020).
We observe, similar to Tack et al. (2022), that the PGD robustness AuA-trained models increases the baseline PGD robustness the later stage of training for both network architectures (see Fig. 5 In contrast, the AA robustness is similar to (Fig.5a), or is worse than (Fig.5b).. the baseline's ss AA has been widely recognized robustness reliably (Croce & Hein 2020). Therefore, we conclude that AuA fails to significantly boost the end robustness over the baseline, and the impressive improvement regarding PGD robust- ness is misleading. The misalignment between PGD robustmess and AA robustness is more explicit in Fig. [b] where the "best" checkpoint detected by PGD robustness different from that of AA robustness. That's 's why the severity of robust overfitting is observed to be negative in Section 3.1 and Section 5.1
more advanced attack that better able estimate adversarial
Our results, and those of Tack et al. (2022), for Automuument on PreAct ResNet18 are inconsistent with previous work that found that the end robustness (Carmon et al. 2019), and even the best ro- bustness (Gowal et al.] 2021a), of AutoAAgment is lower than the best robustness of the baseline augmentation. We suspect that these contradictory results may be partially due to inconsistent use of the term AutoAugment. Tack et al. (2022) explicitly state their AutoAugment includes Hori- zontal Flip, Padcrop and Cutout as in the original work (Cubuk et al. 2019). contrast, Carmon al. 2019); Gowal al. (2021a) seem refer AutoAugment only the color/shape layer and
14

---

Published as a conference paper at ICLR 2023
the remaining three component augmentations may may not have been included. Specifically, Carmon et al. (2019) state AutoAugment is used in addition to Paderop and Horizontal Flip, and Gowal et al. (202 state AutoAugment is used with the original setting, but specific implemen- tation information is given. Another possible account is that they used different training settings, and particularly different capacity model, which has important impact on the performance of data augmentations in adversarial training as shown in Secs. 3.1 and 5.1
B DIFFERENCES BETWEEN DATA AUGMENTATION IN STANDARD AND ADVERSARIAL TRAINING
The idea of jointly handling hardness and diversity has been studied before in standard training (GontijoLoppe al. 2021 Wang et al. 202T). However, the impact of hardness and diversity of data augmentation on adversarial training has never been researched before. This topic is particularly important because many previous attempts (discussed in Section I to solve robust overfitting using data augmentation methods from standard training failed and the cause of this failure is unclear. Our analysis provides some insight on why directly transferring data augmentation methods from standard training to ad adversarial training does not work.
More importantly, we find that the impact of hardness on adversarial training is very different from that on standard training. Gontijo-Lops et al. (2021) and Wang et al. (2021) both claim that increas- ing both diversity and hardness will produce better data augmentation, which is in contrast to our finding that too hard data augmentation hurts both accuracy and robustmess (we all share the same conclusion on diversity). This suggests that these two training paradigms, standard training and adversarial training, may require fundamentally different data augmentations. Therefore, we pro- pose IDBH to maximize the diversity and balance the hardness according to the underlying model architecture, whereas (GontijoLoppes et al. (2021): Wang et al. (2021) propose to maximize them both.
C EXPERIMENTAL SETTINGS
C.I TRAINING SETUP
The experiments in this section were based on the following settings unless otherwise specified. The model's architecture was Wide ResNet34-1 (widening factor of 1) (Zagoruyko & Komodakis 2016). The dataset was CIFARIO (Krizhevsky) (009). Data augmentation, if specified, applied and half the time it was not applied. Models were trained by stochastic gradient descent for 200 epochs with initial learning rate 0. divided by 10 at the epochs 100 and 150. The momentum was 0.9, the weight decay was Se-4 and the batch size was 128. CrossEntropy loss was used. For both adversarial training and evaluation, we used the same attack, lox projected gradient descent (Madry et al.. 2018). with perturbation budget, c, of 8/255 and step size of 2/255. The number of steps was 10 and 50 for training and evaluation respectively. Result were averaged over runs. Experiments were run on Tesla V100 and A100 GPUs.
Section 3
was applied with 50% chance, i.e., half the time augmentation
The experimental settings were identical to those used in Section below. The training method was PGD10 adversarial training (Madry al. 2018 SWA was imple- mented as in Rebuffi et al. 202T). Robustness was evaluated against AA using the implementation of Kim (2021). In contrast to Section 3 data augmentation, used, was always applied. We ad- ditionally evaluated on the datasets SVHN (Netzer et al. (011) and Tiny ImageNet (TIN) (Le & Yang, 2015). As in previous work Yu al. (2022), the baseline augmentation for SVHN was no data augmentation. The baseline data augmentation for TIN was the same as used for CIFAR10 i.c., Horizontal Flip (applied at half chance) and Padcrop with 4 pixel padding. Adversarial training was applied when training with TIN in exactly the same way it as when training CIFAR10. The reported results are averages over three runs, and the standard deviation is reported in Appendix
Section 5
3 unless specified
To train SVHN, the initial learning rate was 0.01, the step size 1/255 and the perturbation budget, e, was increased from 0 to 8/255 linearly in the first five epochs and then kept constants for the remaining epochs in order to stabilize the training, suggested by Andriushchenko & Flam-
15

---

Published as a conference paper at ICLR 2023
Table The strength of individual transformations corresponding the 7 degrees hardness.
Hardness
3
5
7
Robustness(
45
40
35
25
20
15
Transform
Strength
ShearX
0.1
0.24
0.33
0.4
0,45
0.55
0.7
ShearY
0.1
0.2
0.3
0.37
0.45
0.55
0.7
Translate
1
3
4
7
11
Translate Y
1
4
7
10
Rotate
3
7
12
15
19
23
31
Contrast
0.92
0.82
0.73
0.67
0.62
0.56
0.5
Brightness
0.92
0.82
0.75
0.7
0.65
0.6
0.56
Color
0.7
0.3
0.1
Sharpness
0.7
0.3
0.01
Solarize
253
240
224
210
195
185
172
Cropshift
1
7
12
15
18
Cutout-i
3
9
11
13
15
18
marion 2020),
otherwise, the same set-up was used when training on CIFARIO. For adversarial evaluation, robustness on TIN was evaluated against AutoPGD (Croce & Hein 2020) with 50 itera- tions and restarts since did not have access to sufficient computational resources to run AA for TIN on PreAct ResNet18.
We re-optimized the strength of Cutout and Cutmix per model architecture. AuA was parameterized as in [Cuuuk et al. (2019) since we didn't have sufficient resource to optimize. TA is faremeter-free so no tuning was needed. The size of cut-out area for Cutout was searched for within the range of [4x4, 6x6, 8x8, .... 28x28J. The optimal size we found was 8x8 when useing Wide ResNet34-1 and 20x20 when using PreAct ResNet18. Following the procedure used in Yun et al. (2019), for Cutmix the value of the hyper-parameter The optimal a we found was 0.1 on Wide ResNet34-1 and 0.25 on PreAct ResNet18. Cutout and Cutmix were applied with the default (baseline) augmentations in the order of Wlip-Paderop-Cutoon and -Cutmix respectively. Similarly, for AuA (and TA) augmentions were applied in the order of Flip-Padcrop-AuA (TA)-Cutout in Cubuk al. (2019) Muüller Hutter (222
searched for over the range {0.1, 0.25, 0.5, 1.0, 2.0, 4.0).
The source of the result being compared in Tab. 2 is as follows. For PreAct ResNet18, the result of AT, AWP and KD Wide ResNet34-10, 10, the result of AT was determined by us. The result of TRADE, Pre-training and AWP is from Wu al. (2020). Otherwise, the results are copied directly from the original work. All methods use the same training setting, except KD and Pre-training.
determined by us. The result of TRADE is from Dong al. (2022). For
C.2 STRENGTH HARDNESS INDIVIDUAL TRANSPORMATIONS
The 7 degrees of hardness used were: (1.04, 1,17, 1.56, 1.87, 2.34, This cor- responds the denominator in Eq. [I] i.c.. Robustness D Dlest). having values of (45, 40, 35, 30, 25, 20, 15 as the PGD50 robustness of the model on the original test data (no data augmentation applied), i.e., Robustness M, Dtest), was 46.93%. The search range was constrained to be between 0 and for the transformations Color, Sharpness, Brightness and Contrast. The corre- sponding strength of each individual transformation is described in Tab. 5 Note that the correspon- dence between strength and hardness is approximate because the real Robustness(MM D(est) is not exactly equal to the nominal value given above, e.g., the Robustness (M, D(est) of ShearX with strength 0.1 is only close instead of strictly equal 45%. Nevertheless, the variation between the real and the nominal Robustness M, Diast) for all transformations small so this should not effect analysis.
16

---

Published as a conference paper at ICLR 2023
None Rotate
ShearX
TranslateX
Cropshift-1
Contrast
Brightness
Color
ShearY
TranslateY
Cutout-i-1
Solarize
Sharpness
43
79
11
Gap of Robusiness
Und Robustness
41
End Accuracy
77
7-
75
5-
35
71
0
2
5
8
0
3
4
6
7
1
2
3
5
6
7
Hardness (a)
Hardness
Hardness
(b)
(c)
43
11
Gap of Robustness
End Robustness
41
End Accuracy
77
39
75
37
35
0
3 4
7
8
0
2
3 4 Hardness
7
8
0
1
2
5
6
7
Hardness
Hardness
(d)
(e)
Figure 6: The comparison of the exceptional transformations Color, TranslateX and Translate (top row) and the others (bottom row) regarding the effect of hardness on the performance of adversarial training. The results the bottom also appear in Fig. and re-produced here to aid compar- ison. Robustness is evaluated against PGD50.
D ADDITIONAL EXPERIMENTAL RESULTS
D.1 EXCEPTIONAL. TRANSFORMATION HARDNESS EXPERIMENTS
The figures of Translate and Y) and Color present exceptional patterns, as shown in Fig. 6 First, robust overfitting and end robustness jumps abruptly, at hardness 3, to the same level of performance as training without augmentation (the gray dashed line). This suggests that training with any of them at this particular hardness does not provide benefit, and can even impair, the robust generalization and end robustness. This is in contrast to the results produced with other values hardness for these specific transformations, these augmentation don't make the training data harder to although they, recalling how we mea- vulnerable adversarial attacks. For Color, this
well as the results for other augmentations, Hence, at some strengths
sure the hardness, do make the test data may be due to reduction diversity as at hardness (strength 0.1) this augmentation transforms colorful images into gray images (Fig. 7.
Second, ignoring the behavior at hardness 3 discussed above, for Translate, changing hardness does produce the clear trends in the three metrics that are with increasing hardness for the other transformations. This suggests that the complexity of data augmented by Translate doesn't t increase consistently, at least within the evaluated range. with the increase strength. possible explanation for this is that the foreground object lies in the center of the image in most CIFARIO samples and, therefore, translating the image will typically only remove background pixels, and introduce black block at one border. A black block at the border is rare in the natural images, so adding this pattern to the data should increase the complexity. However, increasing the strength of Translate may have little further impact on the data complexity, because it only increases the size of the black regions while remove a few additional informative pixels from the opposite edge of the image. This contrasts with the other investigated transformations, like Shear, in which increased strength leads to increased distortion or, like Cropshift, which introduce more new patterns to the data (adding black blocks at borders), with increasing strength. Hence, applying (versus
17

---

Published as a conference paper at ICLR 2023
Figure 7: Illustration of the effect of Color transformation with hardness (strength 0. on selected CIFARIO 10 images. The original images in the top row, while the augmented images are in the bottom row.
None Rotate
ShearX ShearY
Cropshift-1 Cutout-i-1
Contrast Solarize
Brightness Sharpness
78
79
Best Accuracy
76
End Accuracy
74
75
72
73
70
71
0
2
4 Hardness (a)
3
6
7
8
0
5
6
Hardness (b)
Figure 8: The effect hardness of data augmentation on (a) best accuracy, and (b) end accuracy. The results for end accuracy also appear in Fig. 2b and re-produced here to aid comparison.
applying) Translate effects the three metrics observably, but increasing the strength of Translate has little effect. Overall, these exceptions reflect defect in measure of hardness, Eq. (T). and we leave the work of improving it to the future.
D.2 FIGURES OF BEST ACCURACY FOR HARDNESS EXPERIMENTS
Fig. Sa shows the best accuracy with respect to the hardness for transformations excluding Trans- late Color. It shows more obvious downward trend for accuracy with increasing hardness, compared to the equivalent results for end accuracy Fig. 8b
D.3 COMPARISON WITH METHODS USING EXTRA DATA
Although our method fails to beat the robustness achieved by RST and PORT, it closes the gap between the performance approaches that do not use additional training data with those that do use such additional data, shown in Tab. b Our method doesn't require any additional data, whereas RST and PORT relies on a tremendous amount (0.5 and 10 millions) unlabeled and synthetic data respectively. The acquisition of this volume of extra data is very expensive (Sehwag ett al. 2022) and may be infeasible some particular domains. Moreover, both RST and PORT mix the original and extra data in equal proportions in each mini-batch that their actual batch size,
18

---

Published as a conference paper at ICLR 2023
Table Performance the methods for improving adversarial training with and without extra data for Wide ResNet34-10 10 on 10. Robustness is evaluated against AA. Accuracy and Robustness are evaluated on the "best" checkpoint. The result of RST is copied from Wu al. (2020)
Method
Extra Data (Million)
Accuracy (%)
Robustness (%)
AT (Madry al. 2018)
85.90
53.42
RST Carmon et al. 2019
0.5
89.69
59.53
PORT (Sehwag et al.
2022
10
87.00
60.60
IDBH
strong] (ours)
88.61
55,83
IDBH
weak +SWA (ours)
89.04
57.70
[DBH[weak]++ir (ours)
88.47
57.88
[DBH|svea]]AWWP+SWW (ours)
89.00
58.16
Table 7: Performance of the methods for alleviating robust overfitting for PreAct ResNet18 on SVHN and TIN. Robustness evaluated against AA for SVHN and and PGD20 for TIN. The results for the regularization methods are copied from the original works. Note that the original training set-up for TE (on SVHN) and KD (on TIN) slightly different from ours.
Accuracy (%)
Robustness (%)
Dataset
Augmentation
best
end
diff.
best
end
diff.
baseline
90.55
90.18
0.37
47.48
40.86
6.62
MLCAT
51.90
49.76
2.14
SVHN
TE
90.09
90.91
-0.82
51.44
50.61
0.83
93.70
93.92
-0.22
54.56
54.09
0.47
baseline
46.94
46.60
0.34
23.05
14.41
8.64
CONS
49.46
50.15
0.69
23.31
21.33
1.98
TIN
KD
50.57
51.38
-0.81
21.84
21.45
0.39
ours
50.91
51.21
-0.30
24.12
21.08
3.04
and hence computational cost, is twice that of methods. such as ours, that do not use additional data. The above drawbacks seriously limit the application of RST and PORT.
D.4 COMPARISON WITH REGULARIZATION METHODS ON SVHN AND TIN
As shown Tab. 7 IDBH achieves dramatic improvement of accuracy and robustness over those strong baselines on SVHN. Regarding TIN, IDBH, despite not being optimized for this particular dataset, achieves higher accuracy and robustness compared to KD and CONS.
D.5 STANDARD DEVIATION DATA
Tabs. and provide the standard deviation data for the experimental result reported in Tab, [ Tab. and Tab. 3 respectively. Overall, the standard deviation of the robustness, both best and end, no greater than 0.7.
E AUGMENTATION SCHEDULE AND OPTIMIZATION
Tab. T0 shows the reduced search space used to optimize hyperparametens for our proposed IDBH augmentation method (see Algorithm 2 The flip layer was fixed to Horizontal Flip half chance following convention. For the crop layer, searched over 8 combinations. where each had different strength range (defined by a changed upper bound) and probability of applying the trans- formation. For Random Erasing, and two probabilities of application, 0.5 and 1.0. In this case strength denotes the proportion of the image erased. The aspect ratio of the erased area was always uniformly sampled between 0.3 and
considered two strength ranges, (0.02, 0.33) and (0.02, 0.5),
19

---

Published as a conference paper at ICLR 2023
Table 8: Standard deviation the experimental results in Tab.
and Tab.
W.O. SWA
w. SWA
Augmentation
Accuracy (%)
Robustness (%)
Accuracy (%)
Robustness (%)
best
end
best
end
best
end
best
end
Wide ResNet
34-1
baseline
0.51
0.55
0.15
0.55
0.28
0,09
0.12
0.17
Cutout
0.18
0.43
0.11
0.31
0.11
0.15
0.09
0.15
Cutmix
1.57
1.28
0.29
0.32
0.15
0.08
0.11
0.20
AuA
1.35
0.52
0.28
0.12
0.41
0.30
0.49
0.36
TA
0.33
0.64
0.54
0.28
0.25
0.29
0.11
0.23
IDBH (ours)
0.35
0.39
0.06
0.12
0.22
0.18
0.10
0.04
PreAct ResNet 18
baseline
0.31
0.53
0.57
0.23
0.22
0.13
0.11
0.16
Cutout
0.17
0.40
0.11
0.54
0.13
0.11
0.09
0.12
Cutmix
0.46
0.36
0.33
1.10
0.39
0.09
0.21
0.03
AuA
0.43
0.51
0.38
0.22
1,43
0.06
0.24
0.16
TA
0.31
0.20
0.32
0.40
1.83
0.16
0.39
0.26
IDBH[weak] (ours)
0,21
0.12
0.15
0.15
0.35
0.08
0.16
0.24
[DBH[strong] (ours)
0.42
0.04
0.06
0.25
0.13
0.23
0.25
0.22
IDBH[weak]+ (ours)
0.34
0.48
0.26
0.28
0.11
0.07
0.06
0.08
Wide ResNet34-10
baseline
0.57
0.04
0.59
0.17
0.16
0.04
0.01
0.34
IDBH[weak] (ours)
1.58
0.33
0.70
0.50
0.08
0.16
0.11
0.18
IDBH[strong] (ours)
0.32
0.27
0.22
0.27
0.12
0.14
0.13
0.11
IDBH[wcak)+AWD (ours)
0.45
0.27
0.23
0.25
0.14
0.11
0.30
0.20
Table 9: Standard deviation the experimental results in Tab.
Accuracy (%)
Robustness (%
Dataset
Augmentation
best
end
best
end
baseline
0.60
0.10
0.59
0.28
SVHN
0.12
0.14
0.29
0.07
baseline
0.17
0.69
0.18
0.14
TIN
ours
0.38
0.03
0.17
0.08
20

---

Published as a conference paper at ICLR 2023
Table 10: The reduced search space used for each component IDBH. Strength "1-X" means to uniformly sample value between 1 and X. inclusive. Note that the strength of Cropshift is discrete so "1-5" {I, 2, 3, 4, 5}. The strength of other component augmentations is continuous unless specified otherwise. "prob." denotes the probability of applying the transformation with the specified strength. For color/shape the strength defined by a set of individual transformations as specified in Tab.
Horizontal Flip
Cropshift
Color/shape
Random Erasing
no. strength prob. no. strength
prob.
no.
strength
prob.
no.
strength
prob.
0.5
1-5
0.833 (=5/6)
ColorBiased
1
0
1-6
0.857
(=6/7)
ShapeBiased
1
2
0.02-0.33
0.5
1-7
0.875 (=7/8)
3
0.02-0.5
0.5
1-8
0.889 (=8/9)
0.02-0.33
1
1-9
0.900 (=9/10)
5
0.02-0.5
1
6
1-10
0.909 (=10/11)
1-11
0.917 (=11/12)
8
1-12
0.923 (=12/13)
3.3. Apart from these 4 combinations x2x 2), we add one where Random Erasing is never applied, i.e. where the probability is zero.
Regarding the color/shape layer, two versions were implemented: color transformations biased (ColorBiased) version, and a shape transformations biased (ShapeBiased) version, as defined in Tab. 11 These realizations reflect the prior expectation that different datasets prefer essentially different types of transformation in standard training (Cubuk et al., 2019). The strength range of partially follows (Cubuk et al. 2019, but the lower bound of Brightness/Contraa is raised to 0.5 to be exempt from extremely hard (distorted) augmentations. Based on the results of the preliminary experiments, the strength range of Shear and Rotate was se- lected to have relatively small average strength for the ColorBiased version, and modest strength in the ShapeBiased version. The color (shape) transformations in the ColorBiased (ShapeBiased) instances are assigned a greater probability of being applied. The overall strength of this layer is relatively weak so that we can have more space to fine tune the strength other layers particularly Cropshift which can boost the diversity alongside the hardness. We realize that there many more possible implementations of the color/shape layer, considering additional implementations would greatly increase the computational burden of optimising the augmentation hyperparamters. Never- theless, these two implementations reflect our intuition as to reasonable hyperparameters to search, and our results show that they adequate produce large boost in accuracy and robustness. Overall, the total search space contains 80 (8 x 2 x 5) possible augmentation schedules.
We performed the grid search over the reduced search space for each model architecture on each dataset separately. The exception was we did not optimize our augmentation on TIN and Wide ResNet34-10 due to a lack of computational resources and simply used the same parameters as for PreAct ResNet18 on CIFARIO. The optimal schedules found are described in Tab. T2 We highlight that is necessary to optimize the parameters for different architectures since our analysis (Section 3.1. . as well as our search results (Tab. [2). suggests that the optimal data augmentation is very sensitive to the model architecture. We therefore expect that the results for Wide ResNet34-10 could be further improved if we optimize IDBH for it. The search procedure can be sped-up by being deployed on multiple GPUs parallel. The efficiency can be further boosted by filtering out schedules that are likely to be worse based the feedback of already evaluated schedules. For example, if schedule appears to be overwhelmingly hard, degrading the accuracy and robustness, it is unnecessary evaluate schedules that are harder.
21

---

Published as a conference paper at ICLR 2023
Table 11: ColorBiased and ShapeBiased realizations of the color/shape layer. The strength and "prob." have the same interpretation as Tab. [0] =[0.05-0.155] means a range of 0.05-0.15 with a randomly chosen sign. The strength value of Rotate discrete. The probabilities of all component transformations in both realizations sums up to 1.
ColorBiased
ShapeBiased
Transformation
strength
prob.
strength
prob.
Color
0.1-1.9
0.125
0.1-1.9
0.08
Brightness
0.5-1.9
0.125
0,5-1.9
0.08
Contrast
0.5-1.9
0.125
0.5-1.9
0.04
Sharpness
0.1-1.9
0.125
0.1-1.9
0.08
AutoContrast
0.125
0.04
Equalize
0.125
0.08
ShearX
+[0.05-0.15]
0.0625
0.05-0.35
0.15
YhearY
=[0.05-0.15]
0.0625
0.05-0.35
0.15
Rotate
=[1-10]
0.125
1-30
0.3
Table 12: The optimal augmentation schedules found for our method on various datasets and models.
Horizontal Flip)
Cropshift
Color/shape
Random Erasing
Data
Model
Variant
strength prob.
strength prob.
strength
prob. strength
prob.
CIFAR1O Wide ResNet34-1
0.5
1-5
0.833
ColorBiased
CIFARIO PreAct ResNet18 weak
0.5
1-10
0.909
ColorBiased
0.02-0.33
0.5
CIFARIO PreAct ResNet18 strong
0.5
1-10
0.909
ColorBiased
0.02-0.33
SVHN
PreAct ResNet18
0
1-8
0.889
ShapeBiased
0.02-0.5
Algorithm 2. Pscudo-code for IDBH. Pa. is the prohability of applying the transformation x. Sy is the strength range of the transformation y. rand() uniformly samples a floating floating-point point number between 0 (inclusive) and 1 (exclusive).
Function IDBH (img):
if rand() < Prtp then
img- horizontal flip(ing)
end
rand() Perop then
uniform. uniform-sampleeSS.oo. img cropshift(img. s)
end
rand() Pootor/ then /+ taking an example of ColorBiases ased color/shape layer. */ transform, Stransform = sample crampleerrassfommoooooo Biased) uniform. Sample(Stranssffaeooeee img transformiing, s)
end
rand() < Paropout then
uniform sample S dropout)
img erase(img, s)
end
return img
22

---

Published as a conference paper at ICLR 2023
F ADDITIONAL DISCUSSION
F.1 GENERALIZATION TO OTHER MODEL ARCHITECTURES
Vision transformer (ViT) (Han al. 2023) is an emerging model architecture in the computer vision community. The latest ViT work (Zhai et al. 2022) achieved state-of-the-ar clean accuracy on the ImageNet recognition challenge (Russakovsky et al,, 2015). It therefore naturally attractive to test if ViT could also excel regarding adversarial robustness. (Shao et al., 2023) observed that ViTs exhibited higher adversarial robustness compared to contemporary CNNs with standard training set- tings. (Mo et al.) 2022) found that adversarial training still necessary for ViTs to attain decent adversarial robustness. (Mo et al. 2022) then conducted extensive experiments to evaluate various adversarial training techniques for adversarial robustness. In one of the experiments, they found that CutMix Yun et al. 2019) and Mixup (Zhang et al.) 2018) were beneficial for adversarially-tran ViTs to obtain higher adversarial robustness, while RandAugment (Cubuk et al.] (2200) impaired robustness instead. This is consistent to our observation that data augmentation methods like Ran- dAugment designed for standard training can be too hard for adversarial training. In fact, (Mo al.) 2022) attributes the failure of RandAugment to the same cause: adversarial training of ViTs". Therefore. we believe that our method of balancing hardness for data augmentation should also improve the adversarial robustness of ViTs.
is too difficult for
F.2 GENERALIZATION TO LARGER DATASETS
We argue that Cropshift and IDBH can still be effective on larger datasets like ImageNet (Deng eet al. 2009). The crop-based transformation RandomResizedCUrn (RRC) has been widely used be- fore in adversarial training on ImageNet (Wong et al.] 2020 Shafahi et al. 2019). Although RRC is different from Cropshift's backbone crop transformation Padcrop, it is easy to adapt the idea of Cropshift to RRC to augment its diversity. For example, one can treat the outcome of RRC pipeline as the input image the Cropshift pipeline. On the one hand, the way Cropshift augments data is unseen in augmenegdtt data so it creates effective new data instead of something duplicated. On the other hand, Cropshift retains most of the semantic information strength, so the distribution shift caused by Cropshift can be controlled to be benign, i.e., learnable and beneficial. Regarding IDBH, it inherits component transformations and multi-layered structure from the precedent works AutoAugment (Cubuk et al.) (019)) and Trivial (Müller & Hut- ter, (02I), and optimizes the strength (hardness) of transformations for adversarial training. Given the superior performance of AutoAugment and TrivialAugment on ImageNet, we expect IDBH to improve adversarial training on ImageNet once appropriately optimized. Therefore, we believe that Cropshift and IDBH can benefit adversarial training on ImageNet. However, unfortunately, conduct- ing adversarial training on ImageNet is beyond our computational resources so we can not provide empirical verification of our argument this time.
the input images at normal
F.3 FUTURE WORKS FOR EFFICIENCY AND EFFECTIVENESS IMPROVEMENT
Low-efficiency heuristic search is a considerable drawback of our work that prevents it from scal- ing to larger datasets and being applied easily to new datasets. We believe that automatic search methods like FasterAA (Hataya et al. 2020) and DDAS (Liu et al., 2021) could provide feasi- ble and promising solution to this issue in the future. In general, they parameterize the underlying data augmentation sampler for gradient computation and then integrate it into the backpropagation pipeline so that the parameters of data augmentations can be optimized towards some predefined objectives, e.g., maximizing clean accuracy (Cubuk al. 2019). Some automation techniques can be incorporated into the normal training pipeline to allow continuous, online, updating of the data augmentation strategy. This is in contrast to the offline data augmentation method adopted by our method where the data augmentation strategy is constant throughout training. Online data augmen- tation, ideally, should outperform the offline one, since theoretically adapts data augmentation for models at every stage in training.
23


Reviews:
[{"review_id": "oIzh9hz3Ux", "reviewer": "Reviewer_qfhi", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The work demonstrates that data augmentation alone can alleviate robust overfitting.\nThe authors investigate what factors contribute to the robustness and point out that the hardness and diversity of the augmentation significantly influence the robustness and accuracy.\nThey propose a new image transformation method Cropshift and claimed to be more diverse compared to conventional methods.\nBased on Cropshift, IDBH, a 4-layer image augmentation scheme is proposed and achieved SOTA performance and boosts a lot compared to previous data augmentation methods.\n", "summary_of_the_review": "Given the significance of the insights provided in this work, I tend to weak accept this work.", "strength_and_weaknesses": "Strength\n1. A novel finding that data augmentation alone could alleviate robust overfitting\n2. This work also investigates when data augmentation help and proposes a promising data augmentation to improve model adversarial robustness\n\nWeakness\n1. IDBH is also comparable to Regularization methods in terms of improving model robustness. It is interesting to also study whether the proposed data augmentation can be combined with regularization to further improve model robustness.\n2. The presentation remains to be improved. What is 'end accuracy'? When it can be lower/higher than 'best accuracy' in Tab. 1?\n3. The main evaluation is conducted with PGD10. Does the PGD attack converge with 10 steps on the applied dataset and models?\n4. In Tab. 3, the author aims to demonstrate that the proposed method works well across different datasets. However, some of the strong baselines in Table 2 are removed. Are there specific reasons behind this?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "lvAewSXiXT", "reviewer": "Reviewer_4eAk", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes Cropshift, a new data augmentation to improve the robustness and accuracy of adversarial training. The design is based on the empirical findings that diversity matters and hardness can boost robustness at the cost of accuracy.", "summary_of_the_review": "The method achieves great results, but the paper fails to illustrate important parts of the underlying mechanism.", "strength_and_weaknesses": "Strength\n+ Compelling experimental results. The increase in accuracy and robustness is significant.\n+ The proposed method would be easy to implement with (promised) available codes.\n\nWeakness\n+ Although it is clear that the hardness of a data augmentation method is measured by the robustness difference between the augmented model and the unaugmented one, it is unclear how it is controlled, e.g., to 7 different levels as the paper states. Is increasing hardness achieved by changing the strength of augmentation?\n+ The idea of jointly handling hardness and diversity is not novel, which has been extensively studied in normal training [1]. Extending it to adversarial training seems incremental to me. Besides, the title looks close to [2].\n+ It is claimed that “The probability and the strength of each layer (augmentation strategy) was jointly optimized by a heuristic search to maximize the robustness.” But the details of the heuristic search have not been illustrated. In this regard, it is unclear how the method optimizes the strength.\n\n[1] AugMax: Adversarial Composition of Random Augmentations for Robust Training, NeurIPS 2021.\n\n[2] Data augmentation can improve robustness, NeurIPS 2021.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "a-EyNvlMsm", "reviewer": "Reviewer_KVU6", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work aims to address the robust overfitting issue in adversarial training by using data augmentations. Prior works have shown that adversarial training does not benefit from augmentations such as AutoAugment. The authors firstly study the role of hardness and diversity of augmentations in robustness and accuracy, and find that diversity improves both, while increasing hardness can improve robustness at the cost of clean accuracy initially and later degrades both. To improve diversity, the authors propose a new augmentation Cropshift, and further use this in the proposed augmentation pipeline - Improved Diversity and Balanced Hardness (IDBH) which has better diversity and well-balanced hardness. \n", "summary_of_the_review": "Although there are several novel and interesting aspects in the paper as mentioned above, the results of baselines and proposed approach are suboptimal for WideResNet-34 when compared to prior works. \n\n**Post-Rebuttal comments:** I thank the authors for their response which addresses many of my concerns. I suggest the authors include the explanation of - \"why increasing hardness degrades clean accuracy and improves robust accuracy\" in the paper as well, specifically the clarification that, \"model capacity is insufficient to fit benign samples under a stronger regularization from adversarial training caused by increasing hardness\". \n\nClarification on the suggestion -  Based on my understanding of the algorithm, the explanation can be updated to \"Cropshift first randomly crops a region in the image and then places it at a random location on a new blank image.\"\n\n\n", "strength_and_weaknesses": "Strengths - \n- The analysis on the impact of increasing hardness and diversity of augmentations is insightful. \n- The proposed method shows good improvements on PreActResNet-18 when compared to baselines. \n- The authors perform a more robust evaluation of prior work by Tack et al., and show that their claim of superior robustness using AutoAugment is incorrect. \n\nWeaknesses -\n- As the authors mention, the selection of augmentations is currently a bottleneck, which makes it impractical to use, due to the requirement of manually finding the best set of augmentations. Even using strategies like the one used in AutoAugment is expensive due to the large search space. \n- WideResNet-34 results seem to be much lower than the expected results [1]. Moreover, it is unclear why this is even lower than the results of PreAct-ResNet-18. \n- It is unclear why increasing hardness degrades clean accuracy and improves robust accuracy. If the model capacity is insufficient for learning benign samples, it should be harder to learn adversarial samples. Moreover, the argument of robust generalization can also hold for standard generalization. \n\nSuggestions - \n-  This seems a bit confusing and different from the algorithm - \"Cropshift first randomly crops a region in the image and then shifts it around to a random location in the input space.\" - could be rephrased.\n\n\n[1] Pang et al., Bag of Tricks for Adversarial Training", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "59JBvVV4P4", "reviewer": "Reviewer_5354", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies adversarial training in the context of image classification. The authors show that data augmentation alone can improve robustness. In the previous literature, works showed that augmentation needed to be combined with other components in order to improve performance. The authors propose a new crop transformation called Cropshift which improve robust performance.", "summary_of_the_review": "The idea of proposing a new data augmentation specially designed for robustness is novel and an interesting research direction. The authors provide extensive experiments to support their claims and give an interesting analysis frame with the hardness and diversity study.", "strength_and_weaknesses": "Strength:\n1) Claims are well motivated with a thorough study based on hardness and diversity. The story is quite clear with data augmentations which should be adapted to the capacity of the model.\n2) The introduced data augmentation is tailored for robustness and gives indeed a significant boost in performance.\n3) Thorough experiments with many ablation studies.\n\nWeakness:\n1) Cropshift works well with CIFAR, SVHN and TinyImageNet but datasets with larger images such as ImageNet typically benefit from other types of data augmentations than small datasets. So it is not sure that Cropshift would be helpful in that case.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster