Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The main result of this paper is that some kinds of data augmentation can boost adversarial robustness when combined with model weight averaging. Prior work investigating data augmentation does not perform weight averaging, and reported that data augmentation wasn't useful. This paper reports empirical gains on common benchmarks with CutMix augmentation, and run some experiments to see where the gains in model weight averaging are coming from, the effect of unlabeled data, generalization across different architectures etc. 
main_review: Clarity: Overall the paper was very well written: claims were clear and well motivated. I had a few questions about the results/writing below: 

l131-134: it is unclear to me why "model weight averaging helps robustness to a GREATER extent when the accuracy across iterations can be maintained". The plot only shows that even when there is no robust overfitting, weight averaging (WA) could be helpful

l161-175: How did you decide which augmentations to try to use? In particular, why do you only study cutout, cutmix and mixup compared to other things like rotations, autoaugment etc? It does seem like the kind of data augmentation to use is quite important empirically, and it would be nice to see a more substantial discussion about this in the paper. 

l161-175: Relatedly, do augmentations that help standard training also help robust training (with WA)? What's a good heuristic to decide which augmentations to use in robust training vs standard training?

l194: It says that you train two models for each hyperparam setting so are the reported numbers in the table the average across these models? What's the variance like? From personal experience, SVHN at \eps = 8/255 is quite high variance across training runs so it would be good to check 

l221-229: The paper tries to explain why MixUp doesn't work as well - by saying that the blended images might be very far from original images. However, this could be true even for CutMix where you literally replace parts of an image with another. It would be nice to have some sample illustrations of what the augmentations look like if you believe there is a significant qualitative difference between these kinds of augmentations that lead to the result. 

l235-238: I didn't follow the argument here: how does adv training's underfitting determine which data augmentations work and which do not?

l249-257: The gains of data augmentation when using extra data are quite small- does this improve when you train the models for longer, if underfitting is the issue?

l46-48/ Sec 6.3: The empirical evidence about why data augmentation helps is not really substantiated because the current paper doesn't present results for what the model predictions look like when not using data augmentation. 

Other questions: 
1. What happens if we ensemble different models trained from scratch without data augmentation with early stopping? This is expensive but as per the intuition in the paper, this should also work just as well or better to improve the robust accuracy?
2. How did you make sure that the attack evaluation was not misleading? It's great that the authors use AutoAttack which tunes hyperparameters of the attack directly, but it would be good to know what other checks the authors used to make sure that the improvements are not simply because of bad tuning of the attacks. This is a real concern because the gains are generally a bit small 

Significance: This paper makes a nice observation that data augmentation with WA improves robustness. This makes the story more consistent with standard training where data augmentation typically helps. However, the gains are not big enough to have a substantial empirical advantage. Also, the paper doesn't add much new insights into why some data augmentation reduces overfitting, why do some augmentations work better with standard training vs adversarial training? So it's unclear what future works could build on this paper. 




Originality: The paper takes two well-studied concepts (data augmentation and model weight averaging) and puts them together. This work is also very close to the work of Rice et al. that study robust overfitting. They also reported the observation that data augmentation leads to lesser overfitting i.e. drop upon training (but same best robust accuracy). This paper builds on that observation to use weight averaging to get more gains from data augmentation. 
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper demonstrates that data augmentation with a weight average can mitigate the adversarial overfitting problem and also can boost the robustness with a large margin. Especially, the author found that cutmix and weight average work the best for adversarial training with large performance gain which is state-of-the-art.
main_review: Originality: 
This method is not a novel method combination of cutmix and weight average, however, demonstrates the adversarial training also can get benefit from data augmentation and weight average technique for the first time.  Moreover, the technique can also prevent the robust overfitting problem.

Quality: 
The submission is technically sound. However, it is a bit unclear that whether a weight average boost the performance of robustness or data augmentation does. Also, does weight average help in mitigating overfitting problems or augmentation. Cutmix itself already did not suffer any overfitting while with weight average it can gain large performance gain. However, in pad&crop, weight average seems to mitigate the overfitting problem but it did not boost the performance that large enough.

As I understand, the main message is augmentation helps in robustness. However, the paper mainly figures out that model weight averaging helps robustness to a greater extent when robust accuracy between model iterations can be maintained which seems different content from the title and abstract.

Clarity: Submission is easy to understand. However, it was hard to follow what is the main claim of the paper as I aforementioned.

Significance: Results are interesting that most of the previous work claim that augmentation does not give any benefits to robustness which was wrong in some sense. They provide thorough experiments on different sizes of architecture and diverse datasets. However, since the proposed method is just combining existing methods and didn't demonstrate any plausible reason why those two techniques help in overfitting and performance, I am not sure this work is significant enough.
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper has demonstrated that data augmentation can be one key factor for preventing robust overfitting. This is an interesting observation as it contradicts the previous studies. The author further combined augmentations with weight averaging to improve the robustness. The experiments are extensive and show impressive results.
main_review: ----------------------------------------------------------------------
**Pros**
1. **The finding is interesting that data augmentation can prevent robust overfitting**

2. **The experiments are extensive and show significant improvement**

3. **The writing and presentation are clear**
----------------------------------------------------------------------
**Cons**
1. **Details are missing: cannot find the training objective or algorithm table**\
Did the author utilize CutMix after generating the adversarial example, or first apply CutMix then generate the adversarial example; an ablation study is needed (which one is better and why?)\
How did the author modify the TRADES [8] objective with CutMix...? I believe this part is not trivial as it contains KL divergence in the loss formula.

2. **Concerns about the analysis of data augmentation(DA): Why is CutMix the most effective method**?\
Is it related to the clean accuracy of standard ERM training?\
If so, did the author have tried AutoAugment [4] or RandAugment [5] (which shows the state-of-the-arts performance)? Since these techniques do not need other instances to define the augmentation (unlike CutMix), hence, it is more simple to analyze

3. **Concerns about the analysis of data augmentation(DA):  Which is more critical for improving the robustness, DA, or label mixing**?\
Since the mixing methods are a combination of augmentation and label mixing [3], maybe decomposing the effect might help for the analysis.\
E.g., in Figure 6, the CutMix is even effective under a window length of 30, which is extremely interesting. The window length of 30 will not contain an important foreground feature of another instance. This result might indicate the label mixing (or smoothing) is essential.

4. **Concerns about the novelty**\
The method itself is not specified for adversarial robustness, where it can be applied to various classification problems.\
Also, It seems that weight averaging (WA) and DA is an orthogonal contribution, as the DA prevents the overfitting and WA is for improving the generalization. Note that [2,3] already has shown that loss landscape smoothing is effective for robust overfitting and generalization (as mentioned in line 118).

----------------------------------------------------------------------
**Questions**
1. Is it necessary to use WA rather than Sharpness-aware-minimization (SAM) [6]? Since SAM is known to be an advanced version of WA, it might show better performance. Note that SAM and AWP [1] share the same idea.

2. The author used Adam optimizer for the inner maximization, which is not common (I only have seen such optimization in local linear regularization [7]). Is there a reason that the authors used Adam rather than the common PGD?

3. Why is the combination of AutoAttack and Multi-target attack is less effective than the AutoAttack…? 

4. In Figure 1, which model architecture is used for the comparison?

5. What does "(retrained)" indicate in Table 2? Especially for SVHN and TinyImageNet. Is it the retrained version of Cui et al., or Gowal et al.?

----------------------------------------------------------------------
**To sum up, I recommend (weak) rejection for this conference**.\
I believe the author needs more analysis on the data augmentation as it is the main contribution of this paper. Also, building a method that is specified for adversarial robustness will be great. 

Also, some details are missing or questionable. I did not consider this as a negative factor for the initial review, however, the author needs to some parts for better clarification.

----------------------------------------------------------------------
**You can find the references here**\
[1] “Adversarial Weight Perturbation Helps Robust Generalization”, Wu et al., NeurIPS 2020\
[2] “Robust Overfitting may be mitigated by properly learned smoothening”, Chen et al., ICLR 2021\
 [3] “On the Generalization Effects of Linear Transformations in Data Augmentation”, Wu et al., ICML 2020\
[4] “AutoAugment: Learning Augmentation Strategies from Data”, Cubuk et al., CVPR 2019\
[5] “RandAugment: Practical automated data augmentation with a reduced search space”, Cubuk et al., NeurIPS 2020\
[6] “Sharpness-Aware Minimization for Efficiently Improving Generalization”, Foret et al., ICLR 2021\
 [7] “Adversarial Robustness through Local Linearization”, Qin et al., NeurIPS 2019\
[8] “Theoretically Principled Trade-off between Robustness and Accuracy”, Zhang et al., ICML 2019
limitations_and_societal_impact: Yes, they have discussed the limitations and societal impacts.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors combine adversarial training with popular data augmentation techniques. They discuss how model averaging and data augmentation can be combined to prevent robust overfitting. The authors evaluate their method on several datasets and show state-of-the-art results on CIFAR10.
main_review: ### Quick justification of rating:
I enjoyed reading this paper and think the authors did a good job first explaining, isolating and motivating the problem, then reviewing related work and possible solutions and finally, presenting their own solution along with the necessary experiments and ablation studies.

I second the authors’ sentiment in line 76: “As such, it is rather surprising that they remain ineffective when training adversarially robust networks [15, 38, 49].” Data augmentation techniques are popular in e.g. domain adaptation and corruption robustness. It is an interesting idea to try to understand under which conditions data augmentation can be beneficial for adversarial robustness. Therefore, I think this paper will be relevant to the adversarial community.

The authors report a new state of the art on CIFAR10 for adversarial robustness with their method.

I would appreciate it if the authors answered my two comments below in the rebuttal and also clarified them in the main paper.

1) Figure 6c: Model selection. The authors show here how robust accuracy changes depending on the window length. 1) Did the authors then use the determined best value for the window length (20) to report their main results in Table 3? 2) Was this hyperparameter chosen on a separate validation set or on the CIFAR10 test set? If it was chosen on the test set, is this model selection process legit in this case? (Hyperparameters should be chosen on a separate validation set.) 3) Was the same value used to obtain the results on the other datasets in Table 2? The same questions apply to all other used data augmentation techniques, e.g. which values were used for Cutout. Maybe, it would be helpful if the authors considered subscripting the hyperparameter for the used technique, e.g. writing Cutmix_{20} if this value is different between datasets.

2) No adaptive attacks were considered. Please see [1] and [2] for why adaptive attacks should always be performed when proposing a new defense. 

### Paper writing: very good

The paper is very well written, easy to follow, and I really enjoyed reading it. The authors motivate their method well and also provide a good context for their paper in the related work section. The methodology in section 3 is explained well, e.g. all variables are explained, the definitions are precise. The concept of “robust overfitting” and possible solutions are explained well and are easy to understand.

I greatly appreciate the statement in Fig. 1: “Our method builds on Gowal et al. [15] (shown above with 57.20%) and explores how augmented data can be used to improve robust accuracy by +2.87% without using any additional external data.” This statement allows to easily place this paper in context with related work.

### General comments:

Line 114: Model Averaging is a popular technique in teacher-student training in domain-adaptation where the teacher model uses the EMA weights of the student model, e.g. [1]. I wonder whether there is a connection / interpretation here, similar to how MA is used in domain adaptation?

Line 150: minor comment: Writing “verifying the hypothesis” reads like the authors pursue “an agenda” and design the experiment to confirm their hypothesis. I would suggest rephrasing the statement as “Testing the hypothesis”.

Paragraph in line 150: I think it would be important to explain how “Mixup” and “Cutmix” are implemented. For Mixup, are the authors interpolating between clean and adversarial images? Same for Cutmix, are authors replacing a part of a clean image with an adversarial image? Does it make a difference how the techniques are applied? E.g. does it make a difference for Mixup whether the mixing images are of the same kind, e.g. we interpolate only between clean images and only between adversarial images?

Fig3b: Could the authors explain the drop in robust accuracy after the learning rate drop with WA?

Line 196: “pick the best model by evaluating the robust accuracy on the same validation set.” I realize that the authors follow the procedure in Gowal et al, but this choice seems strange to me. In [2], the authors argue that in adversarial robustness, we are interested in the worst-case robustness and this is the value that should be reported. I am therefore surprised by the unintuitive choice to pick the best model here.
[2] Nicholas Carlini, Anish Athalye, Nicolas Papernot, Wieland Brendel, Jonas Rauber, Dimitris Tsipras, Ian Goodfellow, Aleksander Madry, Alexey Kurakin, “On evaluating adversarial robustness”

I like Fig. 4 since it shows clearly that using WA is better than not using WA and the color code makes it easy to quickly find the different models. 

Line 221: lesser -> worse

Figure 6a: Please include the number for alpha=0. Similar for 6b and 6c, please include values for not using the data augmentation techniques for comparison.

Section 6.3. I found this section a bit hard to understand. I would suggest to the authors to look into [3] where the authors quantify the error consistency between different DNNs (and humans). Maybe the metrics used in [3] would be valuable for this study, and easier to understand than the current setting.
[3] Robert Geirhos, Kristof Meding, Felix A. Wichmann, „Beyond accuracy: quantifying trial-by-trial behaviour of CNNs and humans by measuring error consistency”

Figure 7 is mentioned before Figure 8. The ablation mentioned in Figure 7 does not really fit into subsection 6.3.

Table 2 is mentioned after Table 3, please fix this.

The authors should write what they have put into the Supplement. Currently, the supplement is not even mentioned in the paper. Please include a paragraph detailing the additional experiments and studies.

I appreciate showing results for black-box attacks in Table 4 in the Supplement. Could the authors compare to Gowal et al here as well as they have done in all other evaluations? 

[1] Tarvainen and Valpola: “Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results”
[2] Florian Tramer, Nicholas Carlini, Wieland Brendel, Aleksander Madry: "On adaptive attacks to adversarial example defenses"

### Review updates post rebuttal

I read the other reviews and the authors' responses.

I think the authors did a good job addressing my comments in their rebuttal. In particular, I had two main comments about hyperparameter selection and adaptive attacks. Considering my first comment: The authors wrote in their rebuttal that they used default hyperparameters from the papers where the augmentation techniques were introduced. I think this is a justified choice which means that they definitely did not "cheat" during model selection by choosing hyperparameters on the test set. Additionally, this choice makes their models comparable to models trained by other researchers trained with the used techniques when using the default hyperparameters. Considering adaptive attacks, the authors have a point that those attacks are primarily needed for cases when gradient obfuscation is a concern or when customized elaborate defenses are implemented which make the use of standard attacks impossible. Since they use standard training here and merely add data augmentation and EMA of the weights, gradient obfuscation isn't a concern.

I think this submission is publication-worthy because:
1.	Data augmentation is the standard "go-to" tool to increase robustness towards common corruptions. In self-supervised learning, for example in contrastive learning, choosing the appropriate data augmentation is key. It is peculiar that data augmentation (being such a vital ingredient in other areas of deep learning) has not worked at all so far for adversarial robustness. The authors are the first ones to show what needs to be done to make data augmentation work for adversarial robustness. This is (for me) the main novelty of this paper: The authors take a very popular technique that hasn't worked at all so far and make it work.
2.	The authors report a new state of the art in adversarial robustness on CIFAR10.

Thus, I still think it is a good paper and should be accepted.
limitations_and_societal_impact: Yes, the authors discussed the limitations and potential negative societal impacts of their work.
review_title: null

Meta-review: Despite the different scores, all four reviewers agree on two main points:

A) The empirical results, in particular the non-trivial robustness gains, are interesting.
B) The explanations given for the robustness gains are only preliminary and do not account for all observed phenomena (e.g., why certain augmentations work better than others).

The reviews then arrive at different scores based on how much importance they assign to each point.

Overall I agree with both points and see this as a borderline paper. The authors could have written a stronger paper by avoiding tenuous explanations and instead focusing on a thorough experimental evaluation. A paper with strong empirical results does not have to offer a comprehensive explanation for each observed phenomenon. Instead, it suffices to discuss possible hypotheses as directions for future work. If the authors want to provide explanations, they should be rigorously investigated with experiments designed specifically to test the explanations. Presenting explanations without corresponding experiments undermines the validity of the paper.

Nevertheless, the experimental results still outweigh the lack of rigor in the provided explanations for me and I recommend accepting the paper. For the final version, I suggest that the authors clearly separate experimental findings from potential explanations. The paper could benefit from de-emphasizing the latter, e.g., by moving some of the explanations to the appendix and instead moving more of the adversarial evaluation details to the main text.
Decision: Accept (Poster)