Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies the role of atypical samples in the context of adversarial robustness, and further introduces a variant of adversarial training based on these observations. The proposed Benign Adversarial Training (BAT) incorporates an additional temperature-scaled n-pair loss and weights adversarial examples according to their margin. BAT shows improved clean accuracy at similar levels of robustness compared to baselines.
main_review: Strengths:
- Interesting observations and analysis provided on the behavior of atypical samples in the context of robust overfitting 
- Promising empirical results showing improved clean accuracy at similar or better levels of robustness with strong attacks
- Analysis of the effect of individual components in ablation studies
- Paper is well-written and easy to follow

Weaknesses:
- The choice of threshold for mem(x) used to define an atypical sample seems arbitrary. How do the results vary with different thresholds for mem(x) used to define the atypical set?  
- Even though the paper is initially motivated by atypical samples, the mechanism relating atypical-ness to adversarial robustness is not clear and moreover it seems to me that the analysis in "Poisoning Atypical Samples" and the algorithm design that follows can be better understood from the perspective of margins and adversarial robustness; examples with smaller margins (closer to the decision boundary) are necessarily more susceptible to adversarial attack. In particular:
  * Theorems 1/2 are not specific to the notion of atypical samples, and are about samples near the decision boundary (low margin). These theorems are essentially saying that by trying to accomodate these samples near the decision boundary for an optimal classifier, the model will be less robust.
  * The two components of BAT involve other elements that are not related to atypicalness but rather margin - the reweighting involves a poisoning score that is essentially the margin of the example ("likely to be close to the distribution of a wrong class"), and the discrimination loss may be viewed as attempts to increase the margin between the different classes.

  It would be useful to isolate the effect and importance of the atypical-ness of an example in the algorithm, for instance by applying the algorithm to all examples not just the atypical ones.

Other comments/suggestions: 
- Metric learning was previously used for adversarial robustness in Metric Learning for Adversarial Robustness, NeurIPS 2019 though in a different way
- In light of the relation to margin, some related work from the margin perspective that could be discussed include 
  * Helper-based Adversarial Training: Reducing Excessive Margin to Achieve a Better Accuracy vs. Robustness Trade-off, ICML Workshop on Adversarial ML 2021
  * MMA Training: Direct Input Space Margin Maximization through Adversarial Training, ICLR 2020
  
  In particular, the first paper achieves similar results as the current work by restricting the margin increase through different means; the downweighting of low (or even negative margin) examples in BAT has a similar effect.

- Typos:
  * Sec 3.2 on p4: trails -> trials
  * Theorem 1: the condition $d_{+1} \leq 2\eta$ should be on $d_{-1}$ instead

summary_of_the_review: Overall the proposed BAT algorithm achieves promising results that improve clean accuracy at similar robustness levels, but the design of the algorithm seems to involve elements that are orthogonal from the initial motivation and study of atypical examples. The importance of the atypical-ness of an example should be clarified during the discussion period to better motivate the method. 

**Post response update:** 
The authors have comprehensively addressed my concerns about relation to margin and importance of the atypicality to the components of BAT through ablation studies. I have increased my score accordingly.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors study the interaction between adversarial training (meant to produce robust models) and atypical examples in standard datasets (examples where generalization is driven by a handful of examples, cf [Feldman 2020]). They find that, in contrast to standard training, adversarial training has a hard time generalizing to atypical test examples in the adversarial setting. They also find that adding atypical examples to adversarial training can actually hurt test accuracy, in contrast to standard training where such examples typically help. Finally, the authors propose a method that down-weights  atypical examples during robust training and introduces a contrastive regularizer, which they term BAT (benign adversarial training).
main_review: The premise of the paper is interesting. While previous work has discussed "atypical examples" in the context of adversarial training, this is (to the best of my knowledge) the first work that performs such a study in a rigorous manner by considering proper measures of atypicality through influences. The findings are thus not always surprising given prior work, but are nevertheless interesting and the proposed method does show promise.

There are however a few issues that I would like to point out:

- The theoretical setting is actually quite similar to the one in [Sanyal et al.]([https://arxiv.org/abs/2007.04028](https://arxiv.org/abs/2007.04028)) (cited). Their focus is on label noise instead of atypical examples (which is not a very different setting after all), but both analyses boil down to robustly classifying samples from a Gaussian in the presence of outliers. This does diminish the impact of the paper somewhat. In any case, the discussion of that paper in the manuscript should be expanded.
- The discussion around "poisoning atypical examples" (e.g., Figure 3) is intriguing but somewhat speculative. Is there a more rigorous way to explore this phenomenon? E.g., by choosing samples with high influence and verifying them through human annotation?
- It is not entirely clear why the discrimination loss regularization would be beneficial in this setting. The authors do provide a justification based on the class margin in representation space. However, since the model is trained end-to-end, why wouldn't standard adversarial training discover this solution?

Minor comments (not affecting score):

- The title is confusing, what does "towards the ... effect" mean?
- [Balaji et al. 2019]([https://arxiv.org/abs/1910.08051](https://arxiv.org/abs/1910.08051)) is quite relevant and worth discussing.
- Why are the PGD-generated adversarial examples during adversarial training referred to as "manually generated"?
- Figure 4 is hard to parse in 3-D. It seems that a table would provide the same information in a more readable format.
summary_of_the_review: Overall, despite its shortcomings, the paper presents some findings that would be of interest to the community.

--- 

Post-rebuttal update: Given the authors' clarification of their theoretical contributions and the provided small-scale user study, I increase my score.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper explores the "memorization" in adversarial training. Through some empirical experiments and theoretical analysis, this paper points out two findings: (a) memorizing atypical samples can improve DNN’s accuracy on clean atypical samples, but hardly improve their adversarial robustness and (b) memorizing some atypical samples can even hurt the DNN’s performance on typical samples. Based on these findings, the authors propose a method called benign adversarial training by reweighting. Empirical results show that it can achieve better clean accuracy or robustness than some baseline methods in CIFAR-100 and Tiny-ImageNet.
main_review: Pros:

1. This paper makes a connection between the memorization effect of DNN with adversarial training and provides some comprehensive experimental results on analyzing the effect of training data for adversarial robustness. And this study shows a promising perspective for adversarial robustness, which focuses on the data perspective. 
2. The proposed benign adversarial training gains consistent improvement (at least using the two benchmark datasets) on both natural and robust performance across different evaluations (including the strong attack method---AA). 

Cons:

1. The major concern is the novelty of the main contributions. The similar "important" findings in this paper have already been pointed out and analyzed by previous studies [1,2,3]. For example, in [1], the authors also investigated memorization behavior in adversarial training and have pointed out that overfitting some rare examples in training can just improve the clean accuracy, and ignoring these rare examples helps in adversarial robustness. And in [3], the authors show that using high-quality (not so rare or hard) examples can indeed improve both natural performance and adversarial robustness, which has a consistent high-level idea with benign adversarial training.
2. Some claims are not well-supported. It may be better to add some citations to justify or provide some empirical results, especially for the memorization behavior in adversarial training. For example, how to confirm that these atypical samples have deviated from the main sub-population? Whether DNN can indeed fit every atypical samples' labels since we know that the model capacity for adversarial training is always not enough (in other words, adversarial training may require DNNs with a large model capacity than standard training [4]).
3. The motivation for exploring or connecting memorization with adversarial training is not clear and not strong (At the top of page 2). The recent study mainly focuses on the "overfitting" phenomena in adversarial training, so what's the relationship of memorization in adversarial training with such "overfitting" phenomena? (Personally, since there seems to be a logic gap, it is hard to understand why we need to explore the memorization behavior in adversarial training).
4. The reweighting strategy may need further explanation or illustration. 
5. What's the relationship of the proposed method with adopting curriculum learning in adversarial training?
6. Empirically, if the data distribution is long-tail, how can we apply the benign adversarial training? Can benign adversarial training also effectively utilize the long-tailed data since it seems to be similar to the atypical samples mentioned in this paper?

Minor:

It is better to use $f$ instead of $F$ in all the equations to appropriately indicate the loss function. 

References:

[1] Sanyal, Amartya, Puneet K. Dokania, Varun Kanade, and Philip Torr. "How Benign is Benign Overfitting?." In *International Conference on Learning Representations*. 2020.

[2] Dong, Yinpeng, et al. "Exploring Memorization in Adversarial Training." *arXiv preprint arXiv:2106.01606* (2021).

[3] Dong, Chengyu, Liyuan Liu, and Jingbo Shang. "Data Profiling for Adversarial Training: On the Ruin of Problematic Data." arXiv preprint arXiv:2102.07437 (2021).

[4] Wu, Boxi, et al. "Do Wider Neural Networks Really Help Adversarial Robustness?." *arXiv preprint arXiv:2010.01279* (2020).
summary_of_the_review: Overall, this paper explores the memorization effect in adversarial training and proposes benign adversarial training which further enhances adversarial training. However, although the direction is promising, the major concern is about the novelty of the main contribution since the similar findings in this paper have already been pointed out and analyzed in previous studies [1,2,3], and simply dropping some called "atypical" data can also improve adversarial training [3]. Considering the similar results from the perspective of data quality for adversarial robustness,  the current submission may need some further exploration to provide some different and in-depth findings.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper investigates the memorization effects in adversarial training, especially on the effect of memorizing atypical examples. Based on the empirical observations, the authors propose benign adversarial training (BAT), which is evaluated on CIFAR-100 and Tiny ImageNet.
main_review: Strengths:
- The empirical observations on the effect of memorizing atypical samples are interesting.
- The writing is clear, with detailed descriptions of the proposed method.
- It is good to have ablation studies on different modules in BAT.

Weaknesses:
- Since there are several extra modules in the objective of BAT (Eq. (10)), there should be evaluations under adaptive attacks, whose objective involves the terms like $\mathcal{L}\_{DL}$.
- As described in the paper, it seems that the authors do not apply early-stopping during the adversarial training process. This largely weakens the performance of baselines [b]. Besides, more advanced methods listed in [c] should be considered on CIFAR-100.
- CIFAR-100 and Tiny ImageNet are not quite widely used in the adversarial community. It would be more informative to have the results like Table 1 on CIFAR-10, where many benchmarks exist to better estimate the effectiveness of BAT.

Minors:
- As suggested by [a], weak attacks like FGSM should not be treated as an evaluation, at least not in the main text.
- There should be more details on the settings (e.g., iteration steps) of PGD and CW attacks.
- The $1/\lambda$ is 6 in TRADES paper, while it is set as 5 in, e.g., Table 1.

References:
[a] On Evaluating Adversarial Robustness.
[b] Overfitting in adversarially robust deep learning.
[c] RobustBench: a standardized adversarial robustness benchmark.
summary_of_the_review: Interesting observations on the memorization effects of atypical samples in adversarial training, but the empirical evaluations should involve stronger baselines (e.g., using early-stop) and adaptive attacks (e.g., considering the BAT modules), while it would be much more informative to have, e.g., AA results on CIFAR-10.
review_title: null

Meta-review: This paper tries to improve the training of adversarial deep neural networks by avoiding fitting the “harmful” atypical samples and fitting more “benign” atypical samples. 

Overall, the main concerns are

1. The current presentation can easily cause some misunderstandings on the observations made in Section 3, especially [1] and [3] mentioned by the reviewer iXiX. 
- The authors may consider moving "related work" to the first half of the submission, and organize existing findings with rare/hard/atypical in a more principle manner. 
- Besides, as author mentioned in Section 3.1: "it is equivalent to a classification task based on an extremely small dataset, with one or a few training samples given". Such findings are natural and not novel to the deep learning community. Authors may consider shorten Section 3.1 and elaborate more in Section 3.2.

2. Theorem 1 and 2 do not help much. 
- It does not talk about the training algorithm and models, which over simplifies the learning problem. 
- Besides, the authors can consider some theoretical results how BAT improve the performance of typical samples, but still preserve the ability to fit those "useful" atypical samples. 
This helps to bridge the gap between motivation behind BAT and its algorithm design (raise by reviewer  ytJj and sm19).

3. It is also suggested to make observations more convincing. 
- Since authors want to claim their findings are universal, it is better to consider more adv training methods and datasets; it is also better to change the ratio of "normal samples" v.s. "atypical samples". In this way, the effect of atypical samples in adversarial training can be more carefully quantized.
Decision: Reject