Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper presents a detailed analysis of adversarial training on several datasets (including CIFAR-10/100 and ImageNet), and observes three interesting (class-wise) properties: (1) some classes are extremely vulnerable to adversarial attacks; (2) for a certain class, its robustness is positively correlated to the norm of its FC layer's weight; (3) stronger attacks usually hurt vulnerable classes more. These results could be helpful for future works on furthering model robustness.


Pros:

(1) This paper is very clearly written and easy to follow.

(2) To the reviewer's best knowledge, this is the first attempt to analyze how adversarially trained models behave at the class level (rather than just reporting the overall performance on the whole dataset). And the observations are kind of surprising, e.g., there indeed exists certain classes that are much more vulnerable than others, and the observed confusion patterns (e.g., "class 9" and "class 1" in CIFAR-10 are easily confused with each other) are pretty consistent among different defense methods. 


Cons:

The reviewer believes some conclusions of this paper can be further enhanced if analysis can be provided for the following parts: 

(1) In Section 3.1, this paper finds certain (pairs of) classes are extremely vulnerable to adversarial attacks. As there are already some works that begin to take care of vulnerable samples (a more fine-grained level than vulnerable class) in adversarial training [1,2], it will be good to analyze if such customized strategies successfully reduce class-wise confusions under adversarial attacks.

(2) In Section 3.2, this paper observes that the robustness is correlated to the norm of weight, and hypothesis such phenomenon is caused by insufficient data in adversarial training. As there are already several works [3,4,5] on augmenting extra data for improving adversarial training, the authors can use these models to validate the hypothesis above. i.e., with more training data, will the robustness and the norm of weight become less related?

Besides, the reviewer is a little bit confused about the proposed attack in Section 4.  What is the motivation for adding the temperature parameter in PGD attack? Why it can hurt certain classes more? It seems that only empirical results are provided, but no motivation or explanations of why it can work are provided.  (which makes this section less related to the core parts of this paper and easily confuse readers)

The authors should carefully address these concerns during the rebuttal.


[1] Cheng M, Lei Q, Chen P Y, et al. Cat: Customized adversarial training for improved robustness[J]. arXiv preprint arXiv:2002.06789, 2020.

[2] Balaji Y, Goldstein T, Hoffman J. Instance adaptive adversarial training: Improved accuracy tradeoffs in neural nets[J]. arXiv preprint arXiv:1910.08051, 2019.

[3] Zhai R, Cai T, He D, et al. Adversarially robust generalization just requires more unlabeled data[J]. arXiv preprint arXiv:1906.00555, 2019.

[4] Carmon Y, Raghunathan A, Schmidt L, et al. Unlabeled data improves adversarial robustness[C]//Advances in Neural Information Processing Systems. 2019: 11192-11203.

[5] Uesato J, Alayrac J B, Huang P S, et al. Are labels required for improving adversarial robustness?[J]. arXiv preprint arXiv:1905.13725, 2019.
review_title: detailed & interesting analysis

final_score: rating: 4: Ok but not good enough - rejection
review_content: Summary: This paper examines the robustness of adversarially robust models at the class-level.  Specifically, they note a disparity in the class-wise robustness of models for standard datasets. Furthermore, they suggest that many of these class-level vulnerabilities are eliminated if the model is trained without the corresponding confounding class. Finally, they propose a temperature based attack to further degrade accuracy of vulnerable classes.

Comments: I think that fine-grained studies of existing robust models can be valuable to identify ways to improve their performance.  While the paper is based on an interesting finding, the analysis is not sufficiently novel or extensive, and does not lead to a better understanding of why this phenomenon occurs or actionable insights to improve the performance of robust models.

Novelty: This paper is not the first to note the disparity in class-wise robustness---for instance, a fairly extensive study has been performed in https://arxiv.org/abs/2006.12621. Even if one were to consider both works as concurrent, the paper under review lacks depth and it is not clear what, if any, the impact of its findings are (see below). Relatedly, there is a line of work on adapting adversarial training to account for instance-level disparities in robustness---for instance, https://arxiv.org/abs/1910.08051.

Takeaways: The authors do not provide any conclusive evidence for the cause of class-wise disparities in robustness. They claim that this is linked to the norm of the weights, however they do not establish a causal link between the weights and per-class robustness (Fig. 4 could simply be correlation). Moreover, I expect that a similar relationship would be observed between standard and robust per-class accuracies (from Appendix Figure 6)---instead of the weight norm and robust accuracy---if both these quantities were normalized. In general, it is also tricky to make inferences based on the norm of the weights (without proper normalization) as the outputs of individual neurons in the pre-final layer could have vastly different scales. Additionally, the authors do not demonstrate that their findings can help to improve model robustness.  At the end of every section, the authors do provide some suggestions, however these are vague and unsubstantiated.

Overall, I think in its current form, this paper lacks depth and novelty. The observation of class-wise disparities alone is not substantial enough a contribution. As I mentioned above, the authors do not supplement this finding with a clear explanation of the underlying cause, or any evidence that this finding could help improve model robustness.

Additional comments:

* Are the confusion matrices based on targeted or untargeted attacks? If it is the later, then these numbers do not provide a complete picture of class-wise robustness. For an example of class i, it is possible that the adv. example that maximizes loss within an eps ball is from class j, though adversarial examples for many other examples exist within the ball (and are even closer to the original sample). The authors should repeat the evaluation using pairwise-targeted attacks.

* Many of the experimental choices are not sufficiently justified. For instance, why FGSM is used for ImageNet training, and why the best model is not picked based on cross validation. Further, the authors do not once mention what kind of adversary they consider.

* It is clear from Fig. 6 that, for most datasets, the same classes with lower robust accuracy have lower standard accuracy. Are the confusion matrices in Fig 2 based on all test set examples or just the ones that are correctly classified in the non-adversarial setting? What do the standard confusion matrices for these datasets look like?

* It is odd that the second element in the first row in Figure 3b (18) is larger than the corresponding element in Fig 3a. From what I understand, 3a should be an upper bound on every element in 3b.

* When evaluating the temperature based attack, do the authors apply the final perturbation to the original model (without the temperature scaling) or to the scaled model? If the numbers reported in Table 3 are based on the latter, then they do not indicate the robustness of the original model as the inference process has been modified.

#### Post-rebuttal Update ####

I thank the authors for their detailed response and edits to the paper. However, even after reading the rebuttal some of my original concerns stand:

[Novelty] As I mentioned in my original review, the discovery of disparities in class-wise robustness is not new to this paper. In the rebuttal the authors mention that their finding is different from [1] because in [1] the measure of class-wise robustness is distance to the decision boundary with respect to every other class. However, this is in my view, is just an alternative and well-established measure of robustness---i.e., distance to the decision boundary and robust accuracy are fairly correlated and not fundamentally different.

[Takeaways] 
- The authors do not perform sufficient quantitative analysis to justify the link between robustness and weight norm. Moreover, without establishing the causality of this link, it is unclear to me how this observation provides any new insight to understand robust models. 
- Figure 7 (in the revised manuscript) shows that a similar class-wise disparity is present even in the *standard accuracy* of *standard models*. Thus, although I find the observation of disparities in class-wise robustness interesting, I believe further investigation is needed to understand whether this is just an inherent property of the data distribution that hurts both standard and robust models or is specifically tied to robustness. 
- The authors' comment about improved robustness based on methods adapted from [3] is misleading. As mentioned in Appendix G (I believe there is no Appendix H), this seems to be the case only for a specific attack. The authors themselves demonstrate a different attack under which the improved robustness of vulnerable classes disappears.

[Other comments] There is no evidence to suggest that untargeted attacks will find the *closest* adversarial example within an eps ball. The optimization problem for untargeted attacks is set up to maximize the loss (w.r.t. the ground truth label) and not to find the nearest misclassification. Thus, I still assert that to get a better picture of per-class robustness, the authors need to measure the targeted confusion matrix (or distance to per-class decision boundaries as in [1]).

Due to these concerns, I am unable to raise my score.

review_title: The motivating finding is interesting, but needs further investigation

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper empirically studies the class-wise properties of classification models produced by existing adversarial training methods on benchmark image datasets. In particular, it demonstrates the following observations: 1) robustness varies for different seed-target class pairs; 2) for certain class, retraining the model without other semantic-similar class improves its robustness; 3) class-wise robustness relates to the norm of the model weights; 4) stronger attacks are more powerful for vulnerable classes.


Pros:

+ The paper conducts extensive experiments, including six image benchmarks.

+ Heatmaps of class-wise robustness is visually good.

+ The fine-grained analysis on class-wise properties help understand the model performance to some degree.


Cons:

- The motivation of this paper seems somewhat unclear. The paper motivates its main research question based on the observation that most existing works in adversarial community focused on overall robustness. However, it is unclear to me why focusing on overall robustness is inferior. From my perspective, it is important to clarify the following questions in the introduction: 1) If you aim to improve the overall robustness using class-wise properties, what are the potential solutions? 2) If you think overall robustness is not the best evaluation criteria for adversarial robustness, what are the reasons and alternative criteria?

- The related work section simply listed many existing works related to adversarial training. The connections between these works and your work are not well-explained. More specifically, how will the understanding of class-wise properties contribute to the improvement of existing works is unclear.

- There is a lack of coherence in Sections 3 and 4. The subsections demonstrated in Section 3 are somewhat disconnected with each other. For instance, the transition from Section 3.1 to Section 3.2 is not well-explained. In addition, Section 4 really confuses me, i.e., the proposed Temperature PGD Attack seems not related to the previous sections and does not have an advantage over the vanilla PGD attack.


Minor comments:

- It is hard for me to see a strong positive correlation from Figure 4. I would recommend the authors to compute the correlation coefficient or conduct a correlation test to support the argument.

- The paper will be much stronger if the authors can improve the performance of existing robust models by leveraging the class-wise properties discovered in Section 3.

======= POST-RESPONSE UPDATE ========

I appreciate the author's efforts for responding my questions and providing additional results and I do find the empirical observations of class-wise properties interesting. However, I still feel that the contribution of the current form of the paper is not strong enough to reach the bar of ICLR, so I remain my previous rating. Beyond exploratory analysis, the paper would be much stronger if it can go deeper with the observed class-wise properties of robust models. 
review_title: Recommendation to Reject

final_score: rating: 4: Ok but not good enough - rejection
review_content: The authors study the robustness of adversially-trained models across different classes. They find that classes tend to have largely non-uniform robust accuracy--i.e., some are less robust then others. Moreover, certain datapoints can only be misclassified as specific classes and removing these classes during training can make these datapoints robust. Next, the authors investigate how this robustness discrepancy across classes relates to the norm of the last fully-connected layer of the model, as well as to the strength of the attack used. Finally, the authors propose a new adversarial attack that they evaluate against existing models.

The observation that robust models can have widely varying robust accuracy across classes is quite interesting. At the same time, the insights gained from the current study are not particularly impactful. Specifically:
- **Robust accuracy disparity.** Before drawing conclusions, it is important to perform the same study on a standard model. Are standard class accuracies on these datasets closer to uniform? Note that this will require accounting for the variance introduced by the overall lower accuracy of robust models.
- **Certain examples can only be flipped to specific classes.** Why is this surprising? It is natural to expect that classes that are similar to each other will be more likely to be flipped on multiple examples. As a result, removing one of these classes completely from the dataset will result in the other class becoming more robust.
- **Class robustness is tied to classifier norm.** First of all, the correlation of these quantities needs to be more formally investigated. Clearly, this is not a perfect correlation so we need at least some quantitative analysis to support this conclusion. Nevertheless, it is not clear what conclusion we can draw from this finding. Is the larger norm a cause or a symptom of brittleness? Can we make certain classes more robust by reducing their weight norm? Does this allow us to understand something fundamentally new about robust models?
- **Temperature scaling attack.** It is not immediately clear what the intuition behind this attack is. From what I understand, it will cause PGD to focus more on classes with higher (original) softmax probabilities. However this is not a fundamentally new attack. Moreover, if this viewpoint is correct, its performance should be lower than that of an attack that exhaustively considers all classes and tries to produce a targeted misclassification (e.g., MultiTargeted https://arxiv.org/abs/1910.09338).

Overall, while the topic and initial exploration is interesting, the findings do not provide us with a fundamentally new understanding of robust models. 

======= POST-RESPONSE UPDATE ========

I appreciate the authors' response and additional experiments. Unfortunately, my criticism still stands:
- **Comparison with standard models.** Based on Appendix C, it is still unclear if the class-wise disparity is a unique property of robust models. As we can see in Figure 7, standard models also have disparate accuracies between classes. More importantly, comparing the **absolute difference** between class accuracies is misleading, since robust models have overall lower robust accuracy. A better comparison would be to measure the **relative error** between classes. While it is hard to draw conclusions by just inspecting the graph, it seems that the discrepancy is significantly milder based on this metric.
- **Classifier norm.** The additional experiments still do not demonstrate any causal link between classifier norm and robustness. It is thus still unclear what this metric conveys.
- **Temp-PGD.** While I appreciate the effort to provide additional intuition about the attack, I still do not find the method fundamentally new when compared to other attacks optimizing combinations of logits (e.g., Carlini-Wagner, Multi-Targeted).

Overall, I still think that the original observation is intriguing, yet requires a deeper and more systematic study.
review_title: An interesting observation, requires further investigation

Meta-review: Analyzing class-wise adversarial vulnerability of models is an interesting direction to pursue. However, the authors should consult the references pointed out in the reviews to put their contributions in the right perspective. Overall, the lines of inquiry explored in this paper are of interest but, as some of the reviewers point out, there are improvement in the methodology that still need to be addressed before this paper is ready for publication. (I very much recommend that the authors do build on this feedback and revise the paper, as it will be a valuable contribution then.) 
Decision: Reject