Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper introduces the "clustering effect" of linear components of adversarial robust models. To be specific, the authors propose to remove the non-linear functions in networks and further integrate all linear components into a single weight matrix W in order to directly study the effect of linearity in adversarial models. By conducting experiments with W, the authors find that adversarially trained models tend to show clustering effects in both weight and feature dimensions. This effect further benefits some downstream tasks such as adversarial attack and transfer learning and can be enhanced with additional supervision. In all, the phenomenon showed in this paper is interesting and deserves more and deeper research.

main_review:   - Originality: It seems novel -- I don't see the proposed method before. 
  - Quality: Though the phenomenon shown in the paper is interesting, first of all, the setting is relatively simple, especially considering that the superclasses in most experiments are only animals and non-animals. Also, I do not see a convincing reason for ignoring fine-level accuracy in transfer learning from the authors. It would be good to explain why just compute the coarse-level accuracy. Second, the authors only point out this phenomenon and conduct further experiments on its applications without deep analysis on the reason behind it. I would expect more theoretical analysis on why linear components of the networks can show such clustering effects when trained with adversarial examples. Third, the experimental results are not always consistent and sometimes the improvement is also trivial which further enlarge my concern about the paper. For example, in Figure 5(a), we can see those non-robust models perform the worst as we would expect in certain situations but perform the best at other times. Also in Table 2, the performance after fine-tuning might be affected by the upper bound the backbone can achieve which can not fully support the idea that robust models can benefit more from fine-tuning.
  - Clarity: I have two questions that need to be further answered. First of all, how the proposed regularization loss is added to the training pipeline should be stated more clearly. According to Algorithm 2, after computing the regularization loss before retraining the model, is the feature clusters and the loss fixed or it is computed by the running mean of the samples? Besides, why the performance of AlexNet with R+C in Table 2 is so much higher than NR and R? (91 v.s. 85). I am wondering if this is not an expected phenomenon so some models might not be trained well enough. The writing of the paper needs to be improved greatly. Some paragraphs are hard to follow such as Section 4.1.1 and 4.2.1. There also exist lots of typos that need to be fixed. In the end, I would like to suggest the authors re-draw Figures 5 and 6. It is a little bit hard to distinguish all the shapes and colors without annotations on the graph when the only guidance is the text in the main body.
  - Significance: I think the phenomenon observed in this paper is quite interesting and deserves more research. It might be a good start point for this area if the paper can be updated.
limitations_and_societal_impact: The paper does not discuss the limitations. I think the most critical weakness is the too simple setting of the experiments and lack of deeper analysis. Please also see the main review for other weaknesses.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: Adversarial robustness has received a lot of attention along with the study of adversarial data. So far, almost works have shown that robust classifier not only outperform under comprehensive adversarial attack evaluations but also boost the performance in some downstream tasks. However, the underlying mechanism of adversarial robustness is still not clear.

In this paper, they investigate adversarial robustness from the perspective of linear components. From this very novel perspective, they find that there are some statistical properties for comprehensively robust classifiers. Specifically, robust classifiers show an obvious hierarchical clustering effect on their linear sub-networks. Based on the above investigation, this paper applies this on more tasks, such as robustness boosting and domain adaption.
main_review: Pros:

1. There are rare works to deeply think about the underlying mechanism of adversarial robustness. In this paper, they consider understanding this from the perspective of linear components. From this very novel perspective, they find that there are some statistical properties for comprehensively robust classifiers. Specifically, robust classifiers show an obvious hierarchical clustering effect on their linear sub-networks. This contribution is novel and has a significant impact on the field of adversarial machine learning.

2. They present an intriguing phenomenon of hierarchical clustering effect in robust models and provide a novel yet insightful understanding of adversarial robustness. The clustering effect aligned with class hierarchy demonstrates more semantic and representative feature extraction capacity of robust models, which benefits a lot in various tasks.

3. The paper is technically sound and well written. The proposed solution is based on a very interesting investigation. Figures are very informative. 

4. Besides adversarial-related study, this paper further explores some downstream tasks with the understanding of hierarchical clustering. Experimental results show that the clustering effect and hierarchical classification learned by robust model benefits the task as well.

Cons:

1. The major concerns include the setup of experiments. The domain adaptation is a well-developed area and problem. This paper said their contributions to the domain adaptation field but did not explain details in such experiments. In domain adaptation, we focus the performance on the target domain. It is very confusing what kind of domain adaptation tasks is considered in Table 2. Why is there accuracy on the source domain? This is not the one cared about in the field.

2. I noticed many hyper parameters used in the experiments like dynamically changed learning rate. What are the motivations to design such ways to adjust hyper parameters? 

3. It is unclear why the normalization in Eq. (8) is necessary. More explanations should be given.

4. In the defense part (Table 1), which method does this AT mean? It would be better to see the effectiveness of “+C” with more adversarial training methods. 

--------------------------------------------
My concerns are addressed well. Thus, I will increase my score by 1 point.
limitations_and_societal_impact: The authors have discussed limitations and potential social impacts well.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The paper analyses adversarially robust models in terms of their clustering behavior with respect to the learned weights as well as with respect to the class-wise feature representations. For this purpose, the CNNs are linearized, i.e. non-linearities (ReLUs) are removed from the trained network such that it can be represented by a single weight matrix W, where each column corresponds to one class. The paper comes to the conclusion that hierarchical clustering of these representations is an indicator for adversarial robustness. Consequently, it formulates a regularization term which encourages such clustering behavior. The resulting models are evaluated in terms of adversarial robustness and in the context of domain adaptation. 
main_review: The paper makes an interesting observation in terms of analyzing adversarially robust models. While adversarial robustness is classically hard to grasp besides when evaluating against adversarial attacks, the proposed hierarchical clustering can be directly measured and implemented into a loss in a straight forward way. Therefore, the paper offers in my understanding an interesting new perspective onto the issue. 
That said, several aspects are not clear in the paper. 

From a theoretical perspective, it remains unclear why the analysis of the linearized network should extend to the original network with non-linearities. However, is this correlation holds, I wonder, from a practical perspective, why the feature space analysis is not executed on the linearized network as well. More importantly, in Figure 3, we see the evaluation of several trained models in terms of their linearized weight clustering. However, in Figure 4, only one model is shown in terms of feature clustering. The clustering effect appears to be much weaker here. What would the same evaluation look like for the other models in Fig.3? Also, in Fig 3. what are the accurcies on test/train/val set for the 8 models? Are these in a comparable range for each pair of models?

The experiments show that the additional proposed clustering loss can boost the robustness in terms of accuracy under attack for different attack methods, including the adaptive autoAttack method as well as a black-box attack. Similarly, the results on domain adaptation can be improved. Yet I object to the timing analysis in lines 213, 214. The model with the proposed loss only needs slightly more time to train than the original model with adversarial training. Yet, the entire model has to be fully trained before, in order to determine the clusters to be used in the regularization. Thus, the training time of the adversarially robust model is increased by more than a factor of 2. 
The results in Tab 1 indicate that the accuracy under attack improves with the proposed procedure. Yet, the reported clean accuracy for these models is increased in a similar range (1.2 - 1.3 %) as well. So it is unclear whether the increased accuracy under attack might be caused by better training in general. The drop between clean accuracy and accuracy under attack remains about constant. Therefore, these results are not convincing to me.
The results in the domain adaptation task are more convincing. Yet, they are not well motivated in the context of the paper.
In neither of the evaluations, the proposed approach is compared to the state-of-the-art, but only relative improvements are reported. Would they also hold for the SotA model in terms of adversarial robustness such as:
https://arxiv.org/abs/2010.00467
https://arxiv.org/abs/2002.08619
https://arxiv.org/abs/2011.11164
https://arxiv.org/abs/2002.11242
etc... ?

 
The paper is overall well structured but the language is sometimes hard to understand. (e.g. line 48,49: D_input represents for the dimension...)
 

Minor: 
Intro, line29: if you say "on the other hand", there should be "on the one hand" somewhere before
please define \delta in eq. 4

limitations_and_societal_impact: The limitations and societal impact are not well discussed in the paper. With respect to the societal impact, I would assume that this is tolerable as the paper makes a conceptual contribution to the analysis of adversarial robustness, probably without a direct societal impact. Potential benefit would be, if the model works in practice , that CNNs could be applied in more safety relevant applications.
With respect to the limitations, please refer to the weaknesses I listed above. The model needs a significantly larger training time that the basic adversarially robust model. A comparison to the state of the art is missing and it is unclear whether the relative improvement would translate to a better model.
review_title: null

Meta-review: This paper presents an interesting study of adversarial models that seems novel and creative.
The phenomenon identified for the linear components is intriguing, though it seems it remains speculative to fully understand its implications on the original network. Also, the experimental results could be stronger.
Nevertheless, given the novelty of proposal in an area that has received significant attention, this seems like an interesting contribution. 
Decision: Accept (Spotlight)