Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper studies the robustness of deep neural networks based on the perspective of representation similarity.  Such a perspective provides an interesting direction to delve deeper into the properties of robust representation learning. The authors make several novel discoveries on "salient pitfalls" in robust networks. According to the observations, the author introduces several ways to design and train better robust networks. 
strengths_and_weaknesses: Strength:
1. Analyzing the properties of robustness from the representation similarities is intuitive and has been explored by some previous works. However, the paper provided a very systematic study and provided lots of interesting discoveries. 

2. The paper is well written. The analysis and discussion are conducted in a very logical way. 

3. There are extensive experiments conducted to demonstrate the conclusion and observation. I believe the results are solid. 


Weakness:
1.  I appreciate the efforts in providing different views/frameworks for an important research problem. While I would also like to find more practical effects of the proposed analysis and observations.  

questions: 1.  Is there possible to involve more experimental results on how the proposed ways for better robust learning could improve upon the previous methods?
limitations: yes.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper presents a probing analysis on clean (non-robust) vs. adversarially trained robust models. The paper's novelty is questionable and the insights gained are also sort of obvious, e.g., that the representational differences between inputs increase as one goes deeper down the layers. While this is true (and has been observed by the authors in their experiments), the authors didn't make an attempt to analyze whether two inputs from the same class are do have similar representations, which is a good thing. How does this compare between robust and non-robust models?

The paper introduces a lot of defence mechanisms (mainly centred around the min-max idea). However, it only employs PGD asdversarial training for obtaining a "robust" model. What about other approaches that provide defence mechanisms against potential attacks, e.g. the references 23, 32, 43 etc. that the authors themselves cite?


strengths_and_weaknesses: Strengths:

1. Good analysis work on robust vs. non-robust networks.
2. Uses CKA to measure representational similarities.

Weaknesses:

1. Only one version of robust network considered - those trained with PGD based adversarial examples.
2. The rational of using CKA is not appropriately justified.
3. Some analysis could/should have been at a more detailed level, like one would still want the differences between different classes to be high and similarties between identical classes to be high. What observations can we make regarding this expected behavior in a robust and a non-robust network? 

questions: How do the analysis from the observational differences between robust and non-robust networks can actually be used in practice? Can we use these insights to guide us towards constructing more robust models?
limitations: Limitations are not properly explained. The expression "fundamental disjunct between aggregate properties and layer-wise representation similarity metrics..." is rather vague.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper uses an existing method for comparisons of intermediate neural network activations (CKA) for a comparison of robust and non-robust networks. The authors analyze the similarities in different aspects and try to deduce insights in adversarial training.
strengths_and_weaknesses: - The paper uses an interesting & potentially insightful approach to better understand how adversarially trained networks process information and how they differ from non-robust networks.
- The attached code for this submission is well documented and looks clean, making the results more trustworthy.
- It feels like the paper spends too much time describing plots/results compared to interpreting the results and presenting hypotheses for what results mean.
questions: - Missing related work: There has been previous work on comparing the features of robust vs. non-robust neural networks that should be cited, e.g. using feature visualizations [1].
- L25: Reference missing.
- L45f: Reference [18] should be placed behind Imagewoof and not behind Imagenet; also, a reference for ImageNet is missing. 
- L93: E.g., JPEG compression doesn't seem like an additive perturbation.
- L108: If this is well known, I encourage the authors to support this sentence with (multiple) references.
- L118: A more detailed critical discussion of the possible shortcomings of these metrics and why they don't impact the results of this paper would be good.
- L123f: Why doesn't this impact your results? This definitely needs to be supported by a strong argument. 
- Figure 2/3: On which datasets were these metrics calculated? On the vanilla test data or on adversarially perturbed versions of the test data? What does the figure look like for the other data type (either benign or adversarial)?
- Figure 2/3: Given that the clean accuracy goes down for adversarial training, I'm wondering whether the change in the similarity plots is really due to a special aspect of adversarial training or just due to the drop in performance, i.e. how do these plots look like for non-robust networks that are not trained until convergence but until they reach a similar test accuracy as the robust networks?
- L163f: That seems a bit like an overstatement: The block structure is still clearly visible for 2 out of the 3 datasets - it just gets weaker.
- Multiple typos/grammar mistakes that make it sometimes break the flow of the text, e.g. L172, L187, L201, L216, L221, L235.
- Figure 4: These plots should be larger - the text is hard to read.
- Figure 4: The first sentence of the caption doesn't read right/is difficult to parse.
- L185f: Did the authors observe the same behavior as described by Nguyen et al. when they tried to reproduce their observations for non-robust networks? At the moment it is difficult to confidently say that the observation reported here is because of the robust network or because of the experimental setup used by the authors.
- L213: If this is well known, I again encourage the authors to support this sentence with (multiple) references. Furthermore, this is actually not such a clear property, and there are specific attacks that just aim to create adversarial examples that transfer well between different models.
- Minor comment: The paper uses inconsistent notation - sometimes the authors say "Figure", sometimes "figure"; sometimes "Appendix", sometimes "App.".
- L252f: Where can we see both validation and training loss? The figure only shows one "loss" but doesn't say which one it is.
- L265: I wouldn't call these "adversarial perturbations" but rather use the more commonly used expression common corruptions. Especially, since at least for JPEG compression there is nothing you can optimize - so that doesn't really fit the overall adversarial framework.
- Section 6: What are the specific parameters of the JPEG, snow, and Gabor corruptions?
- L289: What is the conclusion/interpretation of the results?

[1] Leveraging Sparse Linear Layers for Debuggable Deep Networks. Eric Wong, Shibani Santurkar, Aleksander Mądry. 2021
limitations: - While the authors mentioned criticism on the similarity metric they use (CKA), they don't really explain why this doesn't apply to their analysis. This should be properly addressed.
- It is also unclear whether the results differ if a different adversarial attack (with the same thread model) was used for generating the adversarial perturbations - this should ideally also be addressed.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors examine the effects of adversarial robustness training on representation. Specifically, they use CKA to compare robust vs non-robust networks, benign vs. adversarial inputs, and how these comparisons change over the course of learning. They find the following results:
- Robust representations are less specialized; distant layers are more similar in robust networks, and block structure is weaker
- Early layers in robust networks are largely unaffected by adversarial examples; representations are similar for benign vs. perturbed inputs
- Deeper layers overfit during robust learning
- Models trained to be robust to different threat models have similar representations
strengths_and_weaknesses: Strengths: The paper follows in a well-established tradition of using representational similarity analysis to understand neural network behavior and training interventions. The analyses are well-motivated and straightforward. The results are generally presented clearly and easy to understand. The results are interesting.

Weaknesses: The paper seems somewhat limited in scope: It primarily addresses adversarial (worst-case) robustness, which is only one type of robustness. I'm not totally convinced of the utility of this work; it could at the very least do a better job situating itself within existing robustness research. It's unclear if any of the experiments were run in replicate. The figures captions could be more informative and self-contained.

Overall, I think this work could be suitable for publication if it is sufficiently revised.
questions: This work primarily examines adversarial (i.e. worst-case) robustness, which is one of many types of robustness, among others including average-case robustness (Hendrycks and Dietterich, Benchmarking Neural Network Robustness to Common Corruptions and Perturbations) and natural adversarial examples (Hendrycks et al.). It’s not clear how important of a problem adversarial robustness is (see Gilmer et al.’s Motivating the Rules of the Game for Adversarial Example Research). The authors examine robustness to Gabor, snow, and jpeg attacks, but the analysis and results are limited. I would encourage the researchers to more systematically examine how their findings extend to other types of robustness.

Are the results from a single instance of each model? If so, I would strongly encourage the authors to repeat their analyses in triplicate at a minimum. If this is computationally cost-prohibitive, then perhaps they could focus on the most important results.

There is relevant previous work examining the relationship between representational dimensionality and adversarial robustness (Leavitt and Morcos, Linking average- and worst-case perturbation robustness via class selectivity and dimensionality; Sanyal et al., Robustness vai Deep Low-rank Representations; Nayebi and Ganguli, Biologically inspired protection of deep networks from adversarial attacks). These papers show that low rank representations confer adversarial robustness. Can you reconcile these results with your findings that adversarial robustness is associated with greater utilization of model capacity? Additionally, most of these works (certainly Leavitt and Morcos and Sanyal et al.) regularized representational dimensionality and found that it improved robustness; it would be quite interesting to investigate whether that the inverse holds: does adversarial training reduce representational dimensionality?

The claim that “Deeper layers overfit during robust learning” seems like it could be leveraged to improve the generalization gap caused by robust learning. The simplest approach would be to simply stop training when the adversarial loss begins to rise (as in Figure 6). I assume this has drawbacks, but the authors don’t present the data (e.g. accuracy curves over training). Is this overfitting a necessary component of robust learning?

Lines 59-61: “On the other hand, the representations of benign and perturbed inputs from robust networks are indistinguishable from one another with regards to representation similarity metrics.” I think to make this claim the authors need to directly compare CKA(Benign, Perturbed) to CKA(Benign, Benign) and CKA(Perturbed, Perturbed).

The caption for Figure 2 should describe the model architecture(s) used to generate the results, as well as whether the data are benign or perturbed. There are other Figures which lack important experimental details, such as whether the data are benign or perturbed.

Comparing the CKA results in Figure 2d to the results in Figures 2a-c make it clear that the similarity between robust and non-robust networks is lower than the similarity between networks of the same type, but I think it’s important to have a negative baseline—how low is “low”? Accordingly, I think the authors should repeat the analysis (CKA between robust and non-robust networks) using sample-shuffled data (and/or some other suitable random baseline).

Showing that the block structure effect varies with the strength of adversarial training is a nice experiment and result.

The results presented in lines 174-188 (“Impact of robust training strength” and “Impact of architecture”) would be easier to interpret (and their motivation clearer) if you introduced them with Nguyen et al.’s finding that “block structure in the internal representations arises in models that are heavily overparameterized relative to the training dataset.” While you do cite their work (“Previously Nguyen et al. [28] observed that increasing network width, thus capacity, leads to emergence of block-structure in non-robust networks”), I think this is an insufficient description of their results and should be presented earlier.

“Overfitting is predominantly visible in later layers”: I would suggest a follow-up experiment to more thoroughly test this claim: freeze early layers (blocks 1 and/or 2) early in training. If early layers aren’t involved in the overfitting and training instability, then the freezing should have minimal effect on these phenomena.

Lines 278-280: “When using CKA to compare against the Snow threat model, we observe that the highest average similarity is achieved with Gabor. This represents a novel insight into these threat classes, as correspondence between the two was not previously known.” This result seems very specific and not particularly useful. I would encourage the authors to find a more general result (see my earlier comment about examining different attack types).

minor comments:

Lines 21-22: “...such as images, speech or text”. It’s a matter of personal taste, but I am a proponent of the Oxford comma: “...such as images, speech, or text”

Line 22: What does “meaningful” mean in this context? Meaningful with regards to what?

Line 187: “These results suggest that while increasing width in robust networks doesn’t lead a drastic shift in similarity of internal layer representations.” Typo?

Typo in caption of Figure 5 (...to understand similairty)

limitations: The authors devote a paragraph at the end of the discussion to the limitations of their study, which seems appropriate given the space limitations. I do, however, think the authors could be more careful and nuanced about some of their claims.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper contrasts representation similarities of networks trained to perform image classification with and without adversarial noise. To do so, the authors measure the similarity of representations using the Centered Kernel Alignment (CKA) metric (as well two other similarity metrics in appendix) for CIFAR-10 and two subsets of ImageNet. The authors highlight 1) networks trained with adversarial noise have layers similar to one another compared to those of standard trained networks (which have a block structure) 2) representations in early layers are unaffected by adversarial perturbations both for standard and adversarially trained networks 3) networks trained with and without adversarial perturbations have similar representations until the last 10 or so layers 4) early layers converge faster and later layers overfit to local minima. The authors also analyze the similarity of representations when other perturbations are applied JPEG, Gabor, and Snow, finding 
strengths_and_weaknesses: This work investigates representations learned for image classification by contrasting the representation similarities of networks trained with and without adversarial noise—leading to several insights into properties of the learned representations as well as learning dynamics of modern networks. Such properties as the authors point out are not captured by aggregate performance metrics such as loss or accuracy, leading to insights about the learned representations. The paper is well-written, experiments clearly described, and motivation is well-grounded.

The experiments conducted are convincing and well-formulated. For example, multiple similarity metrics are compared, variants of the adversarial perturbations are explored, and experimental claims are well-founded. 
However, the authors'claims should be sufficiently couched within the experimental settings studied: supervised image classification for ResNet-based models on CIFAR-10 and two subsets of ImageNet. For example, the work only studies ResNet-based architectures yet claims to cover “DNNs with different architectures” (line 12) and the “impact of choice of architecture” (line 37). I would expect a comparison of “different architectures” to encompass for example transformer-based architectures, MLP-architectures, etc. I suggest the authors more explicitly couch claims in the introduction, abstract, title, and conclusions within the confines of the experimental settings studied. For example 
The comparison of representations for "threat models" (JPEG, Gabor, and Snow) in Section 6 was not particularly informative and seemed removed from the primary findings of the remainder of the paper. 

Robust representations is a broad term. This work studies robustness to random noise in the input. Yet, the authors intermix “robustness to adversarial examples” and often plainly use the term “robust network” to describe a particular type: robustness to noise (defined via L-p bounds on the input) throughout the work. For example, robust in the context of image classification can just as well refer to robustness with respect to rendering method ImageNet-Sketch, artifacts such as blurring (ImageNet-C), adversarial examples (ImageNet-A), or even robustness to natural transformations such as pose (Alcorn et al.). I suggest the authors clarify the wording to only include the specific definitions of robustness studied here.

The insights gained from the authors’ analysis are interesting and well-described. The finding of most value, in my opinion, is that later layers overfit (matching existing work relating overfitting in later layers to spurious correlations [1]). While the analysis sheds light on differences between networks trained with and without adversarial noise as well as their learning dynamics the findings are confined to the L-infinity definition of robustness studied for supervised image classification using ResNet-based architectures (for the main claims in section 4 and 5).

[1] “Last Layer Re-Training is Sufficient for Robustness to Spurious Correlations” [https://arxiv.org/abs/2204.02937](https://arxiv.org/abs/2204.02937)

questions: included above
limitations: Yes.
review_title: null

Meta-review: metareview: The authors study representations obtained from image classifiers and contrast the classic training with adversarial training, so-called non-robust and robust networks, respectively. The authors primarily use the CKA metric on CIFAR10 and subsets of ImageNet2012 provide several novel insights on "salient pitfalls" in robust networks which suggest that robust representations are less specialized with a weaker block structure, early layers in robust networks are largely unaffected by adversarial examples as the representations seem similar for benign vs. perturbed inputs, deeper layers overfit during robust learning, and that models trained to be robust to different threat models have similar representations.

The reviewers agreed that these contributions are interesting to the larger community and that the presentation of the results is clear and straightforward. The main issues raised by the reviewers were carefully addressed in the rebuttal. Please update the manuscript as discussed.
confidence: Certain
Decision: Accept