Understanding Robust Learning through the Lens of Representation Similarities
Authors: Christian Cianfarani
Arjun Nitin Bhagoji
Vikash Sehwag
Ben Zhao
Haitao Zheng
Prateek Mittal
Venue: NeurIPS 2022 Accept (2022)
Keywords: representation similarities
robust training
visualizations

Abstract: Representation learning, \textit{i.e.} the generation of representations useful for downstream applications, is a task of fundamental importance that underlies much of the success of deep neural networks (DNNs). Recently, \emph{robustness to adversarial examples} has emerged as a desirable property for DNNs, spurring the development of robust training methods that account for adversarial
examples. In this paper, we aim to understand how the properties of representations learned by robust training differ from those obtained from standard, non-robust training. This is critical to diagnosing numerous salient pitfalls in robust networks, such as, degradation of performance on benign inputs, poor generalization of robustness, and increase in over-fitting. We utilize a powerful set of tools known as representation similarity metrics, across 3 vision datasets, to obtain layer-wise comparisons between robust and non-robust DNNs with different architectures, training procedures and adversarial constraints. Our experiments highlight hitherto unseen properties of robust representations that we posit underlie the behavioral differences of robust networks. We discover a lack of specialization in robust networks' representations along with a disappearance of `block structure'. We also find overfitting during robust training largely impacts deeper layers. These, along with other findings, suggest ways forward for the design and training of better robust networks.

Understanding Robust Learning through the Lens of Representation Similarities
Christian Cianfarani"
Arjun Nitin Bhagoji*
Vikash Sehwag" Department of ECE
Department of Computer Science University of Chicago
Department of Computer Science
University of Chicago
Princeton University
Ben Zhao
Haitao Zheng
Prateek Mittal
Department of Computer Science University of Chicago
Department of Computer Science University of Chicago
Department of ECE Princeton University
Abstract
Representation learning, iee. the generation of representations useful for down- stream applications, is task of fundamental importance that underlies much of the success of deep neural networks (DNNs). Recently, robustness to adversarial desirable property for DNNs, spurring the development
examples has emerged of robust training methods that account for adversarial examples. In this paper, we aim to understand how the properties of representations learned by robust training differ from those obtained from standard, non-robust training. This is critical to diagnosing numerous salient pitfalls robust networks, such as, degradation of performance on benign inputs, poor generalization of robustness, and increase in over-fitting. We utilize a powerful set of tools known as representation similarity metrics, across three vision datasets, to obtain layer-wise comparisons between robust and non-robust DNNs with different training procedures, architectural pa- rameters and adversarial constraints. Our experiments highlight hitherto unseen properties robust representations that we posit underlie the behavioral differ- ences of robust networks. We discover a lack of specialization in robust networks' representations along with a disappearance of 'block structure'. We also find overfitting during robust training largely impacts deeper layers. These, along with other findings, suggest ways forward for the design and training of better robust networks.
Introduction
Representation learning is fundamental to machine learning and task which deep neural networks perform remarkably well. Given complex high-dimensional input signals, such as images, speech, or text, deep neural networks (DNNs) can learn meaningful, lower-dimensional representations [33] The downstream utility of these representations underlies DNNs success For standard image classification, neural networks learn well- separated representations of inputs from different classes to achieve high accuracy. The discovery of adversarial examples 4 perturbed inputs that induce high error in model predictions (even for well-trained classifiers) has motivated the need for another desirable property: adversarial robustness. Robust training methods typically protect against adversarial examples during the training phase by converting the standard empirical risk minimization objective to min-max one (Figure 46 65). However, they typically have far worse performance on benign inputs, converge
6 57
66
different tasks 5
5
3
'Equal Contribution
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Given the importance of representation learning for good downstream performance, we posit that an effective method to identify potential improvements for robust training is via developing a better understanding of robustly learned representations
slower and appear to require larger architectures to be effective 17 38 56 60)
Our approach is to juxtapose representations of non-robust and robust networks and analyze them through unified lens. Aggregate metries like loss and accuracy do shed some light on the differences, but these are task-driven and do not admit a layer-wise comparison of representations I 20). We need a comprehensive analysis of the impact of choice of architectural parameters like depth and width, nature of the threat model and corresponding attack strength, and the choice of dataset on robustly learned representations and their link downstream performance.
To this end, we leverage recent work on representation similiarity (RS) that enables fine-grained comparisons of representations obtained across all layers in a deep neural network (DNN) B1 --1 Our key findings, using both benign and adversarial inputs as probes over CIFAR-I0 10 B2] and the Imagenette and Imagewoof [2 subsets of the Imagenet [TC] dataset, are:
Robust representations are less specialized 33: Robust networks exhibit much higher similarity across spatially distant layers compared non-robust networks, which have far more localized similarity. The latter thus exhibit the expected specialization in layers, which is unexpect- edly missing in the former. Further, we find reducing the width of the network has similar effect to increas- ing the budget used in robust training, with both changes increasing long-range similarity among the learned repre- sentations. This indicates that the relative capacity of a network drops as the budget increased.
Non-robust classification
Robust classification
Figure 1: Non-robust vs robust train- ing. Adversarially robust training aims learn decision boundaries that are robust to adversarial perturbations (foc this sketch), naturally learning more complex decision boundaries than non- robust training.
Early layers are largely unaffected by adversarial examples ( The representations of perturbed inputs from non-robust and robust networks are, expected, starkly different. On the other hand, the representations of benign and perturbed inputs from robust networks are indistinguishable from one another with regards to representation similarity metries.
Deeper layers overfit (55): For both non-robust and robust training, early layers the network converge within the first few epochs, and exhibit very little variation in subsequent training. This indicates they perform similar roles in both benign and robust networks. Deeper layers converge slower while visiting drastically different representations over the course of training. These different representations usually occur when the loss indicates overfitting, leading us to conclude that deeper layers overfit during robust training.
Visually dissimilar adversarial constraints can lead to similar robust representations ( Re- gardless of the adversarial constraint, we find high degree of similarity between the representations of perturbed and benign inputs for robust networks. Representations from networks robust to different, but allied (e.g. t,). threat models are similar across attack strength. We find surprising similarities between the representations of visually unrelated threat models at lower attack strengths.
What do our findings imply? The lack of specialization in robust networks even with heavily overparamettrized architectures suggests a need for explicit tools differentiate between layers during training. Further, the overfitting we find in deeper layers of neural networks is an indication that layer- wise specialized regularization methods need be adopted for robust training. We hope the lessons from this paper and our accompanying website f with open- open-sourced code spur the development of robust coorniing-pecifiee architectures and training methods.
Paper Layout. We provide an overview of RS tools and robust training methods in 2 representations from non-robust and robust networks using benign inputs in $
We compare repeats this
We note that we use word 'robust' as shorthand 'adversarially robust' throughout the paper. When considering other types robustness, such as to common corruptions, we explicitly mention it. https: https://robustrs.o wit.hb.tthoooo.o

---

Network
Benign accuracy
Robust accuracy
Train
Test
Train
Test
A
Non-robust
100.0%
94.9%
5.1% 1%
0.0% 98.2%
0.0%
0.0%
Robust
100.0%
87.6%
12.4%
44.2%
54.0%
Table 1:
Performance of non-robust and robust Wide Resnet 28-10 (WRN-28- 10) on the CIFAR-10 dataset. Note, in particular, the high generalization gap in robust networks.
analysis with perturbed inputs. The evolution of robust representations over training is studied in while the impact of different threat models is in $ discusses lessons and limitations.
2 Background and Setup
We provide brief overview of the techniques we use to train robust networks and the tools we use to probe the intermediate representations of neural networks. Further background and related work is in Appendix A
2.1 Adversarial Examples and Robust Learning
Adversarial examples are perturbed inputs to machine learning models that intentionally induce misclassification [4] [54] For an input x, an untargeted adversarial example with respect to model f and loss function ((-) generated by adding a perturbation 6 such that ô arg maxx+6EN {xf y). where y the label of the original class and N-(-) represents the adversarial constraint or neighborhood. The most commonly used constraints are Lp-ball based, although we also experiment with other unrelated classes of attacks, such as JPEG, Gabor, or Snow based perturbations (28). We characterize activations of internal layer in a network as: 1) Benign representation: if input is non-adversarial 2) Adversarial representation: input an adversarial example.
Adversarial training 355 has emerged as the most effective methods to train robust classifiers where accounts for adversarial examples during training. It modifies the empirical risk minimization procedure in non-robust training to min-ma max optimization, where the inner maximization focuses on finding the strongest adversarial example for each input during training:
max e(fo(x + 5), (x)
min E (x.))ED
(1)
where D a data distribution and fo is a model with parameters 9. Newer methods, like TRADES 1651, attempt to improve adversarial training by better balancing the tradeoff between robustness and accuracy. We refer to models trained using methods that account for adversarial examples as robust models and those trained without these methods non-robust models.
We refer to accuracy on pontterurde input as benign accuracy and on adversarial examples as robust accuracy. Adversarial training is well known to trade-off benign accuracy to gain robustness and exhibits a high generalization gap in robust accuracy (as we show in Table T for the CIFAR-10 dataset).
2.2 Probing Neural Network Representations
Representation similarity (RS) metrics allow for the quantitative comparison of the activations of different sets of neurons on a given dataset. Comparing learned, internal representations within and between neural networks of different architectures is important for understanding the performance and behavior of these networks under changing conditions. Several metrics have been proposed for this purpose: Canonical Correlation Analysis (CCA) [22] and its extensions, Singular Vector CCA
(SVCCA) 44 and Projection Weighted CCA (PWCCA) EV Centered Kernel Alignment (CKA)
These metrics can handle different sized representations, but require that the number of datapoints be the same. Other work has used the fitting of sparse linear layers to understand the behavior of deeper layer representations
BI the Orthogonal Procrustes distance 14] and model stitching 2 g
559 47
3

---

We choose CKA to measure representation similarity, due to its desirable properties such as orthogonal and isotropic invariance, and wide use in previous work 20
581 We also find CKA to be computationally more efficient (10x and 30 faster than Procrustes and SVCCA respectively). while yielding similar high-level insights (see Appendix [.T]. D. Ding et [14] have shown that CKA is insensitive to deletions of principal components of representations that strongly correlate with
12
+
model accuracy. However, since we do not compare neural networks with representations of reduced dimensions, this does not affect our results.
Centred Kernel Alignment (CKA): Let X E R with rows z;, and Y IR=xp2 with rows y be the activation matrices of layers with p1 and p2 neurons respectively. defined on the same set of n points. These activation matrices are then the representations induced by particular layer. We also define a n X n centering matrix H In 11 The CKA between these activation matrices is then
ir WKHLH Ir [HHH H H)LHHH(HHHHH) (HLLHHH)))
CKA(K,L)
where Kú; k( 2z)) Lyj Iyy yj), with is and I being kernels. When the kernels are linear and the activation matrices are centered, we get the Linear CKA between activation matrices to be
Y X
(vec( XXT), (YC X X Y
Linear CKA = |X X |
(3)
Since Kornblith et al. (31 find the use of a linear kernel to be as accurate as and much faster than the RBF kernel, we focus only linear kernels. To reduce memory usage, we use online CKA which uses batching to get an unbiased estimate of CKA 42
2.3 Experimental Setup
We provide brief overview of our experimental setup here. All of experiments were run machines with either NVidia Titan RTX GPUs with 24GB of memory or RTX A4000 GPUs with 16GB of memory. Further details are in Appendix A
Models and datasets. We consider three commonly used image datasets: CIFAR-10 32 Ima- geNette [5] and Image (25], where the latter two are subsets of ImageNet [I], which we use due to the high cost of adversarial training on the full ImageNet dataset. We use the Wide Resnet [63 class of models on CIFAR-I0 dataset and ImageNet-based ResNets for ImageNette and Imagewoof datasets.
Adversarial training. We follow standard convention with Eo perturbations and use = CIFAR-10 10 and 4 253 for ImageNet based datasets. We use 10-step projected gradient descent (PGD) attack during training and use steps at test time to calculate adversarial representations.
255 for
Representation similarity. We will be using mainly using CKA to compare activation matrices derived from different benign adversarial images. In online CKA, we use batch-size of 1024 and take 3 passes over the dataset to reduce any stochasticity in the output similarity score. In layer-wise similarity analyses, we consider all layers, including convolutions, pooling, activation, and batch-normalization layers.
3 How do representations from robust and non-robust networks differ?
Robust training is expected to be harder training objective than non-robust training, since it optimizes for both accuracy and robustness. The resulting trade-off is also reflected in final performance, where the benign accuracy of robust networks is much lower than non-robust networks [35]. this section, we aim to understand the impact of robust training internal representations, in particular by comparing benign representations in non-robust and robust networks. Further experiments and results from this section included in Appendix We take the following two-fold approach:
1) Comparing characteristies: Absence of block structure. When comparing layerwise similarity in non-robust networks, numerous works have revealed 'block structure', with high similarity between the layers comprising block, and low similarity outside
31
42 25 (Non-robust network
plots in Figure 2 We observe that when moving from benign to robust training, while keeping

---

Non-robust Non-robus.
Robust Robust
Nonurabust Non Noon-robust
Robust Robust
1.0
0.8
100
Layer
0.6
75
0.6
0.4
50
0.4
0.2
25
0.2
0
0.0
0.0
30
45
60
75
30
45
60
75
25
50
75
100
25
50
75
100
Layer
Layer
(a) CIFAR-I0 (WRN-22 (WRN-28-10)
(h) ImageNette (ResNet-50)
Non-robust
100
CIT48-16 06704-8-100
(2arki4-22)
teagsWiol (aadiidd
Layer
9
22
13
140
100
50
75
100
25
50
75
100
Laper
Layer
(d) Similarity between representations of non-robust and robust models.
(c) Image Woof (ResNet-50)
Figure 2: Stark differences in the cross-layer similarity and block structure for robust networks. We measure the similarity of benign representations for three datasets across all pairs of layers in both non-robust and robust networks. b, c): Robust networks have amplified degree of cross- layer similarity, even at long-range, which diminishes the well known block structure characteristic observed in non-robust networks [2 (d) We observe low similarity between representations of non-robust and robust models with identical experimental setups.
everything else constant, the block structure significantly fades. The much weaker block structure in robustly trained networks is accompanied by the presence of tong-range similarities between layers, with layers having high similarity those much farther away the network (Figure 2
2) Direct comparison: Poor similarity between non-robust and robust network representations. To further investigate the unique characteristics of robust networks, conduct cross-layer compar- ison of benign representations in non-robust and robust networks (Figure 2d). We find very low degree of similarity across both network representations, which further shows that robust networks had indeed learned strikingly different representations than non-robust networks.
We also show that both of these observations generalize across architectures, datasets and threat models (see Appendix B). We now delve deeper into analyzing cross-layer charucteristics of robust networks representations.
Impact of robust training strength. We investigate this disappearance of block structure in robustly trained networks further by varying both the adversarial budget (e) used for training and the width of the network (Figure Bp. As the training increases, the block structure becomes fainter, eventually disappearing entirely. The block structure is also impacted by network width, with it being marginally more pronounced in larger networks. However, this distinction is largely absent at larger values of maintaining This establishes that increasing the value of is making the task more complex, the same model capacity leads decrease in relative capacity (indicated by high similarity across representations). This conclusion relates to several findings in the adversarial robustness literature on the topic of low-rank representations. Encouraging low dimensionality of learned representations through regularization has been found to increase adversarial robustness 40
While it is outside the scope of this work, it would be interesting for future work to study if the task of learning low-dimensional robust representations is inherently more complex, leading to higher utilization of model capacity and thereby reducing differentiation between layers.
34 39]
Impact of architecture. Previously Nguyen et al. 42 observed that increasing network width, thus capacity, leads to emergence block-structure in non-robust networks. Thus we incrementally increase the width of robust Wide-ResNe architecture on CIFAR-10 dataset and measure the cross- layer representation similarity of benign features (Figure Sb. Even after increasing network capacity by an order of magnitude, we do not significant variation in the characteristies of robust network

---

WIT-25-1,
2.0
wm--2-.- wIT-28-1, 20---28--1. 16.0
60
1.0
WTT-26-2,
wm-28-, wm-20-2,
wm--002,
40
0.
0.6
Lager mndex
0.6
40
62
0.0
75-
45
30
Layer index
Figure 3: Delving deeper into cross-layer similarity. To better understand the unique cross-layer characteristics in benign representations of robust networks (Figure 2. we ablate along two main design choices: 1) Strength of adversarial attacks in robust training (by increasing perturbation budget (()) and 2) Changing network size (by increasing network width). We observe that increasing the strength of adversarial perturbations in training predominantly leads to amplification of cross-layer similarity, i.e., higher degree of brightness in the plots. Increasing network capacity, even by order of magnitude, does not substantially change this phenomenon. We used Wide-ResNet architectures with the CIFAR- dataset for this experiment.
representations. These results suggest that increasing width in robust networks doesn't 't lead to drastic shift similarity of internal layer representations.
Probing the impact of adversarial perturbations on representations
Robust training aims to learn both high quality benign and robust representations. In the previous section we characterized the impact robust training on benign representations. In this section, we focus understanding the characteristics of adversarial representations in robust networks. Detailed results, as well as ablations over architecture and threat model, are in Appendix [ We first analyze how adversarial perturbations distort the internal representations compare adversarial and benign representations. Next, we analyze whether the distortions introduced by adversarial perturbations also provide insights into why such perturbations transfer between different networks.
go deeper into the network, iee.
4.1 Impact of adversarial perturbations across layers non-robust and robust networks
In this experiment, measure the CKA similarity between benign image representations and corresponding adversarial image representations.
Early layers are the least affected. Adversarial perturbations are crafted to corrupt final layer representations. In our experiments, expected, the cross-layer similarity score decays near zero by the final layer (Figure representations share a high degree of similarity with a fraction (up to one-third for ImageNette and Imagewoof datasets) of early layers. We find this observation holds across architectures and budgets, indicating that earlier layers representations, even benign networks, are robust adversarial perturbations.
However, we observe another surprising trend in which adversarial

---

1.0
1.0
1.0
0.8
0.6
0.6
0.6
0.2 Twww
0.2
0.2
rolsutt
120
Layer index
Layer index
Layer index
(a) CIFAR-I0
(b) ImageNette
(c) ImageWoof
Figure 4: Divergence of adversarial representations from benign ones. We observe how adversarial perturbations, which are known heavily impact model output, distort the internal representations in comparison to non-adversarial representations. we compare adversarial representations from corresponding layers in non-robust and robust networks. As expected, adversarial and benign representations in non-robust networks display high dissimilarity by the end of network. Unexpectedly, such divergence is very small for non-trivial fraction of early layers (up one-third). Robust training successfully reduces the impact of adversarial perturbations, thus leading to similar benign and adversarial representations in robust networks.
Robust networks successfully prevent the distortion of internal representations. For robustly trained networks, benign and robust representations maintain high degree of similarity throughout the network, with divergence only occurring the last 10 layers or so. In our ablation on network size (Figure 20 in Appendix C we see that increasing the size of the network leads to increased differentiation, implying sufficient capacity for increased separability between benign and perturbed representations.
4.2 Understanding transferability through the lens of internal representation similarity
1.00
0.79
24t 86
0.53
0.50
8.00
wor 2011
0.25
40 Layoe Irdex
Laper Endex
Layer Zidex
Thogec (Rcbuureu witeo
(b) Non-robust WRN-28- (c) Robust WRN-28-10 10 (d) Robust WRN-28-10
(a) Transfer accuracy of ad- 10 vs. non-robust Wide and other robust Resnets.
Wide and other non-robust Wide
versarial examples.
Resnets.
Resnets
Figure 5: Analyzing cross-architectur representations to understand similarity. (a) Measuring the robust accuracy adversarial examples transferred between robust (R) and non-robust (NR) Wide-ResNer networks of varying width (e.g., R-5 refers Robust WRN-28-5), Low robust accuracy implies high transferability. (b, c, d) We measure CKA between source and target networks, where extracted using adversarial examples generated for the source network.
these representations When both networks are trained in the same fashion (benign or robust), the layer wise similarity is generally high at earlier layers but drops off towards the end. In contrast, it drops right after the first. layer from robust to non-robust network, set of networks which also have least transferability.
It is well-known that adversarial examples are transferable from one architecture to another, i.c.. adversarial examples generated on one network achieve the misclassification objective on others too. In Figure 5a we measure the transferability between non-robust and robust networks. Intriguingly, as previously validated by M. transferability higher among some networks (non-robust to non-robust) while much poorer for others (robust to non-robust). High transferability indicates that the target network has learned a similar decision boundary to source network. We aim to better understand this phenomenon by comparing the internal representations of source and target networks. We first generate adversarial examples, and their corresponding internal layer representations, from source network (WRN-25 WVRN-28-100 10 in this case). We compare them with internal layer representations of these
7

---

Robust Model
Midel
Trades Model
Neaslekoot Lze Oratestaion
Tehoot Tasr ('attatsion)
Nax bubent (Tress)
1.0
1.0
1.0
0.69
0.6
0.0
02
Block
Block ]
0.2
Ang.
Epoch
Figure 6: Variation in convergence of different layers. We compare representations from each epoch of training with the final learned representations of network across both non-robust and robust (both PGD-based and TRADES) training techniques. We observe that representations obtained from the output of the first two blocks converge to their final state very early in the training process and have very little variation in both non-robust and robust network. We also find that later layers exhibit more complex behavior, with signs of overfitting when early stopping is not used, and lack of stability from one epoch the next. CKA values are averaged over three training runs with different random initializations.
adversarial examples on the target network. We consider transfer from non-robust to robust, robust to robust, and robust non-robust networks (Figure 5
For transfer from non-robust to non-robust or robust-robust, where highest transferability occurs, we observe high degree of similarity between representations early layers. It indicates that the transferred adversarial examples distort the internal representations in a similar fashion as source network. The most intriguing aspect in Figure 5a is extremely poor transferability from robust to non-robust networks. We establish that this is due to significant differences in representations where, even across identical architectures, shows that adversarial perturbations from robust nework distorts the internal representations of non-robust network in strikingly different manner (even very early layers), which likely renders them ineffective to cause an adversarial effect at the output of the network.
observe large drop in CKA right after input layer.
5 Evolution of internal representation dynamics during robust training
In this section, we track the evolution of representations learned by robust training methods over course of training epochs in the optimization process. We study the following two convergence properties: 1) Rate of convergence at different layers, and 2) Stability of different layers throughout learning. As earlier sections, we use CKA to measure similarity, but in this section we compare the benign representationn of the same layer across time. We present results for Wide-Rest ResNet architecture in Figure 6, where we compare the representations at each epoch to the final epoch (epoch 100). We further investigate across architectures, layers, and attack budgets in Appendix D
Early layers converge faster than later layers. Benign representations after the first two blocks of WRN-28- 1 O are highly similar to the representation obtained at the end of training, after just ten epochs (Figure G). This is true for all the training methods considered. Appendix D we show this largely holds true across architectures and smaller budgets. Our observation relates the intuition that early layers learn simpler features 33
62 4 thus are learned much faster than complex feature 16 in Fig. 26). i.e., stronger attacks, we learned by later layers. Only with high budget (see 255
observe a deviation from this trend where second block convergence is slower. We further validate this with layer-freezing experiments in Appendix D.5
'Robust networks are known to be highly resilient to adversarial examples generated from non-robust networks (see Figure Sa).

---

f
15. JPBG
1- Gabor
VS. Show
8.0
800
80
60
00-
60
00
40-
60-
1.0
20
20
20
20
20
0.8
20
20
20
6.
0.6
Snow VS. 1
Show VS. F.
Snow 1%.. JDE
Snrow vis. Gaabor
Scow Vs. Snow
9.4
20-
60-
60-
60
60-
0.2
0.0
20
20
20
20
20
20 40
60 20 60
60
80
20
40
60
60
20
60
80
20 60
Denign Input Layer
Figure 7: Layer-wise similarity plots between different threat models. The top row shows comparisons between all threat models and 1 and the second row shows comparisons between all threat models and Snow. The Ep threat models are similar to each other across layers. Among the other threat models, Snow and Gabor display the highest similarity, which is surprising due to large differences in their visual representations.
Overfitting predominantly visible in later layers. From the cross-epoch heatmap of similarities between adversarial representations (Figure 25). we observe that later layer representations only exhibit similarity off-diagonal at the beginning (until epoch 30) and end of training. This clearly shows that multiple local minima are found during training. This is true to a less pronounced degree for benign representations. In Fig. C we also observe the adversarial validation loss increasing after around 30 epochs, while the training loss keeps decreasing, indicating overfitting, After this point in training. the CKA similarity between drops with respect to both the epoch 30 and final representation drops significantly in later layers, while the earlier layers remain unaffected. This strongly indicates that the later layers have overfit to the training data. While it is already known that adversarial loss induces overfitting at the output layer 48 our results show that this degree of overfitting is not present in all layers of the network. This finding supports previous work which shows that retraining just the final layer of a deep network can greatly increase its robustness spurious correlations in training data 29 We hypothesize that a layer-by-layer analysis of overfitting in networks may be useful in fine-tuning robust networks.
Adversarial training reduces stability of internal representations during training. While sub- sequent epoch representations tend to be very similar to each other in non-robust training, there there can be very large jumps in similarity between one epoch and the next, indicating a less stable convergence. This effect further accentuates when training with higher perturbation budget when training a larger network (Appendix D).
6 How does the threat model impact representations?
Throughout the paper, we have focused on single threat model: Exo perturbations. In this section, we extend our analysis of robust representations to understand what exactly the link is between the visual similarity of threat models and the representations of models trained to be robust to them. To this end, we consider four additional threat models: f2. JPEG, Gabor and Snow [28] (visualized in Figure We also examine the generalizability of our earlier results across these threat models. For space reasons, all associated figures from this section are in Appendix
Do visually similar attacks lead to similar robust representations? Representation similarity gives us method by which we can compare how different classes of adversarial perturbations affect learned representations. Using CKA, we plotted the cross-layer similarity of robust WRN-28- 10 networks adversarially trained against five different threat models (fz, lxx. JPEG, Gabor, and Snow) (Figure 7). These comparisons lend credence to conventional knowledge about these threat models. The layerwise similarity plot between l2 and loo robust models is strikingly similar CKA plots between robust models trained the same threat model. This mirrors theoretical results correspondences

---

between lp threat models 1551. We also observe higher average similarity when comparing an loc model with a JPEG model than with Snow or Gabor models. This makes intuitive sense, given the so constraint inherent in the JPEG attack. When using CKA to compare against the Snow threat model, we observe that the highest average similarity achieved with Gabor. This represents a novel insight into these threat classes, as correspondence between the two was not previously known, and may indicate the possibility of models jointly robust to these threat models.
Alignment of budgets across different attacks. CKA can also give insight into the varying effects of changing perturbation strengths within different threat models. Expanding on our experiments above (all figures are in Appendix E). we compared representations between different threat models while varying perturbation strengths. Between &oe and Gabor models, the layerwise similarity structure is much more sensitive to changes in the strength used in -robuststt training than in Gabor-robust training. For more closely related attacks, like t<o and JPEG, we see similar degradations of the structure when changing the strength of either attack. Furthermore, the structure is largely preserved when the attack strengths are changed in tandem. These results may be useful for developing techniques for training models to be jointly-robust against different classes of attack.
Layerwise structure across threat models. We find that across all threat models except Snow, robust and benign activations exhibit high similarity (Figure 22 Changing budgets across the same threat model (Figure 30) we find that lp threat models' representations evolve more gradually.
Discussion
Limitations. The key limitations of our study arise from the fact that all discoveries are predicated on the effectiveness of the RS metric used to discern differences between representations. Further, the nature of our study can only unearth correlations between layer-wise and aggregate properties such as accuracy and loss. Establishing causal links needs further investigation, perhaps inspired by our findings as we discuss below. Finally, methods to improve robust training are beyond the scope of this paper. They need to be derived from a careful analysis of the observations in this paper for actionable changes to training strategies and architectures.
Future Work. Our observations highlight that robust networks could be improved in key ways:
1. Staggered freezing of layers during training: Our results from $ 5 indicate that early layers do not need to be updated post few epochs of training, since their learned representations do not change much during training. Thus, knowledge of a network's 's training dynamics derived from CKA analysis can be used increase the efficiency of training through the freezing of early layers.
2. Increasing layer-wise differentiation during training: Our results show there is a much greater degree of local similarity among learned representations for robust networks when compared to non- robust ones. This similarity also increases when the training budget is increased. We suspect this lack of layer-wise differentiation is part of the reason why robust networks do not achieve high accuracy on clean data. Using regularization methods that promote increased layer-wise differentiation during robust training may alleviate this issue, and is a compelling direction for future work.
3. Choosing threat models for joint robust training: Past work on training models jointly robust to multiple attacks has focused on different lp perturbations. We posit that our analysis of the similarity of representations obtained from different threat models can be utilized to determine sets of threat models for which joint robustness is possible, and if the model has sufficient capacity. Further, the layer-wise analysis can be used to add appropriate regularizers to ensure convergence, an issue exacerbated by the presence of multiple types of adversarial examples.
Acknowledgments and Disclosure of Funding
We thank our anonymous reviewers for their insightful feedback. CC, ANB, BYZ and HZ were supported in part by NSF grants CNSS-149665 CNS-1923778, CN5-1705042, the C3.ai DTI, and the DARPA GARD program. VS and PM were supported in part by NSF grants CNS-1553437 and CNS-1704105 the ARL's A212, the ONR Young Investigator Award, the ARO Young Investigator Prize, Schmidt DataX award, Princeton E-ffiiiates Award, and Princeton Gordon Y. S. Wu Fellowship. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of any funding agencies.
10

---

References
[1] Andriushchenko, M. and Flammarion, N. (2020). Understanding and improving fast adversarial training. Advances in Neural Information Processing Systems, 33: 160489
[2] Bansal, Y., Nakkiran, P., and Barak, B. (2021). Revisiting model stitching to compare neural representations. Advances in Neural Information Processing Systems, 34.
[3] Beal, J., Wu, H.-Y., Park, D. H., Zhai, A., and Kislyuk, D. (2022). Billion-scale pretraining with vision transformers for multi-task visual representations. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 564-573.
[4] Biggio. B., Corona, I., Nelson, B., Rubinstein, B. I., Maiorea, D., Fumera, G., Giacinto, G., and Roli, F. (2014). Security evaluation of support vector machines in adversarial environments. In Support Vector Machines Applications, pages 105-153, Springer.
[5] Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A.. Shyam, P., Sastry, G., Askell, A., et al. (2020). Language models are few-shot learners. Advances neural information processing systems, 33:1877-1901.
[6] Conneau, A., Khandelwal, K., Goyal, N.. Chaudhary, V., Wenzek, G., Guzmán, F., Grave, E., Ott, M., Zettlemoyer, L., and Stoyanov, V. (2019). Unsupervised tross-lingual representation learning scale. arXiv preprint arXiv:1911.02.0
[7] Croce, F., Andriushchenko, M., Sehwag, V., Debenedetti, E., Flammarion, N., Chiang, M., Mittal, P., and Hein, M. (2020). Robustbench: a standardized adversarial robustness benchmark. arXiv preprint arX1v:2010.09670.
[8] Croce, F. and Hein, M. (2021). Adversarial robustness against multiple $1 3_pSSttheeat models the price of one and how to quickly fine-tune robust models to another threat model. arXiv:2105.12500 [cs].
[9] Csiszárik, A., Körösi-Szabó, P., Matszangosz, Á., Papp, G., and Varga, D. (2021). Similarity and matching of neural network representations. Advances in Neural Information Processing Systems, 34.
[10] Deng, J., Dong, W., Socher, R., Li, L.-J,, Li, K., and Fei-Fei, L. (2009a). ImageNet: A Large-Scale Hierarchical Image Database. In CVPR09.
[11] Deng, J., Dong, W., Socher, R., Li, L.-J,, K., and Fei-Fei, L. (2009b). Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. leee.
[12] Deng, J., Guo, J., Xue, N., and Zafeiriou, S. (2019). Arcface: Additive angular margin loss for deep face recognition. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 4690-4699.
[13] Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 11X0::8800004055.
[14] Ding, F,, Denain, J.-S., and Steinhardt, J. (2021). Grounding representation similarity through statistical testing. Advances in Neural Information Processing Systems, 34.
[15] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. (2021). An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations.
[16] Dwivedi, K., Huang. J., Cichy, R. M., and Roig, G. (2020). Duality diagram similarity: generic framework for initialization selection in task transfer learning. In European Conference Computer Vision, pages 497-513. Springer.
[17] Gao, R., Cai, T., Li, H., Hsieh, C.-J., Wang, I.., and Lee, J. D. (2019). Convergence of adversarial training in overparametrized neural networks. Advances Neural Information Processing Systems, 32.
11

---

[18] Gavrikov, P. and Keuper, J. (2022). Adversarial robustness through the lens of convolutional filters. arXiv preprint arXiv:2204.02481. 024481..
[19] Gilboa, D. and Gur-Ari, G. (2019). Wider networks learn better features. arXiv preprint
[20] Gowal, S., Qin, C., Uesato, J., Mann, T., and Kohli, P. (2020). Uncovering the limits of adversarial training against norm-bounded adversarial examples. arXiv preprint arXiv:2010.03599,
[21] Grigg, T. G., Busbridge, D., Ramapuram, J., and Webb, R. (2021). Do self-supervised and supervised methods learn similar visual representations? arXiv preprint arXiv:21 10.00528.
[22] Hardoon, D. R., Szedmak, S., and Shawe- Tawlee, J. (2004). Canonical correlation analysis: An overview with application to learning methods. Neural computation.
[23] Hendrycks, D. and Dietterich, T. (2018). Benchmarking neural network robustness to common corruptions and perturbations. International Conference on Learning Representations.
[24] Hinton, G., Deng, L., Yu, D., Dahl, G. E., Mohamed, A.rr, Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T. N., et al. (2012). Deep neural networks for acoustic modeling in speech recognition: The shared views four research groups. IEEE Signal processing magazine, 29(6)882-97.
[25] Howard, J. (2020). imagenette.
[26] Huang, H., Wang. Y., Erfani, S., Gu, Q.. Bailey, J., and Ma, X. (2021). Exploring architec- tural ingredients of adversarially robust deep neural networks. Advances in Neural Information Processing Systems, 34.
[27] Ilyas, A., Santurkar, S., Tsipras, D., Engstrom, L., Tran, B., and Madry, A. (2019). Adversarial examples are not bugs, they are features. Advances in neural information processing systems, 32.
[28] Kang, D., Sun, Y., Hendrycks, D., Brown, T., and Steinhardt, J. (2019). Testing robustness against unforeseen adversaries. arXiv preprint arXiv: 1e0X.v::90..00 1908.08016
[29] Kirichenko, P., Izmailov, P., and Wilson, A. G. (2022). Last layer re-training is sufficient for robustness to spurious correlations. arXiv preprint arXiv:2204.0299
[30] Korbblith, S., Chen, T., Lee, H., and Norouzi, M. (2021). Why do better loss functions lead to less transferable features? Advances in Neural Information Processing Systems, 34.
[31] Kornblith, S., Norouzi, M., Lee, H., and Hinton, G. (2019). Similarity of neural network representations revisited. In International Conference on Machine Learning, pages 3519-3529. PMLR.
[32] Krizhevsky, A., Hinton, G., et al. (2009). Learning multiple layers of features from tiny images.
[33] Krizhevsky, A., Sutskever, I., and Hinton, G. E. (2012). Imagenet classification with deep convolutional neural networks. Advances neural information processing systems, 25.
[34] Leavitt, M. L. and Morcos, A. (2020). Linking average-aa and worst-case perturbation robustness via class selectivity and dimensionality. arXiv preprint arX1v:2010.07693.
[35] Madry, A., Makelov, A., Schmidt,, L., Tsipras, D., and Vladu, A. (2018). Towards deep learning models resistant to adversarial attacks. In ICLR.
[36] Mahajan, D., Girshick, R., Ramanathan, V., He, K., Paluri, M., Li, Y., Bharambe, A., and Van Der Maaten, L. (2018). Exploring the limits of weakly supervised pretraining. In Proceedings of the European conference on computer vision (ECCV), pages 181-196.
[37] Morcos, A., Raghu, M., and Bengio, S. (2018). Insights on representational similarity in neural networks with canonical correlation. Advances Neural Information Processing Systems, 31.
[38] Nakkiran, (2019). Adversarial robustness may at odds with simplicity. arXiv preprint 10X5021901.00532.
12

---

[39] Nayebi, A. and Ganguli, S. (2017). Biologically inspired protection of deep networks from adversarial attacks. arXiv preprint arXiv:1703.09202.
[40] Neyshabur, B., Sedghi, H., and Zhang, C. (2020). What is being transferred in transfer learning? Advances in neural information processing systems, 33:512-55 523.
[41] Nguyen, Q. and Hein, M. (2018). Optimization landscape and expressivity of deep enns. In International conference on machine learning, pages 9730-3739. PMLR.
[42] Nguyen, T., Raghu, M., and Kornblith, S. (2020). Do wide and deep networks learn the same things? uncovering how neural network representations vary with width and depth. arXiv preprint 200.:2070.1132...
[43] Raghu, A., Raghu, M., Bengio, S., and Vinyals, O. (2020). Rapid learning or feature reuse? towards understanding the effectiveness of maml. In International Conference on Learning Representations.
[44] Raghu, M., Gilmer, J., Yosinski, J., and Sohk-Dickstein, J. (2017). Sveea: Singular vector canomical correlation analysis for deep learning dynamics and interpretability. Advances in neural information processing systems, 30.
[45]
Raghu, M., Unterthiner, T., Kornblith, S., Zhang. C.,, and Dosovitskiy, A. (2021). Do vision transformers see like convolutional neural networks? Advances in Neural Information Processing Systems, 34.
[46] Raghunathan, A., Steinhardt, J., and Liang, (2018). Certified defenses against adversarial examples. In International Conference on Learning Representations.
[47] Ribeiro, M. T., Singh, S., and Guestrin, C. (2016). why should i trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pages 1135-1144.
[48] Rice, L.., Wong, E., and Kolter, Z. (2020). Overfitting in adversarially robust deep learning. In International Conference on Machine Learning, pages 8093-8104. PMLR.
[49] Sanyal, A., Kanade, V., Torr, P. H., and Dokania, P. (2018). Robustness via deep low-rank representations. arXiv preprint /XX:::800.00990
[50] Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., and Batra, D. (2017). Grad- cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision, pages 618-626.
[51] Shen, J., Pang, R., Weiss, R. J., Schuster, M., Jaitly, N., Yang. Z., Chen, Z., Zhang, Y., Wang, Y., Skerrv-Raan R., et al. (2018). Natural tts synthesis by conditioning wavenet on mel spectrogram predictions. In 2018 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 4779-4783, IEEE.
[52] Simonyan, K., Vedaldi, A., and Zisserman, A. (2013). Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint 13I2::332.6034.
[53] Sundararajan, M., Taly, A., and Yan, Q. (2017). Axiomatic attribution for deep networks. In International conference on machine learning, pages 3319-3328. PMLR.
[54] Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., and Fergus, R. (2013). Intriguing properties of neural networks. arXiv preprint arXiv: 1312.6199
[55] Tramèr, F. and Boneh, D. (2019). Adversarial training and robustness for multiple perturbations. Advances in Neural Information Processing Systems, pages 5866-5876.
[56] Tsipras, D., Santurkar, S., Engstrom, I.., Turner, A., and Madry, A. (2018). Robustness may be at odds with accuracy. arXiv preprint arXiv:1805.12152.
[57] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L.., Gomez, A. N., Kaiser, t.., and Polosukhin, I. (2017). Attention is all you need. Advances in neural information processing systems, 30.
13

---

[58] Vulić, I., Ponti, E. M., Litschko, R., Glavaš, G., and Korhonen, A. (2020). Probing pretrained language models for lexical semantics. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)) pages 7222-7240, Online. Association for Computational Linguisties.
[59] Wong, E., Santurkar, S., and Madry, A. (2021). Leveraging sparse linear layers for debuggable deep networks. In International Conference Machine Learning, pages 11205-11216 PMLR.
[60] Xie, C. and Yuille, A. (2020). Intriguing properties of adversarial training at scale. In ICLR. arXiv: 1906.03787.
[61] Yang, Y., Hodgkinson, L., Theisen, R., Zou, J., Gonzalez, J. E., Ramchandran, K., and Mahoney, M. W. (2021). Taxonomizing local versus global structure neural network loss landscapes. Advances in Neural Information Processing Systems, 34.
[62] Yosinski, J., Clune, J., Nguyen, A., Fuchs, T., and Lipson, H. (2015). Understanding neural networks through deep visualization. arXiv preprint arXiv:/506.0633 06579.
[63] Zagoruyko. S. and Komodakis, N. (2016). Wide residual networks. arXiv preprint 1rX1v11605.07166.
[64] Zeiler, M. and Fergus, R. (2014). Visualizing and understanding convolutional networks. In European conference on computer vision, pages 818-833. Springer.
[65] Zhang, H., Yu, Y., Jiao, J., Xing, E., El Ghaoui, L., and Jordan, M. (2019). Theoretically principled trade-off between robustness and accuracy. International conference on machine learning, pages 1482-7482. PMLR.
[66] Zhou, J., Cui, G., Hu, S., Zhang, Z., Yang. C., Liu, Z., Wang, L., Li, C., and Sun, M. (2020). Graph neural networks: A review methods and applications. Al Open, 1:57-81.
14


Reviews:
[{"review_id": "zr6vVvkyLP", "reviewer": "Reviewer_HbNq", "summary": "This paper uses an existing method for comparisons of intermediate neural network activations (CKA) for a comparison of robust and non-robust networks. The authors analyze the similarities in different aspects and try to deduce insights in adversarial training.", "questions": "- Missing related work: There has been previous work on comparing the features of robust vs. non-robust neural networks that should be cited, e.g. using feature visualizations [1].\n- L25: Reference missing.\n- L45f: Reference [18] should be placed behind Imagewoof and not behind Imagenet; also, a reference for ImageNet is missing. \n- L93: E.g., JPEG compression doesn't seem like an additive perturbation.\n- L108: If this is well known, I encourage the authors to support this sentence with (multiple) references.\n- L118: A more detailed critical discussion of the possible shortcomings of these metrics and why they don't impact the results of this paper would be good.\n- L123f: Why doesn't this impact your results? This definitely needs to be supported by a strong argument. \n- Figure 2/3: On which datasets were these metrics calculated? On the vanilla test data or on adversarially perturbed versions of the test data? What does the figure look like for the other data type (either benign or adversarial)?\n- Figure 2/3: Given that the clean accuracy goes down for adversarial training, I'm wondering whether the change in the similarity plots is really due to a special aspect of adversarial training or just due to the drop in performance, i.e. how do these plots look like for non-robust networks that are not trained until convergence but until they reach a similar test accuracy as the robust networks?\n- L163f: That seems a bit like an overstatement: The block structure is still clearly visible for 2 out of the 3 datasets - it just gets weaker.\n- Multiple typos/grammar mistakes that make it sometimes break the flow of the text, e.g. L172, L187, L201, L216, L221, L235.\n- Figure 4: These plots should be larger - the text is hard to read.\n- Figure 4: The first sentence of the caption doesn't read right/is difficult to parse.\n- L185f: Did the authors observe the same behavior as described by Nguyen et al. when they tried to reproduce their observations for non-robust networks? At the moment it is difficult to confidently say that the observation reported here is because of the robust network or because of the experimental setup used by the authors.\n- L213: If this is well known, I again encourage the authors to support this sentence with (multiple) references. Furthermore, this is actually not such a clear property, and there are specific attacks that just aim to create adversarial examples that transfer well between different models.\n- Minor comment: The paper uses inconsistent notation - sometimes the authors say \"Figure\", sometimes \"figure\"; sometimes \"Appendix\", sometimes \"App.\".\n- L252f: Where can we see both validation and training loss? The figure only shows one \"loss\" but doesn't say which one it is.\n- L265: I wouldn't call these \"adversarial perturbations\" but rather use the more commonly used expression common corruptions. Especially, since at least for JPEG compression there is nothing you can optimize - so that doesn't really fit the overall adversarial framework.\n- Section 6: What are the specific parameters of the JPEG, snow, and Gabor corruptions?\n- L289: What is the conclusion/interpretation of the results?\n\n[1] Leveraging Sparse Linear Layers for Debuggable Deep Networks. Eric Wong, Shibani Santurkar, Aleksander Mądry. 2021", "limitations": "- While the authors mentioned criticism on the similarity metric they use (CKA), they don't really explain why this doesn't apply to their analysis. This should be properly addressed.\n- It is also unclear whether the results differ if a different adversarial attack (with the same thread model) was used for generating the adversarial perturbations - this should ideally also be addressed.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- The paper uses an interesting & potentially insightful approach to better understand how adversarially trained networks process information and how they differ from non-robust networks.\n- The attached code for this submission is well documented and looks clean, making the results more trustworthy.\n- It feels like the paper spends too much time describing plots/results compared to interpreting the results and presenting hypotheses for what results mean.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "zHE4_I-J1kO", "reviewer": "Reviewer_mj83", "summary": "This paper studies the robustness of deep neural networks based on the perspective of representation similarity.  Such a perspective provides an interesting direction to delve deeper into the properties of robust representation learning. The authors make several novel discoveries on \"salient pitfalls\" in robust networks. According to the observations, the author introduces several ways to design and train better robust networks. ", "questions": "1.  Is there possible to involve more experimental results on how the proposed ways for better robust learning could improve upon the previous methods?", "limitations": "yes. ", "rating": 6, "confidence": 2, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n1. Analyzing the properties of robustness from the representation similarities is intuitive and has been explored by some previous works. However, the paper provided a very systematic study and provided lots of interesting discoveries. \n\n2. The paper is well written. The analysis and discussion are conducted in a very logical way. \n\n3. There are extensive experiments conducted to demonstrate the conclusion and observation. I believe the results are solid. \n\n\nWeakness:\n1.  I appreciate the efforts in providing different views/frameworks for an important research problem. While I would also like to find more practical effects of the proposed analysis and observations.  \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "YriM7CZhnoC", "reviewer": "Reviewer_EXva", "summary": "The authors examine the effects of adversarial robustness training on representation. Specifically, they use CKA to compare robust vs non-robust networks, benign vs. adversarial inputs, and how these comparisons change over the course of learning. They find the following results:\n- Robust representations are less specialized; distant layers are more similar in robust networks, and block structure is weaker\n- Early layers in robust networks are largely unaffected by adversarial examples; representations are similar for benign vs. perturbed inputs\n- Deeper layers overfit during robust learning\n- Models trained to be robust to different threat models have similar representations", "questions": "This work primarily examines adversarial (i.e. worst-case) robustness, which is one of many types of robustness, among others including average-case robustness (Hendrycks and Dietterich, Benchmarking Neural Network Robustness to Common Corruptions and Perturbations) and natural adversarial examples (Hendrycks et al.). It’s not clear how important of a problem adversarial robustness is (see Gilmer et al.’s Motivating the Rules of the Game for Adversarial Example Research). The authors examine robustness to Gabor, snow, and jpeg attacks, but the analysis and results are limited. I would encourage the researchers to more systematically examine how their findings extend to other types of robustness.\n\nAre the results from a single instance of each model? If so, I would strongly encourage the authors to repeat their analyses in triplicate at a minimum. If this is computationally cost-prohibitive, then perhaps they could focus on the most important results.\n\nThere is relevant previous work examining the relationship between representational dimensionality and adversarial robustness (Leavitt and Morcos, Linking average- and worst-case perturbation robustness via class selectivity and dimensionality; Sanyal et al., Robustness vai Deep Low-rank Representations; Nayebi and Ganguli, Biologically inspired protection of deep networks from adversarial attacks). These papers show that low rank representations confer adversarial robustness. Can you reconcile these results with your findings that adversarial robustness is associated with greater utilization of model capacity? Additionally, most of these works (certainly Leavitt and Morcos and Sanyal et al.) regularized representational dimensionality and found that it improved robustness; it would be quite interesting to investigate whether that the inverse holds: does adversarial training reduce representational dimensionality?\n\nThe claim that “Deeper layers overfit during robust learning” seems like it could be leveraged to improve the generalization gap caused by robust learning. The simplest approach would be to simply stop training when the adversarial loss begins to rise (as in Figure 6). I assume this has drawbacks, but the authors don’t present the data (e.g. accuracy curves over training). Is this overfitting a necessary component of robust learning?\n\nLines 59-61: “On the other hand, the representations of benign and perturbed inputs from robust networks are indistinguishable from one another with regards to representation similarity metrics.” I think to make this claim the authors need to directly compare CKA(Benign, Perturbed) to CKA(Benign, Benign) and CKA(Perturbed, Perturbed).\n\nThe caption for Figure 2 should describe the model architecture(s) used to generate the results, as well as whether the data are benign or perturbed. There are other Figures which lack important experimental details, such as whether the data are benign or perturbed.\n\nComparing the CKA results in Figure 2d to the results in Figures 2a-c make it clear that the similarity between robust and non-robust networks is lower than the similarity between networks of the same type, but I think it’s important to have a negative baseline—how low is “low”? Accordingly, I think the authors should repeat the analysis (CKA between robust and non-robust networks) using sample-shuffled data (and/or some other suitable random baseline).\n\nShowing that the block structure effect varies with the strength of adversarial training is a nice experiment and result.\n\nThe results presented in lines 174-188 (“Impact of robust training strength” and “Impact of architecture”) would be easier to interpret (and their motivation clearer) if you introduced them with Nguyen et al.’s finding that “block structure in the internal representations arises in models that are heavily overparameterized relative to the training dataset.” While you do cite their work (“Previously Nguyen et al. [28] observed that increasing network width, thus capacity, leads to emergence of block-structure in non-robust networks”), I think this is an insufficient description of their results and should be presented earlier.\n\n“Overfitting is predominantly visible in later layers”: I would suggest a follow-up experiment to more thoroughly test this claim: freeze early layers (blocks 1 and/or 2) early in training. If early layers aren’t involved in the overfitting and training instability, then the freezing should have minimal effect on these phenomena.\n\nLines 278-280: “When using CKA to compare against the Snow threat model, we observe that the highest average similarity is achieved with Gabor. This represents a novel insight into these threat classes, as correspondence between the two was not previously known.” This result seems very specific and not particularly useful. I would encourage the authors to find a more general result (see my earlier comment about examining different attack types).\n\nminor comments:\n\nLines 21-22: “...such as images, speech or text”. It’s a matter of personal taste, but I am a proponent of the Oxford comma: “...such as images, speech, or text”\n\nLine 22: What does “meaningful” mean in this context? Meaningful with regards to what?\n\nLine 187: “These results suggest that while increasing width in robust networks doesn’t lead a drastic shift in similarity of internal layer representations.” Typo?\n\nTypo in caption of Figure 5 (...to understand similairty)\n", "limitations": "The authors devote a paragraph at the end of the discussion to the limitations of their study, which seems appropriate given the space limitations. I do, however, think the authors could be more careful and nuanced about some of their claims.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: The paper follows in a well-established tradition of using representational similarity analysis to understand neural network behavior and training interventions. The analyses are well-motivated and straightforward. The results are generally presented clearly and easy to understand. The results are interesting.\n\nWeaknesses: The paper seems somewhat limited in scope: It primarily addresses adversarial (worst-case) robustness, which is only one type of robustness. I'm not totally convinced of the utility of this work; it could at the very least do a better job situating itself within existing robustness research. It's unclear if any of the experiments were run in replicate. The figures captions could be more informative and self-contained.\n\nOverall, I think this work could be suitable for publication if it is sufficiently revised.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "YMUytX7qYxe", "reviewer": "Reviewer_4F8X", "summary": "This paper presents a probing analysis on clean (non-robust) vs. adversarially trained robust models. The paper's novelty is questionable and the insights gained are also sort of obvious, e.g., that the representational differences between inputs increase as one goes deeper down the layers. While this is true (and has been observed by the authors in their experiments), the authors didn't make an attempt to analyze whether two inputs from the same class are do have similar representations, which is a good thing. How does this compare between robust and non-robust models?\n\nThe paper introduces a lot of defence mechanisms (mainly centred around the min-max idea). However, it only employs PGD asdversarial training for obtaining a \"robust\" model. What about other approaches that provide defence mechanisms against potential attacks, e.g. the references 23, 32, 43 etc. that the authors themselves cite?\n\n", "questions": "How do the analysis from the observational differences between robust and non-robust networks can actually be used in practice? Can we use these insights to guide us towards constructing more robust models?", "limitations": "Limitations are not properly explained. The expression \"fundamental disjunct between aggregate properties and layer-wise representation similarity metrics...\" is rather vague.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. Good analysis work on robust vs. non-robust networks.\n2. Uses CKA to measure representational similarities.\n\nWeaknesses:\n\n1. Only one version of robust network considered - those trained with PGD based adversarial examples.\n2. The rational of using CKA is not appropriately justified.\n3. Some analysis could/should have been at a more detailed level, like one would still want the differences between different classes to be high and similarties between identical classes to be high. What observations can we make regarding this expected behavior in a robust and a non-robust network? \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "TOno0nETe5x", "reviewer": "Reviewer_9y9Q", "summary": "The paper contrasts representation similarities of networks trained to perform image classification with and without adversarial noise. To do so, the authors measure the similarity of representations using the Centered Kernel Alignment (CKA) metric (as well two other similarity metrics in appendix) for CIFAR-10 and two subsets of ImageNet. The authors highlight 1) networks trained with adversarial noise have layers similar to one another compared to those of standard trained networks (which have a block structure) 2) representations in early layers are unaffected by adversarial perturbations both for standard and adversarially trained networks 3) networks trained with and without adversarial perturbations have similar representations until the last 10 or so layers 4) early layers converge faster and later layers overfit to local minima. The authors also analyze the similarity of representations when other perturbations are applied JPEG, Gabor, and Snow, finding ", "questions": "included above", "limitations": "Yes.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This work investigates representations learned for image classification by contrasting the representation similarities of networks trained with and without adversarial noise—leading to several insights into properties of the learned representations as well as learning dynamics of modern networks. Such properties as the authors point out are not captured by aggregate performance metrics such as loss or accuracy, leading to insights about the learned representations. The paper is well-written, experiments clearly described, and motivation is well-grounded.\n\nThe experiments conducted are convincing and well-formulated. For example, multiple similarity metrics are compared, variants of the adversarial perturbations are explored, and experimental claims are well-founded. \nHowever, the authors'claims should be sufficiently couched within the experimental settings studied: supervised image classification for ResNet-based models on CIFAR-10 and two subsets of ImageNet. For example, the work only studies ResNet-based architectures yet claims to cover “DNNs with different architectures” (line 12) and the “impact of choice of architecture” (line 37). I would expect a comparison of “different architectures” to encompass for example transformer-based architectures, MLP-architectures, etc. I suggest the authors more explicitly couch claims in the introduction, abstract, title, and conclusions within the confines of the experimental settings studied. For example \nThe comparison of representations for \"threat models\" (JPEG, Gabor, and Snow) in Section 6 was not particularly informative and seemed removed from the primary findings of the remainder of the paper. \n\nRobust representations is a broad term. This work studies robustness to random noise in the input. Yet, the authors intermix “robustness to adversarial examples” and often plainly use the term “robust network” to describe a particular type: robustness to noise (defined via L-p bounds on the input) throughout the work. For example, robust in the context of image classification can just as well refer to robustness with respect to rendering method ImageNet-Sketch, artifacts such as blurring (ImageNet-C), adversarial examples (ImageNet-A), or even robustness to natural transformations such as pose (Alcorn et al.). I suggest the authors clarify the wording to only include the specific definitions of robustness studied here.\n\nThe insights gained from the authors’ analysis are interesting and well-described. The finding of most value, in my opinion, is that later layers overfit (matching existing work relating overfitting in later layers to spurious correlations [1]). While the analysis sheds light on differences between networks trained with and without adversarial noise as well as their learning dynamics the findings are confined to the L-infinity definition of robustness studied for supervised image classification using ResNet-based architectures (for the main claims in section 4 and 5).\n\n[1] “Last Layer Re-Training is Sufficient for Robustness to Spurious Correlations” [https://arxiv.org/abs/2204.02937](https://arxiv.org/abs/2204.02937)\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept