Linking average- and worst-case perturbation robustness via class selectivity and dimensionality
Authors: Matthew L Leavitt
Ari S. Morcos
Venue:  (2021)
Keywords: robustness
adversarial robustness
corruptions
class selectivity
deep learning

Abstract: Representational sparsity is known to affect robustness to input perturbations in deep neural networks (DNNs), but less is known about how the semantic content of representations affects robustness. Class selectivity—the variability of a unit’s responses across data classes or dimensions—is one way of quantifying the sparsity of semantic representations. Given recent evidence that class selectivity may not be necessary for, and in some cases can impair generalization, we sought to investigate whether it also confers robustness (or vulnerability) to perturbations of input data. We found that class selectivity leads to increased vulnerability to average-case (naturalistic) perturbations in ResNet18, ResNet50, and ResNet20, as measured using Tiny ImageNetC (ResNet18 and ResNet50) and CIFAR10C (ResNet20). Networks regularized to have lower levels of class selectivity are more robust to average-case perturbations, while networks with higher class selectivity are more vulnerable. In contrast, we found that class selectivity increases robustness to multiple types of worst-case (i.e. white box adversarial) perturbations, suggesting that while decreasing class selectivity is helpful for average-case perturbations, it is harmful for worst-case perturbations. To explain this difference, we studied the dimensionality of the networks' representations: we found that the dimensionality of early-layer representations is inversely proportional to a network's class selectivity, and that adversarial samples cause a larger increase in early-layer dimensionality than corrupted samples. We also found that the input-unit gradient was more variable across samples and units in high-selectivity networks compared to low-selectivity networks. These results lead to the conclusion that units participate more consistently in low-selectivity regimes compared to high-selectivity regimes, effectively creating a larger attack surface and hence vulnerability to worst-case perturbations.

Under review as a conference paper at ICLR 2021
LINKING AVERAGE- AND WORST-CASE PERTURBATION ROBUSTNESS VIA CLASS SELECTIVITY
AND
DIMENSIONALITY
Anonymous authors Paper under double-blind review
ABSTRACT
Representational sparsity is known to affect robustness to input perturbations in deep neural networks (DNNs), but less is known about how the semantic content of representations affects robustness. Class selectivity-the variability of unit's s responses across data classes or dimensions is one way of quantifying the sparsity of semantic representations. Given recent evidence that class selectivity may not be necessary for, and some cases can impair generalization, we sought to investigate whether it also confers robustness (or vulnerability) perturbations of input data. We found that class selectivity leads to increased vulnerability to average-case (naturalistic) perturbations in ResNet18, ResNet50, and ResNet20, as measured using Tiny ImageNetC (ResNet18 8 and ResNet50) and CIFAR 1OC (ResNel20). Networks regularized have lower levels of class selectivity are more robust average-cass perturbations, while networks with higher class selectivity are more vulnerable. In contrast, we found that class selectivity increases robustness to multiple types of worst-case (i.e. white box adversarial) perturbations, suggesting that while decreasing class selectivity is helpful for average-case perturbations, it is harmful for worst-case perturbations. To explain this difference, studied the dimensionality of the networks' representations: we found that the dimension- ality of early-layer representations is inversely proportional to network's 's class selectivity, and that adversarial samples cause larger increase in early-layer dimen- sionality than corrupted samples. We also found that the input-unit gradient was more variable across samples and units in high-selectivity networks compared to low-selectivity networks. These results lead the conclusion that units participate more consistently in low-selectivity regimes compared to bigh-sclectivity regimes, effectively creating a larger attack surface and hence vulnerability worst-case perturbations.
INTRODUCTION
Methods for understanding deep neural networks (DNNs) often attempt to find individual neurons or small sets of neurons that are representative of a network's 's decision (Erhan et af.) 2009 Zeiler
and Fergus 2014 Karpathy et al.) 2016 Amjad et al. 2018 Lillian al. 2018:] Dhamdhere al. 2019 Olah et al. 2020). Selectivity in individual units (i.c. variability in a neuron's s activations across
data features) has been of particular interest researchers trying better
understand deep neural networks (DNNs) (Zhou et al. 2015 Olah et al. Morcos et al. 2018
Zhou al. 2018 Meyes et al.
2019 Na et al. 2019 Zhou et al. 20199] Rafegas et al.) 2019 Bau
et al. 2020 Leavitt and Morcos 2020) However, recent work has shown that selective neurons can
be irrelevant, or even detrimental to network performance, emphasizing the importance of examining
distributed representations for understanding DNNs (Morcos et al. 2018 Donnelly and Roegiest
2019 Dalvi et al. 2019b. Leavitt and Morcos 2020).
In parallel, work on robustness seeks to build models that are robust to perturbed inputs (Szegedy 2013.
et al.
Carlini and Wagner 2017a by Vasiljevic et al. 2016 Kurakin et al. 2017 Gilmer
al. 2018 Zheng al. (2016). Hendrycks and Dietterich (2019) distinguish between two types
of robustness: corruption robustness, which measures a classifier's performance on low-quality or naturalistically-pertui inputs- inp thus is "average-case" measure- and adversarial robustness,

---

Under review as a conference paper at ICLR 2021
which measures a classifier's performance small, additive perturbations that are tailored to the classifier and thus is a "worst-case" measure !
Research on robustness has been predominantly focused on worst-case perturbations. which is affected by weight and activation sparsity (Madry et al.) 2018 Balda et al., 2020, Ye et al. 2018) Guo et al.
2018,
Dhillon et al.) 2018) and representational dimensionality (Langeberg et al. 20199 |Sanyal et al.
2020 Nayebi and Ganguli 2017). But less is known about the mechanisms underlying average-case
perturbation robustness and its common factors with worst-case robustness. Some techniques for improving worst-case robustness also improve average-case robustness (Hendrycks and Dietterich 2019; Ford et al., 2019 Yin et al., 2019), thus it possible that sparsity and representational dimensionality also contribute to average-case robustness. Selectivity in individual units can be also be thought of measure the sparsity with which semantic information is represented And because class selectivity regularization provides method for controlling selectivity, and class selectivity regularization has been shown to improve test accuracy on unperturbed data (Leavitt and Morcos 2020), we sought to investigate whether it could be utilized to improve perturbation robustness and elucidate the factors underlying it.
In this work we pursue series of experiments investigating the causal role of selectivity in robustness to worst-case and average-case perturbations in DNNs. To do so, we used a class selectivity regularizer (Leavitt and Morcos 2020) directly modify the amount of class selectivity learned by DNNs, and examined how this affected the DNNs' robustness to worst-case and average-case perturbations. Our findings are as follows:
Networks regularized to have lower levels of class selectivity are more robust to average-case perturbations, while networks with higher class selectivity are generally less robust to average-case perturbations, as measured in ResNets using the Tiny ImageNetC and CIFARIOC datasets. The corruption robustness imparted by regularizing against class selectivity was consistent across nearly all tested corruptions.
In contrast to its impact on average-case perturbations, decreasing class selectivity reduces robustness to worst-case perturbations in both tested models, as assessed using mudient-bbsse white-box attacks.
The variability of the input-unit gradient across samples and units is proportional to network's 's overall class selectivity, indicating that high variability in perturbability within and across units may facilitate worst-case perturbation robustness.
The dimensionality of activation changes caused by corruption markedly increases in early layers for both perturbation types, but is larger for worst-case perturbations and low-selectivity networks. This implies that representational dimensionality may present trade-off between worst-case and average-case perturbation robustness.
Our results demonstrate that changing class selectivity, and hence the sparsity of semantic representa- tions, can confer robustness to average-case or worst-case perturbations, but not both simultaneously. They also highlight the roles of input-unit gradient variability and representational dimensionality in mediating this trade-off.
RELATED WORK
2.1 PERTURBATION ROBUSTNESS
The most commonly studied form of robustness in DNNs is robustness to adversarial attacks, in which an input is perturbed in manner that maximizes the change in the network's output while
We use the terms "worst-case perturbation" and "average-case perturbation" instead of "adversarial attack" and "corruption", ", respectively, because this usage is more general and dispenses with the implied categorical distinction of using eeemingly-unrelatee terms. Also note that while Hendrycks and Dietterich (201)) assign specific and distinct meanings "perturbation" and "corruption", we use the term perturbation" more generally to refer to any change to an input.
*Class information semantic. And because class selectivity measures the degree which class information represented in individual neurons, can be considered form of sparsity. For example, if network has high test accuracy on a classification task, it is necessarily representing class (semantic) information. But if the mean class selectivity across units is low, then the individual units do not contain much class information, thus the class information must be distributed across units; the semantic representation in this case is not sparse, it is distributed.
2

---

Under review as a conference paper at ICLR 2021
attempting minimize or maintain below some threshold the magnitude of the change to the input (Serban et al. 2019; WordeWWrde-Faale Warde-Farley and Goodfellow, (017) Because white-box adversarial attacks optimized to best confuse given network, robustness to adversarial attacks are "worst-case" measure of robustness. Two factors that have been proposed to account for DNN robustness to worst-case perturbations are particularly relevant to the present study: sparsity and dimensionality.
Multiple studies have linked activation and weight sparsity with robustness to worst-case perturbations. Adversarial training improves worst-case robustness Goodfellow et al. (2015); Huang et al. (2016)
and results in sparser weight matrices (Madry et al. 20188 Balda et al. 2020). Methods for increasing
the sparsity of weight matrices (Ye et al. 2018 Guo et al. 2018) and activations (Dhillon et al. 2018
likewise improve worst-case robustness, indicating that the weight sparsity caused by worst-case
perturbation training is not simply a side-effect.
Researchers have also attempted to understand the nature of worst-case robustness from perspective complementary to that of sparsity: dimensionality. Like sparsity, worst-case perturbation training reduces the rank of weight matrices and representations, and regularizing weight matrices and representations to be low-rank can improve worst-case perturbation robustness (Langeberg et al. 1019|Sanyye et al. 2020 Nayebi and Ganguli) 2017). Taken together, these studies support the notion that networks with low-dimensione representations are more robust to worst-case perturbations.
Comparatively less rescarch has been conducted to understand the factors underlying average- case robustness. Certain techniques for improving worst-case perturbation robustness also help against average-case perturbations (Hendrycks and Dietterich 2019 Geirhos al., 2018 Ford et al., 2019). Examining the frequency domain has elucidated one mechanism: worst-case perturbations for "baseline" models tend to be in the high frequency domain, and improvements in average- case robustness resulting from worst-case robustness training are at least partially ascribable to models becoming less reliant on high-frequency information (Yin et al., 2019; Tsuzuku and Sato 2019; Geirhhs et al. 20188. But it remains unknown whether other factors such as sparsity and dimensionality link these two forms of robustness.
2.2 CLASS SELECTIVITY
One technique that has been of particular interest to researchers trying to better understand deep
(and biological) neural networks is examining the selectivity of individual units (Zhou al. 2015
Olah et al.
2017 Morcos et al. 22018] Zhou et al.] 20188) Meyes et al. 2019 Na et al. 2019
Zhou
al. 22019 Rafegas et al./ 2019: Bau al.) 2020 Leavitt and Morcos| 2020; Sherrington 1906
Kandel et al.
2000). Evidence regarding the importance of selectivity has mostly relied on single unit
ablation, and
has been equivocal (Radford al. 2017; Morcos et al. 2018 Amjad et al., 2018 Zhou
et al. 2018 Donnelly and Roegiest [019]/ally et al. 2019a). However Leavitt and Morcos (2020)
examined the role of single unit selectivity network performance by regularizing for or against class selectivity in the loss function, which sidesteps the limitations of single unit ablation and correlative
approaches and allowed them to investigate the causal effect of class selectivity. They found that
reducing class selectivity has little negative impact on-and can even improve test accuracy in CNNs trained on image recognition tasks, but that increasing class selectivity has significant negative effects on test accuracy. However, their study focused examining the effects of class selectivity on test accuracy in unperturbed (clean) inputs. Thus it remains unknown how class selectivity affects robustness to perturbed inputs, and whether class selectivity can serve as elucidate link between worst-case and average-case robustness.
3 APPROACH
detailed description of our approach is provided in Appendix A.
Models and training protocols Our experiments were performed on ResNet18 and ResNet50 (He et al. 2016) trained on Tiny ImageNet (Fei-Feii et al. (2015), and ResNet20 (He et al.) 2016) trained on CIFARIO (Krizhevsky) 2009). We focus primarily on the results for ResNet18 trained on Tiny ImageNet in the main text for space, though results were qualitatively similar for ResNet50, and ResNet20 trained on 10. Experimental results were obtained with model parameters from the epoch that achieved the highest validation set accuracy over the training epochs, and 20 replicate
3

---

Under review as a conference paper at ICLR 2021
models (ResNet18 8 and ResNet20) or 5 replicate models (Resnet50) with different random seeds were run for each hyperparameter set.
Class selectivity index Following (Leavitt and Morcos 2020). A unit's class selectivity index is calculated as follows: At every ReLUU, the activation in response to single sample was averaged across all elements of the filter map (which we refer to as a "umit"). The class-conditional mean activation was then calculated across all samples in the clean test set, and the class selectivity index (ST) was calculated as follows:
Hmar M-maz
SI
Pmaz p-max
where Umaa is the largest class-conditional mean activation and ja- is the mean response to the remaining (i.e. non-)4max) classes. The selectivity index ranges from 0 1. A unit with identical average activity for all classes would have a selectivity of O, and a unit that only responds to a single class would have a selectivity of I.
As Morcos et al. (2018) note, the selectivity index is not a perfect measure of information content in single units. For example, a unit with litte bit of information about many classes would have low selectivity index. However, it identifies units that are class-selective similarly to prior studies (Zhou
al. 2018 Most importantly, is differentiable with respect to the model parameters.
Class selectivity regularization We used (Leavitt and Morcos 2020 's class selectivity regularizer to control the levels of class selectivity learned by units in a network during training. Class selectivity regularization is achieved by minimizing the following loss function during training:
loss Σ yo log(g))) apst
The left-hand term in the loss function is the standard classification cross-entropy, where is the class index, C' is the number of classes, Ve is the true class label, and ye is the predicted class probability. The right-hand component of the loss function, -apAST is the class selectivity regularizer. The regularizer consists of two terms: the selectivity term,
PSI = L E Σ ST.,I
where is convolutional layer, I. is number of layers, u is unit, U is the number of units in a given layer, and S1,, is the class selectivity index of unit u. The selectivity term of the regularizer is obtained by computing the selectivity index for each unit in layer, then computing the mean selectivity index across units within each layer, then computing the mean selectivity index across layers. Computing the mean within layers before computing the mean across layers (as compared to computing the mean across all units in the network) mitigates the biases induced by the larger numbers of units in deeper layers. The other term in the regularizer is a, the regularization scale, which determines whether class selectivity is promoted or discouraged. Negative values of a discourage class selectivity in individual units and positive values encourage it. The magnitude of a controls the contribution of the selectivity term to the overall loss. During training, the class selectivity index was computed for each minibatch. The final (logit) layer was not subject selectivity regularization or included in our analyses because by definition, the logit layer must be class selective in a classification task.
Measuring average-case robustness To evaluate robustness to average-case perturbations, we tested our networks on CIFARIOC and Tiny ImageNetC, two benchmark datasets consisting of the CIFARIO or Tiny ImageNet data, respectively, to which a set of naturalistic corruptions have been applied (Hendrycks and Dietterich 2019 examples in Figure We average across all corruption types and severities (see Appendix 1.1.2 for details) when reporting corrupted test accuracy.
Measuring worst-case robustness We tested our models' worst-case (i.e. adversarial) robustness using two methods. The fast gradient sign method (FGSM) (Goodfellow et al. 2015) is a simple attack that computes the gradient of the loss with respect to the input image, then scales the image's pixels (within some bound) in the direction that increases the loss. The second method, projected gradient descent (PGD) (Kurakin et al.) 2016 Madry et al. 2018), is an iterated version of FGSM. We used step size of 0.0001 and an T.x0 norm perturbation budget (e) of 16/255.
Computing the stability of units and layers To quantify variation in networks' perturbability, first computed the la norm of the input-unit gradient for each unit u in a network. We then computed the mean ((.)) and standard deviation (a,,) of the norm across samples for each unit. a au/pa, yields
4

---

Under review as a conference paper at ICLR 2021
35
Saleciiity Rogulartzation Scale
-3 -1. -07
-07-44 -22
02 ne BY
10 5.0
Low
25
Seleciiiit
Suledivity
20
15
10
0
Shot
Brightness
Picelate
Glass
Motion
Impuise
Frost
Contrast
Defocus
Slastic
Snow
Flg
Gaussian
Zoom
Figure 1: Reducing class selectivity improves average-case robust- ness. Test accuracy (y-axis) as a function of corruption type (x-axis), class selectivity regularization scale (a; color), and corruption severity (ordering along y-axis). Test accuracy reduced proportionally to corrup- tion severity, leading to an ordering along the y-axis: corruption severity (least severe) is at the top, corruption severity 5 (most severe) at the bottom. (b) Mean test accuracy across all corruptions and severities (y-axis) as function of (X-axis). Results shown are for ResNet18 trained Tiny ImageNet and tested on Tiny ImageNetC. Error bars 95% confidence intervals of the mean. See Figure A6 for CIFARIOC results.
144
13
-2.0 1.0 -0.7 4..4--442 0.2 0.0 0.4
Regulitraaion Stale (o]
the coefficient of variation (Everitt 2002) for a unit (CV)), a measure of variation in perturbability for individual units. We also quantified the variation across units in layer by computing the standard deviation of (r,, across units in a layer 1, (1ww) oi and dividing this by the corresponding mean across units p([u)) ju, yield the CV across units 01/M4 /M CV.
4 RESULTS
4.1
AVERAGE-CASE ROBUSTNESS IS INVERSELY PROPORTIONAL. TO CLASS SELECTIVITY
Certain kinds of crusity--inlluiig reliance on single directions Morcos et al. 2018), and the semantic sparsity measured by class selectivity (Leavitt and Morcos| 2020) have been shown to impair network performance. We sought to extend this question to robustness: how does the sparsity of semantic representations affect robustness to average-case perturbations of the input data? We used a recently-introduce method (Leavitt and Morcos (2020); Approach (3) to modulate the amount of class selectivity learned by DNNs (Figure|A2) demonstratees effects of selectivity regularization). We then examined how this affected performance on Tiny ImageNetC and CIFAR1OC, two benchmark datasets for average-case corruptions (Approach 3 example images Figure AT).
Changing the level of class selectivity across neurons in network could one of the following effects on corruption robustness: If concentrating semantic representations into fewer neurons (i.c. promoting semantic sparsity) provides fewer potent dimensions which perturbed inputs can act, then increasing class selectivity should confer networks with robustness to average-case perturbations, while reducing class selectivity should render networks more vulnerable. Alternatively, if distributing semantic representations across more units (i.e. reducing sparsity) dilutes the changes induced by perturbed inputs, then reducing class selectivity should increase a network's robustness to average- case perturbations, while increasing class selectivity should reduce robustness.
We found that decreasing class selectivity leads increased robustness to average-case perturbations for both ResNet18 18 tested on Tiny ImageNeiC (Figure I and ResNet20 tested on CIFAR IOC (Figure A6). In ResNet18, we found that mean test accuracy on corrupted inputs increases as class selectivity decreases (Figure D with test accuracy reaching maximum at regularization scale a = -2.0 (mean test accuracy across corruptions and severities at a-2.0 =))), representing a 3.5 percentage point (pp) increase relative to no selectivity regularization (i.e. og; test accuracy at 0g- = 13.5). In contrast, regularizing to increase class selectivity has either no effect or negative impact on corruption robustness. Corrupted test accuracy remains relatively stable until declines. The results are qualitatively similar for ResNet50 tested on Tiny ImageNetC (Figure|AA) and for ResNet20 tested CIFARIOR 1OC (Figure except the vulnerability to corruption caused by increasing selectivity is even more dramatic in ResNet20. We also found similar results when controlling for the difference in clean accuracy for models with different (Appendix A.3)
1.0, after which point it
We observed that regularizing to decrease class selectivity causes robustness to average- pertur- bations. But it's possible that the causality is unidirectional, leading to the question of whether the
5

---

Under review as a conference paper at ICLR 2021
b)
par Selertuiity
6683
6003
0.00
5305 OO2S KESIN EXI STTL BETN BETIN 2.07
6001
Stags
Figure 2: Class selectivity imparts worst-case perturbation robustness. (a) Test accuracy (y-axis) as function of perturbation intensity (c; x-axis) and class selectivity regularization scale (a; color) for the FGSM attack. (b) Test accuracy (y-axis) as function of attack optimization steps (x-axis) and for the PGD attack. (c) Network stability, as measured with ly norm of the input-output Jacobian (y-axis), as a function of a (x-axis). All results are for R trained on Tiny ImageNet. Shaded region and bars = 95% confidence interval of the mean, See Figure A12 for ResNer20 results.
converse also true: does increasing robustness to average-case perturbations cause class selectivity to decrease? We investigated this question by training with AugMix, a technique known to improve worst-case robustness (Hendrycks et al. 2020a). We found that AugMix does indeed decrease the mean level of class selectivity across neurons in a network (Appendix A..4; Figure AII] AugMix decreases overall levels of selectivity similarly to training with class selectivity regularization scale of approximately a -0.1 or -0.2 in both 8 trained on Tiny ImageNet (Figures Alla and ATIb) and ResNet20 trained on CIFARIO (Figures A and [TTd). These results indicate that the causal relationship between average-case perturbation robustness and class selectivity is bidirectional: not only does decreasing class selectivity improve average-case perturbation robustness, but improving average-case
cntturbation-robustter also causes class selectivity to decrease.
We also found that the effect of class selectivity on perturbed robustness is consistent across corruption types. Regularizing against selectivity improves perturbation robustness in all 15 Tiny ImageNetC corruption types for ResNet18 8 (Figure(A4 and 14 of 15 Tiny ImageNetC corruption types in ResNet50 (Figure [A and 14 of 19 corruption types in CIFARIOC for ResNet20 (Figure(A7) Together these results demonstrate that reduced class selectivity confers robustness to average-case perturbations, implying that distributing semantic representations across neurons-is low sparsity-may dilute the changes induced by average-case perturbations.
4.2 CLASS SELECTIVITY IMPARTS WORST-CASE PERTURBATION ROBUSTNESS
We showed that the sparsity of network's semantic representations, as measured with class selectivity, is causally related to a network's ' robustness to average-case perturbations. But how does the sparsity of semantic representations affect worst-case robustness? We addressed this question by testing our class reguulaity-e networks on inputs that had been perturbed using using one of two gradient-based methods (see Approach
If distributing semantic representations across units provides more dimensions upon which worst- case perturbation is potent, then worst-case perturbation robustness should be proportional to class selectivity. However, if increasing the sparsity of semantic representations creates more responsive individual neurons, then worst-case robustness should be inversely proportional to class selectivity.
Unlike average-case perturbations, decreasing class selectivity decreases robustness worst-case perturbations for & (Figure 2 and ResNet50 (Figure A13) trained on Tiny ImageNet, and ResNer20 trained on CIFAR10 10 (Figures ATT.. For small perturbations (i.e. close x=0), the effects of class selectivity regularization on test accuracy (class selectivity is inversely correlated with unperturbed test accuracy) appear overwhelm the effects of perturbations. But as the magnitude of perturbation increases, a stark ordering emerges: test accuracy monotonically decreases as a function of class selectivity in ResNet18 and ResNet50 for both FGSM and PGD attacks (ResNet18: Figures 2a and 2b: ResNet50: Figures A13a and A13b). The ordering is also present for ResNet20, though less consistent for the two networks with the highest class selectivity (a 0.7 and a = 1.0). However, increasing class selectivity is much more damaging to test accuracy in ResNer20 trained on CIFAR compared to ResNet18 trained on Tiny ImageNet (Leavitt and Morcos| 2020 Figure A2). the the substantial performance deficits of extreme selectivity in ResNet20 likely mask the This result demonstrates that networks with sparse semantic representations are less vulnerable to worst-case perturbation than networks with distributed semantic representations. We also verified that the worst-case robustness of high-selectivity networks is not fully explained by mrakiigg (Athalye et al.,, 2018, Appendix A.5).
perturbation--obusiine
Interestingly, class selectivity regularization does not appear to affect robustness to "natural" adver- sarial examples (Appendix A.6). which are "unmodified, real-world xxaplee...selected to cause

---

Under review as a conference paper at ICLR 2021
model to make mistake" (Hendrycks et al. 2020b). Performance on ImageNet-A, benchmark of natural adversarial examples (Hendrycks et | 2020b), was similar across all tested values of for both ResNet18 (Figure AT5a) and ResNet50 (Figure A15b), indicating that class selectivity regularization may share some limitations with other methods for improving both worst-case and average-case robustness, many of which also fail to yield significant robustness improvements against ImageNet-A (Hendrycks et al.) 2020b).
We found that regularizing increase class selectivity causes robustness to worst-case perturbations. But is the converse true? Does increasing robustness to worst-case perturbations also cause class selectivity to increase? We investigated this by training networks with a commonly- to improve worst-case perturbation robustness, PGD training. We found that PGD training does indeed increase the mean level of class selectivity across neurons in a network, and this effect is proportional to the strength of PGD training: networks trained with more strongly-perturbel samples have higher class selectivity (Appendix (A.7). This effect was present in both ResNet1 trained on Tiny ImageNet (Figure A and ResNet20 trained on CIFARIO (Figure[A A166). indicating that the causal relationship between worst-case perturbation robustness and class selectivity is bidirectional.
technique
Networks whose outputs are more stable to small input perturbationn are known to have improved generalization performance and worst-case perturbation robustness (Drucker and Le Cun 1992]
Novak et al. 2018 (Sokolic et al. 2017 Rifai et al. 2011 Hoffman et 2019). To examine whether increasing class selectivity improves worst-case perturbation robustness by increasing network stability, we analyzed each network's input-output Jacobian, which is proportional to its stability large-magnitude Jacobian means that a small change to the network's 'ss input will cause a large change to its output. If class selectivity induces worst-case robustness by increasing network stability, then networks with higher class selectivity should have smaller Jacobians. But if increased class selectivity induces adversarial robustness through alternative mechanisms, then class selectivity should have no effect on the Jacobian. We found that the la norm of the input-output Jacobian is inversely proportional to class selectivity for ResNer18 88 (Figure 2cp, ResNet50 (Figure/AT (ATcc), and ResNer20 (Figure 21ch, indicating that distributed semantic representations are more vuinerable to worst-case perturbation because they are less stable than sparse semantic representations.
4.3 VARIABILITY OF THE INPUT-UNIT GRADIENT ACROSS SAMPLES AND UNITS
We observed that the input-output Jacobian is proportional to worst-case vulnerability and inversely proportional to class selectivity, but focusing on input-output stability potentially overlooks phenom- ena present in hidden layers and units. If class selectivity imparts worst-case robustness by making individual units less reliably perturbable- because each unit is highly tuned to a particular subset of images- thege we should expect to see more variation across input-unit gradients for units in high-selectivity networks compared to units in low-selectivity networks. Alternatively, worst-case ro- bustness in high-selectivity networks could be achieved by reducing both the magnitude and variation of units' perturbability, in which case we would expect to observe lower variation across input-unit gradients for units in high-selcttivity networks compared to low-selectivity networks.
We quantified variation in unit perturbability using the coefficient of variation of the input-unit gradient across samples for each unit (CV.; Approach 3) The CV is measure of variability that normalizes the standard deviation of a quantity by the mean. A large CV indicates high variability, a small CV indicates low variability. To quantify variation in perturbability across units, we computed the CV across units in each layer, (CV; Approach
We found that units in high-selectivity networks exhibited greater variation in their perturbability than units in low-selectivity networks, both within individual units and across units in each layer. This effect was present in both ResNet18 trained on Tiny ImageNet (Figure and ResNet20 trained on CIFARIO 10 (Figure A18). although the effect was less consistent for across-unit variability in later layers in ResNet18 (Figure 3b). Interestingly, class selectivity affects both the numerator (a) and denominator (f)) of the CV calculation for both the CV across samples and CV across units (Appendix A.S). These results indicate that that high class selectivity imparts worst-case robustness by increasing the variation in perturbability within and across units, while the worst-case vulnerability associated with low class selectivity results from more consistently perturbable units. It is worth noting that the inverse can be stated with regards to everage-case robustness: low variation in perturbability both within and across units in sew-sttecyivity networks is associated with robustness to average-case perturbations, despite the these units (and networks) being more perturbable on average.
7

---

Under review as a conference paper at ICLR 2021
4.4
DIMENSIONALITY IN EARLY LAYERS PREDICTS PERTURBATION VULNERABILITY
a)
Prior research has elucidated the mechanisms of worst- case perturbations using the framework of dimensional- ity (eaggeberg et al. 2019 Nayebi and Ganguli| [2017] (Sanyal et al. 2020). Investigating the dimensionality of the changes induced by average-case and worst-case per- turbations could reveal a common factor linking them to each other and to class selectivity.
10gus Untu Sodeetk Verebiityy
107
33-5
One possible explanation for the discrepancy between class selectivity's impact on worst-case and average-case corruptions is that different corruption types impact rep- resentations with varying dimensionalities. For example, if only a few neurons are needed to change a network's ' decision, the dimensionality of the change in the represen- tation due to perturbation might be very low, as only few units need to be modified. We thus measured dimension- ality using a straightforward, linear method: we applied Principal Component Analysis (PCA) to the activation matrices of each layer in our networks and computed the number of components necessary to explain 95% of the variance (and replicated our results with different variance thresholds; Appendix A.1.4. Appendix (.9). We first ex- amined the dimensionality of the representations of the clean test data. If the sparsity of semantic representations is reflected in dimensionality, then networks with more class selectivity should have lower- sewer-dimensionll represen- tations than networks with less class selectivity. Alter- natively, if high-selectivity representations are of similar dimensionality to low- oww-electivvty representations thhougg perhaps occupying different sub-spaces then dimension-
Layer
b)
20P
Verueality Lays (CV)
In UROUAATTO Gredeen
30
14
16
Layer
Figure 3: Class selectivity causes higher variation in perturbability within and across units (a) Coefficient of variation of input-unit gradient for each unit (CVs; see Approach [] y-axis) as a function of layer (x-axis). (b) CV of input-une unit gradient across units in a layer (CV;: y-axis)
function of layer (x-axis). Results shown are for ResNet18 trained on Tiny ImageNet. Shaded regions- = 95% confidence intervals of the mean. See Appendix A for results for ResNet20 trained on CIFARIO.
ality would be unaffected by class selectivity.
We found that the sparsity of DNN's semantic representations corresponds directly to the dimen- sionality of those representations. Dimensionality is inversely proportional to class selectivity in early ResNet18 layers (Slayer 9; Figure 4ab. and across all of ResNet20 (Figure (A2Id). Networks with higher class selectivity tend to have lower dimensionality, and networks with lower class selectivity tend to have higher dimensionality. These results show that the sparsity of network's semantic representations is indeed reflected in those representations' dimensionality.
We next examined the dimensionality of perturbation induuced changes in representations by subtract- ing the perturbed activation matrix from the clean activation matrix and computing the dimensionality of this "difference matrix" (see Appendix A.1.44.. Intuitively, this metric quantifies the dimensionality of the change in the representation caused by perturbing the input. If it is small, the perturbation impacts fewer units, while if it is large, more units are impacted. Interestingly. we found that the dimensionality of the changes in activations induced by both average-case (Figure 4b) and worst-case perturbations (Figure 4c) was notably higher for networks with reduced selectivle,,,,,,, suggesting that decreasing class selectivity causes changes in input to become more distributed.
We found that the activation changes caused by average-cass perturbations are higher-dimensien than the representations of the clean data in both R (compare Figures/4bJande 4a) and ResNer20 (Figures A21e and A21d)) and that this effect inversely proportional to class selectivity (Figures 4b and A21c): the increase in dimensionality from average-case perturbations low-sclectivity networks than in high-sclectivity networks. These results indicate that class selectivity not only predicts the dimensionality of representation, but also the change in dimensionality induced by an average-case perturbation.
more pronounced in
Notably, however, the increase in early-layer dimensionality much larger for worst-case perturba- tions than average-case perturbations (Figure 4c] Figure These results indicate that, while the changes in dimensionality induced by both naturalistic and adversarial perturbations are proportional
8

---

Under review as a conference paper at ICLR 2021
a)
Clare Selectiity tegdartoadon Scale
xe na ex a2 2* na 26
03
0.
0.
82
60
Figure 4: Dimensionality in early layers predicts worst-case vulnerability. (a) Fraction of dimensionality (y-axis; see Appendix A.1.4) as function of layer (x-axis). (b) Dimensionality of difference between clean and average-case perturbation activations (y-axis) as function layer (x-axis). (c) Dimensionality of difference between clean and worst-case perturbation activations (y-axis) as a function of layer (x-axis). Results shown are for R8 trained on Tiny ImageNet. See Appendix A21 for ResNet20.
to the dimensionality of the network's representations, these changes do not consistently project onto coding-relevant dimensions of the representations. Indeed, the larger change in early-layer dimensionality caused by worst-case perturbations likely reflects targeted projection onto coding- relevant dimensions and provides intuition as why low-selectivity networks are more susceptible to worst-case perturbations.
Hidden layer representations in DNNs are known to lie on non-linear manifolds that are of lower dimensionality than the space in which they re embedded (Goodfellow et al.] [0066] Ansuini et al.) 2019). Consequently, linear methods such as PCA can provide misleading estimates of hidden layer dimensionality. Thus we also quantified the intrinsic dimensionality (ID) of each layer's representations (see Appendix (A.1.4). Interestingly, the results were qualitatively similar to what observed when examining linear dimensionality (Figure (Fig in both ResNet18 trained on Tiny ImageNet (Figure A22a A22c) and ResNet20 trained on CIFARIO (Figure A22d)A221) Thus both linear and non-linear measures of dimensionality imply that representational dimensionality may present a trade-off between worst-case and average-case perturbation robustness.
5 DISCUSSION
Our results demonstrate that changes in the sparsity of semantic representations, as measured with class selectivity, induce a trade-off between robustness to average-case vs. worst-case perturbations: highly-distributed semantic representations confer robustness to average-cas perturbations, but their increased dimensionality and consistent perturbability result in vulnerability to worst-case perturbations. In contrast, sparse semantic representations yield low-dimensional representations and croonsiteetlly-peturbbbbi units, imparting worst-case robustness. Furthermore, the dimensionality of the difference in early-layer activations between clean and perturbed samples is larger for worst-case perturbations than for average-case perturbations. More generally, our results link average- and worst-case perturbation robustness through class selectivity and representational dimensionality.
We hesitate to generalize too broadly about our findings, as they are limited to CNNs trained on image classification tasks. It is possible that the results we report here specific to our models and/or datasets, and also may not extend to other tasks. Scaling class selectivity regularization to datasets with large numbers of classes also remains an open problem (Leavitt and Morcos 2020
Our findings could be utilized for practical ends and to clarify findings in prior work. Relevant to both of these issues is the task of adversarial example detection. There conflicting evidence that intrinsic dimensionality can be used to characterize or detect adversarial (worst-case) samples (Ma) al. 2018 Lu et al. 2018). The finding that worst-case perturbations cause a marked increase in both intrinsic and linear dimensionality indicates that there may be merit in continuing to study these quantities for use in worst-case perturbation detection. And the observation that the causal relationship between class-selectiviny and worst- and average-case robustness is bidirectional helps clarify the known benefits of sparsity (Madry et al.) 2018: [Bldaa et al., 2020 Ye al.] 2018 Guo et al. 2018 Dhillon et al. 2018)) and dimensionality (Langeberg et al. 2019 Sanyal et al. 2020 Nayebi and Ganguli 2017 7) on worst-case robustness. furthermore raises the question of whether enforcing low-dimensional representations also causes class selectivity to increase.
Our work may also hold practical relevance to developing robust models: class selectivity could be used as both metrie for measuring model robustness and method for achieving robustness (via regularization). We hope future work will more comprehensively assess the utility of class selectivity as part of the deep learning toolkit for these purposes.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Rana Ali Amjad, Kairen Liu, and Bernhard C. Geiger. Understanding Individual Neuron Importance http://rgg/b///.8...00
Using Information Theory. April 2018.
Alessio Ansuini, Alessandro Laio, Jakob И Macke, and Davide Zoccolan. Intrinsic di- mension of data representations in deep neural networks. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'textquotesingle Alché-Buc, E. Fox. and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 6111-6123 Curran Associates, Inc.,
2019. URL httphtttpp/pppprsppppcpppppppppp B www.::iniiiidmmmiinn-n-ee.ee data-
Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated Gradients Give a False Sense of Secu- rity: Circumventing Defenses to Adversarial Examples. In International Conference on Machine
Learning. pages 274-283. PMLR, July 2018. URL pttp:///procecchnggs...pp.pepe
130/athalyel8a.tte.
ISSN: 2640-3498.
Emilio Balda, Niklas Koep, Arash Behboodi, and Rudolf Mathar, Adversarial Risk Bounds through Sparsity based Compression. In International Conference Artificial Intelligence and Statistics, pages 3816-3825. PMLR, June 2020. URL. htttp://proceed ngs.m
r.press/vi00 n.p..r..rssss..sssiiii......
ba 23lda200.htm
ISSN: 2640-3498.
David Bau, Jun- Yan Zhu, Hendrik Strobelt, Agata Lapedriza, Bolei Zhou, and Antonio Tor- ralba. Understanding the role of individual units in deep neural network. Proceedings of the National Academy of Sciences, September 2020. ISSN 0027-8424, 1091-6490. doi: 10. 10.1073/pnas.110 http:/mmn...9777111111 URL WtRL:ttps:// http://wwwwppp.aee wwtppp/www......poeooooooooeeeee cont 31/1907375117 Publisher: National Academy of Sciences Section: Physical Sciences.
Nicholas Carlini and David Wagner. Adversarial Examples Are Not Easily Detected: Bypassing Ten Detection Methods. In Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security, AlSec '17, pages 3-14, Dallas, Texas, USA, November 2017a. Association for Computing Machinery. ISBN 978-1-4503-5202--44 doi: 10. 1421144//122572.3111
URL https!
//dbl.org/10.115//112333
Nicholas Carlini and David Wagner. Towards Evaluating the Robustness of Neural Networks. In 2017 IEEE Symposium Security and Privacy (SP), pages 39-57, May 2017b. doi: 10.1109/ SP.2017.49 49. ISSN: 2375-1207.
Fahim Dalvi, Nadir Durrani, Hassan Sajjad, Yonatan Belinkov, Anthony Bau, and James Glass. What Is One Grain of Sand in the Desert? Analyzing Individual Neurons in Deep NLP Models. Proceedings of the AAAI Conference on Artificial Intelligence, 33(01):6309-6311 July 2019a.
ISSN 2374-3468.
doi: 10.16099a2ai.v333000 333016399
URL https::/aaaa..rgg////
Fahim Dalvi, Avery Nortonsmith, Anthony Bau, Yonatan Belinkov, Hassan Sajjad, Nadir Durrani, and James Glass. NeuroX: Toolkit for Analyzing Individual Neurons in Neural Networks.
Proceedings of the AAAI Conference on Artificial Intelligence. 33(01))9851-9822 July 2019b.
ISSN 2374-3468. doi: 10. 13.11/00/aaaa.33301..30000
StLbttps: htttp:::www..aa.....p. /ww.aaaa.......//j/////////
Kedar Dhamdhere, Mukund Sundararajan, and Qiqi Yan. How Important is Neuron. In International Conference on Learning Representations, 2019. URI.
ew.ne//
Forum? d=Sy KooOcKm.
Guneet S. Dhillon, Kamyar Azizzadenesheli, Zachary C. Lipton, Jeremy D. Bernstein, Jean Kossaifi, Aran Khanna, and Animashree Anandkumar. Stochastic Activation Pruning for Robust Adversarial Defense. February 2018. /ttla//ppp////eeeeeeeeeeee///aap
Jonathan Donnelly and Adam Roegiest. On Interpretability and Feature Representations: An Analysis of the Sentiment Neuron. In Leif Azzopardi, Benno Stein, Norbert Fuhr, Philipp Mayr, Claudia Hauff, and Djoerd Hiemstra, editors, Advances in Information Retrieval, Lecture Notes in Computer Science, pages 795-802, Cham, 2019. Springer International Publishing. ISBN 978-3-030-15511-88 doi: 10. 1000999788-0001577122-
10

---

Under review as a conference paper at ICLR 2021
Harris Drucker and Yann Le Cun. Improving generalization performance using double backpropaga- tion. IEEE Transactions on Neural Networks, 3(6):991-997, 1992.
Dumitru Erhan, Yoshua Bengio, Aaron C. Courville, and Pascal Vincent. Visualizing Higher-Layer Features of Deep Network. 2009.
Brian Everitt. The Cambridge Dictionary of Statistics, volume 106. Cambridge University Press, Cambridge, 2002.
Elena Facco, Maria d'Errico, Alex Rodriguez, and Alessandro Laio. Estimating the intrin- sic dimension of datasets by minimal neighborhood information. Scientific Reports, 7 (1):12140, December 2017. ISSN 2045-2322.
doi: 11011/////11118000-111877333 URL http:://www..ttttrrer.cco//rrrrs
Li Fei-Fei, Andrej Karpathy, and Justin Johnson. Tiny imagenet visual recognition challenge, 2015.
URL htip:ipppss//Ennnnnimaaaeeeeeeep
Nicolas Ford, Justin Gilmer, Nicholas Carlini, and Ekin Cubuk. Adversarial Examples Are Nat- ural Consequence of Test Error in Noise. In International Conference Machine Learning,
pages 2280-2289. PMLR, May 2019. URL
gi meml 9a.
ISSN: 2640-3498.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. September 2018. URL http:://openeeriiew..tttt
[orrm??dd-Byhh3jOXX
Justin Gilmer, Ryan P. Adams, Ian Goodfellow, David Andersen, and George E. Dahl. Motivating the Rules of the Game for Adversarial Example Research. July 2018. URL httpss://azxiv.rg//
abs/1807.06 772022
Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning, volume 1.
MIT press Cambridge, 2016.
Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and Harnessing Adver- sarial Examples. urXiv:1412.655 [cs, stat], March 2015. URL
1412.6572
arXiv: 1412.6572.
Yiwen Guo, Chao Zhang, Changshui Zhang, and Yurong Chen. Sparse DNNs with Improved Adversarial Robustness. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Binn Resa-Bianchi, and Garnett, editors, Advances Neural Information Processing Systems 31, pages 242 251. sparse
Curran Associates, Inc., 2018. URL http:
dnns senns-with-improoee http::/-mmppooeddeeeerrrrraaaaaa www.aaavveaairrr
robustness.pd otrprp/eeeec......ararrrrrraaaar
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition,
pages 770-778, 2016.
Dan Hendrycks and Thomas G. Dietterich. Benchmarking Neural Network Robustness to Common Corruptions and Surface Variations. arXiv:1807 001697 [cs, stat], April 2019. URL http://
Doxiv.org/abs/lie /ttp////0bb110000177777
arXiv: 1807.01697.
Dan Hendrycks, Norman Mu, Ekin D. Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshmi- narayanan. AugMix: A Simple Data Processing Method Improve Robustness and Uncertainty. arXiv:1912.0271 [cs, stat], February 2020a. URL httttttp//aaa UR//appp//aarxvocr/////////22///
arXiv: 1912.02781.
Dan Hendrycks. Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural Adversarial Examples. arXiv: 11001110007711 174 [cs, stat], January 2020b. URL hatt::/aax.v.ozg//
1907.07174 74, arXiv: 1907.07174.
Judy Hoffman, Daniel A. Roberts, and Sho Yaida. Robust Learning with Jacobian Regularization.
Xrxiv: 008.02729 [cs, stat], August 2019. URL http://aaaaavorr//aa///10...0
arXiv: 1908.02729.

---

Under review as a conference paper at ICLR 2021
Ruitong Huang, Bing Xu, Dale Schuurmans, and Csaba Szepesvari. Learning
Leamingg with Strong Adversary. arXiv:
arXiv:1511 1.03034 [cs], January 2016. URL httpp//aaxxi..rrrr/as//11111...1 1511.03034.
Yerlan Idelbayev. wwwmaster/yytoocchhhhhooooo resphheeeeette__eeeeeeeeeeeeeeee cifar10, January 2020. URL Uttnttp::///iihubbccmm
original-date: 2018-01-15709:50:566
E R Kandel, J H Schwartz, and Jessica Chao. Principles of neural science. McGraw-Hill, New York, 2000.
Andrej Karpathy, Justin Johnson, and Li Fei-Fei. Visualizing and Understanding Recurrent Networks. In International Conference on Learning Representations, page 11, 2016.
Alex Krizhevsky. Learning Multiple Layers of Features from Tiny Images. Technical report, 2009.
Alexey Kurakin, lan J. Goodfellow, and Samy Bengio. Adversarial examples in the physical world. SioufaIIi
November 2016.
Alexey Kurakin, lan J. Goodfellow, and Samy Bengio. Adversarial Machine Learning at Scale. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France.
April 24-26, 2017. Conference Track Proceedings. OpenReview.net, 2017. URL https wwopenrevie..ee/foooroo www.ret//oorummmdddommmmmee
Peter Langeberg, Emilio Rafael Balda, Arash Behboodi, and Rudolf Mathar. On the Effect of Low-Rank Weights Adversarial Robustness of Neural Networks. arXiv: 1XXv:1901.11077 [cs, stat], January 2019. URL http:::///aaxvv..//aaaaas/
arXiv: 1901.10371.
Matthew L. Leavitt and Ari Morcos. Selectivity considered harmful: evaluating the causal impact of class selectivity in DNNs. arXiv:2003.01262 [cs, q-bio, stat], March 2020. URL http
//axxi.org/abs: http://////ab///0000000005 2003.0126
Xrxi:: 2003.01262.
Elizaveta Levina and Peter Bickel. Maximum Likelihood Estimation of Intrinsic Dimension. In L. Saul, Y. Weiss, and L. Bottou, editors, Advances in Neural Information Processing Systems,
volume 17, pages 777-784. MIT Press, 2005. URL http://pps///ppeeeeeeeoeeeooeee
Peter E. Lillian, Richard Meyes, and Tobias Meisen. Ablation of a Robot's Brain: Neural Networks
Under a Knife. December 2018.
Lu Lu, Yeonjong Shin, Yanhui Su, and George Em Kamiaaakis.. Dying ReLU and Initialization: Theory and Numerical Examples. arXiv:1903. 1903.063333 [cs, math, stat], November 2019. URL
arXiv: 1903.06733.
Pei-Hsuan La, Pin-Yu Yu Chen, and Chia-Mu Yu. On the Limitation of Local Intrinsic Dimensionality for Characterizing the Subspaces of Adversarial Examples. February 2018. URL https
Xingjun Ma, Bo Li, Yisen Wang, Sarah M. Erfani, Sudanthi Wijewickrema, Grant Schoenebeck, Dawn Song. Michael E. Houle, and James Bailey. Characterizing Adversarial Subspaces Using Local Intrinsic Dimensionality. February 2018. URL http:::::///peneeeeeee..ttttttt.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards Deep Learning Models Resistant to Adversarial Attacks. February 2018. httpst
Sopenneeview.net//orumzzzzz
Richard Meyes, Melanie Lu, Constantin Waubert de Puiseau, and Tobias Meisen. Ablation Studies in Artificial Neural Networks. 1rXiv:1901.08644 [cs, b-bio], February 2019. URL http:// arxiv.org/ abs /abps//99..0844
arXiv: 1901.08644.
Ari S. Morcos, David G. T. Barrett, Neil C. Rabinowitz, and Matthew Botvinick. On the importance of single directions for generalization. In International Conference on Learning Representations. 2018. httl.hhttts://peeeee ww..t///wwiieeeteeeee.e for.mmuttffuuuu wettorrummiiriiiil d= wttp:ummmmirrriiuuuooooo
12

---

Under review as a conference paper at ICLR 2021
Seil Na, Yo Joong Choe, Dong-Hyun Lee, and Gunhee Kim. Discovery of Natural Language Concepts in Individual Units of CNNs. In International Conference on Learning Representations. 2019.
Aran Nayebi and Surya Ganguli. Biologically inspired protection of deep networks from adversarial attacks, arXiv: 170.00920 [cs. q-bio, stat], March 2017. URL httt:://axx...or//alll
1703.09202 arxiv: 1703.09202.
Roman Novak, Yasaman Bahri, Daniel A. Abolañia, Jeffrey Pennington, and Jascha Sohl-Dicksteii Sensitivity and Generalization Neural Networks: an Empirical Study, arXiv: aet:::802.8888 [es, stat], June 2018. URL. http::pp/aaa...oo/r//a//////////
arXiv: 1802.08760.
Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Feature Visualization. Distill, 2(1 1(17)) URL uttpttts://asssl1.ppp
November 2017. ISSN 2476-0777. doi:
2017/faauure-vssuallaatt....
Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom In: An Introduction Circuits. Distill, 5()))0000000000000 March 2020. ISSN 2476-0757. doi: 1/33t///iisilll.000224
httts::::: http///istillll// pub. wttppppppp///pcppppp rcui ts www.tc/tttccoca./et..rrr zoom
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An Imperative Style, High-Performant Learning Library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. Alché Buc. Fox, and R. Gar- nett, editors, Advances in Neural Information Processing Systems 32, pages 8024-8035, Curran Associates, Inc., 2019. URL http:////////////o.aaaaaaa////aa
Deep
Alec Radford, Rafal Jozefowicz, and Ilya Sutskever. Learning Generate Reviews and Discovering Sentiment. 1eXiv:174.0014440 [cs] April 2017. URL/tttp://aaxivorrr/a////TOO... arXiv: 1704.01444.
Ivet Rafegas, Maria Vanrell, Luis A. Alexandre, and Guillem Arias. Understanding trained CNNs by indexing neuron selectivity. Pattern Recognition Letters, page $0167865519330299, October 2019. ISSN 01678655. doi: 1702.00382
www.aa..01..11..... 10.013. URL httl:tttt///xxxv.r///////
arXiv: 1702.00382.
Salah Rifai, Pascal Vincent, Xavier Muller, Xavier Glorot, and Yoshua Bengio. Contractive auto- encoders: explicit invariance during feature extraction. In Proceedings of the 28th International Conference on International Conference on Machine Learning, ICML' 1, pages 833 840, Belle- vue, Washington, USA, June 2011. Omnipress. ISBN 978- 778-1-450330011-
Amartya Sanyal, Varun Kanade, Philip H. S. Torr, and Puneet K. Dokania. Robustness via Deep Low- Rank Representations. 10Xiv:1804.07000 [cs, stat], February 2020. URL hRRlhttt:///axxv.oo/ abs/1804.07090
arXiv: 1804.07090.
Alexandru Constantin Serban, Erik Poll, and Joost Visser. Adversarial Examples A Complete Characterisation of the Phenomenon. arXiv: 1810.01 185 [es], February 2019. URL http://
brxivvorg/abs//e 010.01885
arXiv: 1810.01185.
Charles Sherrington. The integrative action of the nervous system. The integrative action of the nervous system. Yale University Press, New Haven, CT, US. 1906. doi: 10. (0.1037/13798-000
Jure Sokolic, Raja Giryes, Guillermo Sapiro, and Miguel R. D. Rodrigues. Robust Large Margin Deep Neural Networks. IEEE Transactions on Signal Processing, (5(16):426544288) August 2017. ISSN 1053-587X, 1941 10476. doi: 10. (99/TS8..017.2780039. URL http:::/ttai...orrrrr/ Abs/1605.08250 arXiv: 1605.08254.
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. December 2013. URL https
7 //arxiv.org/ab/////// 2tt..///g///b//11111999199
13

---

Under review as a conference paper at ICLR 2021
Yusuke Tsuzuku and Issei Sato. On the Structural Sensitivity of Deep Convolutional Networks the Directions of Fourier Basis Functions. In 2019 IEEE/CVF Conference on Computer Vision and
Pattern Recognition (CVPR), pages 5 51-60, Long Beach, CA, USA, June 2019. IEEE. ISBN 978- 1-72813-293-8. doi: 10.1109/CVPR 2019000014.
URL httph:tt///// //ee/e//eeeepppee www.::owwo....e..rr......
(eccumnnt/89508851
Igor Vasiljevic, Ayan Chakrabarti, and Gregory Shakhnarovich. Examining the Impact of Blur on Recognition by Convolutional Networks. November 2016.
111100560v2
Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Courna- peau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nel- son, Eric Jones, Robert Kern, Eric Larson, C. J. Carey, Ilhan Polat, Yu Feng. Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quin- tero, Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 0 Contributors. SciPy I.O-Fundamental Algorithms for Scientific Com- puting in Python. arXiv: 1907. 10121 [physics], July 2019. URL
1907.10121 arXiv: 1907.10121.
David Warde-Farley and Ian Goodfellow. Adversarial Perturbations of Deep Neural Networks. In Perturbations, Optimization, and Statistics, pages 311-342 MITP, 2017. ISBN 978-0-262-337993 9. /tLp:tttts///eeepp.oooo http://ieeeexpoo..eee..rrr. /tw.//ottttttneeeeeeee Conference Name: Per- turbations, Optimization, and Statistics.
Michael Waskom, Olga Botvinnik, Drew O'Kane, Paul Hobson, Saulius Lukauskas, David C. Gemperline, Tom Augspurger, Yaroslav Halchenko, John B. Cole, Jordi Warmenhoven, Ju- lian de Ruiter, Cameron Pye, Stephan Hoyer, Jake Vanderplas. Santi Villalba, Gero Kunter, Eric Quintero, Pete Bachant, Marcel Martin, Kyle Meyer, Alistair Miles, Yoav Ram, Tal Yarkoni. Mike Lee Williams, Constantine Evans, Clark Fitzgerald, Brian, Chris Fonnesbeck, Antony Lee, and Adel Qalieh. mwaskom/seaborn: v0.8.1 (September 2017), September 2017. URL https:
Shaokai Ye, Siyue Wang, Xiao Wang, Bo Yuan, Wujie Wen, and Xue Lin. Defending DNN Adversarial Attacks with Pruning and Logits Augmentation. February 2018. URL https
Dong Yin, Raphael Gontijo Lopes, Jon Shlens, Ekin Dogus Cubuk, and Justin Gilmer. A Fourier Perspective on Model Robustness in Computer Vision. H. Wallach, H. Larochelle, A. Beygelzimer, F. ditextquotesingle Alché-Bue Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 13276-13286. Curran Associates, Inc.. 2019. URL http://ppppp/ppperrsppp.p.c////p / -ttp:/aapppeeaa----------------- wowaaaaooorrrereeeeeeeeeeeeeeeee iverpeeeeiiiieon ve-on- model poodell-roouttteesss---ooootteee
Matthew Zeiler and Rob Fergus. Visualizing and Understanding Convolutional Networks. In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars, editors, Computer Vision - ECCV 2014, pages 818-833, Cham, 2014. Springer International Publishing. ISBN 978-3-319-11590011
Vision-ECCV
Stephan Zheng, Yang Song, Thomas Leung, and Ian Goodfellow. Improving the Robustness of Deep Neural Networks via Stability Training. April 2016. CRLLttts::// http:///xxxvv.or/aabb./ /LL.ppps://axx...or//////
1604.04326v1
B. Zhou, D. Bau, A. Oliva, and A. Torralba. Interpreting Deep Visual Representations via Network Dissection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(9):113122155, September 2019, ISSN 1939-3539. doi: 10.1109/TPAM. 2011 201.228879.
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Object Detectors Emerge in Deep Scene CNNs. In International Conference on Learning Representations, April 2015. B.RR/tttp://axxi.orr///// /a0os./1412..555
arXiv: 1412.6856.
Bolei Zhou, Yiyou Sun, David Bau, and Antonio Torralba. Revisiting the Importance of Individual Units in CNNs via Ablation. 3XXi:188060293 [cs], June 2018. URL httt::://tti...///////
abs/1806.02891
arXiv: 1806.02891.
14

---

Under review as a conference paper at ICLR 2021
A APPENDIX
A.1 DETAILED APPROACH
Unless otherwise noted: all experimental results were derived from the corrupted or adversarial test set with the parameters from the epoch that achieved the highest clean validation set accuracy over the training epochs; 20 replicates with different random seeds were run for each hyperparameter set; error bars and shaded regions denote bootstrapped 95% confidence intervals; selectivity regularization was not applied to the final (output) layer, nor was the final layer included in any of our analyses.
A.1.1 MODELS
All models were trained using stochastic gradient descent (SGD) with momentum = 0.9 and weight decay 0.0001. The maxpool layer after the first batchnorm layer in ResNet18 (see [e al. (2016) was removed because of the smaller size of Tiny ImageNet images compared to standard ImageNet images (64x64 vs. 256x256, respectively). ResNet18 and ResNet50 were trained for 90 epochs with minibatch size of 4096 (ResNet1 8) or 1400 (ResNet50) samples with learning rate of 0. multiplied (annealed) by 0.1 at epochs 35, 50, 65, and 80.
ResNer20 (code modified from Idelbayev (2000) were trained for 200 epochs using minibatch size of 256 samples and a learning rate of 0.1, annealed by 0.1 at epochs 100 and 150.
A.1.2 DATASETS
Tiny Imagenet (Fei-Fei et al. 2015) consists 500 training images and 50 images for each of its 200 classes. We used the validation set for testing and created new validation set by taking 50 images per class from the training set, selected randomly for each seed. We split the 50k CIFAR 10 training samples into a 45k sample training set and 5k validation set, similar to our approach with Tiny Imagenet.
All experimental results were derived from the test set with the parameters from the epoch that achieved the highest validation set accuracy over the training epochs. 20 replicates with different random seeds were run for each hyperparameter set. Selectivity regularization was not applied the final (output) layer, nor was the final layer included any of our analyses.
CIFARIOC consists of dataset in which 19 different naturalistic corruptions have been applied to the CIFAR 10 test set at 5 different levels of severity. Tiny ImageNetC also has 5 levels of corruption severity, but consists of 15 corruptions.
We would like to note that Tiny ImageNetC does not use the Tiny ImageNet test data. While the ccooppiig and scaling images
two datasets were created using the same data generation from the same 200 ImageNet they differ in the specific ImageNet images they use. It is possible that the images used to create Tiny ImageNetC are out-of-distribution with regards to the Tiny ImageNet training data, in which case our results from testing Tiny ImageNetC actually underestimate the corruption robustness of our networks. The creators of Tiny ImageNetC kindly provided the clean (uncorrupted) Tiny ImageNetC data necessary for the dimensionality analysis, which relies on matches corrupted and clean data samples.
A.1.3 SOFTWARE
Experiments were conducted using Py ToTch (Paszke et al. 2019). analyzed using the SciPy ecosystem (Virtanen al., 2019) and visualized using Seaborn (Waskom et al. (017))
1.1.4 QUANTIFYING DIMENSIONALITY
We quantified the dimensionality of layer's representations by applying PCA to the layer's activation matrix for the clean test data and counting the number of dimensions necessary explain 95% of the variance, then dividing by the total number of dimensions (i.e. the fraction of total dimensionality; also replicated our results using the fraction of total dimensionality necessary to explain 90% and 99% of the variance). The same procedure was applied to compute the dimensionality of perturbation- induced changes in representations, except the activations for perturbed data set were subtracted
15

---

Under review as a conference paper at ICLR 2021
b)
d)
Figure A1: Example naturalistic corruptions from the Tiny ImageNetC dataset. (a) Clean (no corruption). (b) Brightness. (c) Contrast. (d) Elastic transform. (e) Shot noise. All corruptions are shown at severity level S/S.
from the corresponding clean activations prior to applying PCA. For average-case perturbations, we performed this analysis for every corruption type and severity, and for the worst-case perturbations we used PGD with 40 steps.
Hidden layer representations in DNNs are known to lie on non-linear manifolds that are of lower dimensionality than the space which they're 'r embedded (Goodfellow al. 2016 Ansuini et al. 2019). Consequently, linear methods such as PCA can fail to capture the "intrinsic" dimensionality of hidden layer representations. Thus we also quantified the intrinsic dimensionality (ID) of each layer's representations using the method of (Facco et al. 2017). The method, based on that of Levina and Bickel (200)), estimates ID by computing the ratio between the distances to the second and first nearest neighbors of each data point. We used the implementation of Ansuini et al. (2019). Our procedure was otherwise identical as when computing the linear dimensionality: we computed the dimensionality across all test data for each layer, then divided by the number of units per layer. We then computed the dimensionality of perturbation-indie innduuee--dddeed changes in representations, except the activations for a perturbed data set were subtracted from the corresponding clean activations prior to computed ID. For average-case perturbations, we performed this analysis for every corruption type and severity, and for the worst-case perturbations we used PGD with 40 steps.
A.2 EFFECTS OF CLASS SELECTIVITY REGULARIZATION ON TEST ACCURACY
a)
b)
80
Scale in)
0.0
70
Test Accurecy
103.0
0.1
20.0
0.2
10.0
3.0
50
30
5.2
003
1000
20
002
10.0
-01
100.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.6
0.6
0.8
Mean Class Selectivity
Meain Closs Selectivity
Figure A2: Effects of class selectivity regularization test accuracy. Replicated as in Leavitt and Morcos 2020 (a) Test accuracy (y-axis) as function of mean class selectivity (x-axis) for ResNet18 trained on Tiny ImageNet. a denotes the sign and intensity of class selectivity regularization. Negative a lowers selectivity, positive increases selectivity, and the magnitude of a changes the strength of the effect. Each data point represents the mean class selectivity across all units in single trained model. (b) Same as (a), but for ResNer20 trained CIFAR10.
16

---

Under review as a conference paper at ICLR 2021
A.3 ADDITIONAL RESULTS FOR AVERAGE- CASE PERTURBATION ROBUSTNESS
Because modifying class selectivity can affect performance on clean (unperturbed) inputs (Leavitt) and Morcos (2020); Figure A2). it is possible that the effects observe of class selectivity perturbed test accuracy are not caused by changes in perturbation robustness per se, but simply by changes in baseline model accuracy. We controlled for this by normalizing each model's perturbed test accuracy by its clean (unperturbed) test accuracy. The results are generally consistent even after controlling for clean test accuracy, although increasing class selectivity does not cause the same deficits in measured using non-normalized perturbed test accuracy in ResNet18 trained on Tiny ImageNet (Figure A3ap. Interestingly, in ResNet20 trained on CIFARIO, normalizing perturbed test accuracy reveals more dramatic improvement in perturbation robustness caused by reducing class selectivity (Figure Ffg). The results for Resnet50 trained on Tiny ImageNet are entirely consistent between raw normalized measures (Figure A9b vs. Figure A9c)
b)
0.42
Corruption
Strongest
Corrupte Testt Accurasy
Clean ee8 Accuracy
0.40
20
0.38
0.36
0.34
-2.0 1100 -..0--.7 -04---0--002--2 -00 0.2 D.7 10 2.0
.6.7 Regularizttinn Scale (c)
Regularbation Scale 1o)
Figure A3: Controlling for clean test accuracy, and effect corruption severity across corruptions. (a) Corrupted test accuracy normalized by clean test accuracy (y-axis) as a function class selectivity regularization scale (a; x-axis). Negative lowers selectivity, positive a increases selectivity, and the magnitude of changes the strength of the effect. Normalized perturbed test accuracy appears higher in networks with high class selectivity (large a)), but this is likely due to floor effect: clean accuracy already much closer bound- chance- in networks with very high class selectivity, which may reflect different performance regime, making direct comparison difficult. (b) Mcan test accuracy across all corruptions (y-axis) a function of (x-axis) for different corruption severities (ordering along y-axis; shade of connecting line). Error bars indicate 95% confidence intervals of the mean.
lower
Class Selectivity Regi Reguldoitzation Scale
Baselie
High
10
Brightnees
Impults
Frunt
Izeg
Dafocue
Too
Zoom
Na50
Transfor
Noti:
Figure A4: Mean test accuracy across corruption intensities for each corruption type for ResNet18 tested on Tiny ImageNetC. Test accuracy (y-axis as function of corruption type (x-axis) and class selectivity regularization scale (o, color). Reducing class selectivity improves robustness against all 15/15 corruption types.
17

---

Under review as a conference paper at ICLR 2021
Regudariatio ontion
Comption Severity
-12.0
-20
0
40
Comuuted Text
20
Clean Test Accuracy
Figure A5: Trade-off between clean and perturbed test accuracy in ResNet18 tested on Tiny ImageNetC. Clean accuracy (X-axis) perturbed lest accuracy (y-axis) for different corruption severities (border color) and regularization scales (ax, color). Mean computed across all corruption types. Error bars 95% confidence intervals of the mean.
a)
40
Class Selectivity Seguderzatiio
30
42
Serersar
seiettiiy
Selcthety
Jeng
Spatter
Detccus
Couestar Fost Transfum
Saturate Brightness Show
Caueden
Motton
Hose
Noli
Nuse
b)
0.74
70
0.77
66
0.79
85
Conrpptd
64
0.79
0.78
Conuuttinn Severiy
50
#rakeet
.0.7 -0. .0.0 6.4 1.0
Regultrization Scale 65)
Regudtrization Scale
3.0
-3.2
1.0
Figure A6: Reducing class selectivity confers robustness to average-case perturbations in ResNet20 tested on CIFARIOC. (a) Test accuracy (y-axis) function of corruption type (x-axis), class selectivity regularization scale (a; color), and corruption severity (ordering along y-axis). Test accuracy is reduced proportionally corruption severity, leading to an ordering along the y-axis, with corruption severity (least severe) at top and corruption severity 5 (most severe) at the bottom. Negative a lowers selectivity, positive a increases selectivity, and the magnitude of a changes the strength of the effect (see Figure A2b and Approach B}. (b) Mean test accuracy across all corruptions and severities (y-axis) test accuracy normalized by clean test accuracy (y-axis) as a function of class selectivity regularization scale (o; x-axis). Negative a lowers selectivity, positive a increases selectivity, and the magnitude of a changes the strength of the effect. Normalized perturbed test accuracy appears higher in networks with higher class selectivity (larger a)), but this is likely due floor effect: clean test accuracy is already much closer to the lower bound chance--i networks with very high class selectivity, which may reflect different performance regime, making direct comparison difficult. (d) Mean test accuracy across all corruptions (y-axis) as function of a (x-axis) for different corruption severities (ordering along y-axis; shade of connecting line). Error bars indicate 95% confidence intervals of the mean.
function of (x-axis). (e) Corrupted
18

---

Under review as a conference paper at ICLR 2021
40
80
Tox Acuuac
10
10
50
Class Selectivity Rogui faction Scale 1ol
40
Seleciiity
Low
Sanel
18gh Selectivity
Zcom
Stortde
Glves
Soatter
Defonse
Faudie
Show
Gauesan
Controct
Fiie
Figure A7: Mean test accuracy on CIFARIOC. Test accuracy (y-axis) as function corruption type (X-axis) and class selectivity regularization scale (a, color). Reducing class selectivity improves robustness against 14/19 corruption types. Error bars 95% confidence intervals of the mean.
corruption intensities for each corruption type for ResNet20 tested
Scale
0,0
Seweriy
Computed Tex Sccussyy
1.0
0.4
60
10.0
40
20
68
100
Clean Tesil Accuracy
Figure A8: Trade-off between clean and corrupted test accuracy in ResNet20 tested CIFAR1OC. Clean test accuracy (x-axis) vs. corrupted test accuracy (y-axis) different comupttion severities (border color) and regularization scales (a, fill color). Mean is computed across all corruption types.
19

---

Under review as a conference paper at ICLR 2021
42
Class Selectivity Regulaioation Scale on)
Bassire
high
Sodl
BiaT
BAr
Nobe
B
Transfor
Bat
30
25
0.42
Comppted
0:40
25
16
20
0.36
-3.7 10.4 -0.2 0. 3.5
-10
-0.7
-0.4
-02
0.2 0.7
-0.7 -0.4 -3.2 0.0
0.2
C.4 1.0
Repulataatiio Scale lo)
Regularizaiin Scale 101
Regulaniztion Scale (0)
Figure A9: Reducing class selectivity confers robustness to average-case perturbations in ResNet50 tested on Tiny ImageNetC. (a) Test accuracy (y-axis) regularization scale (a; color), and corruption severity (ordering along y-axis). Test accuracy is reduced proportionally to corruption severity, leading to an ordering along the y-axis, with corruption severity (least severe) at the top and corruption severity 5 (most severe) at the bottom. Negative a lowers selectivity, positive a increases selectivity, and the magnitude of a changes the strength of the effect (see Approach 3). (b) Mcan test accuracy across all corruptions and severities (y-axis) normalized by clean test accuracy (y-axis) as a function class selectivity regularization scale (a; x-axis). Negative a lowers selectivity, positive a increases selectivity, and the magnitude of a changes the strength of the effect. (d) Mcan test accuracy across all corruptions (y-axis) as function of (x-axis) for different corruption severities (ordering along y-axis; shade of connecting line). Error bars indicate 95% confidence intervals of the mean. Note that confidence intervals are larger in part due a smaller sample size- only replicates per instead 20.
function of corruption type (x-axis). class selectivity
function of (x-axis). (c) Corrupted test accuracy
Class Selectirity Regulartzaiinn Scale tol
Lon
Sacsioe
lich
Hilf the
Ter ACUnoy
Brightrwaa
Pios
Frost
Contraat
Deloca
Coumilon
Zoort
Figure A10: Mean test accuracy across corruption intensities for each corruption type for ResNet50 tested on Tiny Imtg Test accuracy (y-axis) as function of corruption type (x-axis) and class selectivity regularization scale (a, color). Reducing class selectivity improves robustness against 14/15 corruption types. Error bars= = 95% confidence intervals of the mean. Note that confidence intervals are larger in part due to smaller sample size-only 5 replicates per instead of 20.
20

---

Under review as a conference paper at ICLR 2021
A.4 THE CAUSAL RELATIONSHIP BETWEEN CLASS SELECTIVITY AND AVERAGE-CAN ROBUSTNESS IS BIDIRECTIONAL
We found that regularizing decrease class selectivity causes robustness average-case perturbations. But is the converse also true? Does increasing robustness to average-case perturbations also cause class selectivity increase? We investigated this question by training with AugMix, a technique known to improve worst-case robustness (Hendrycks et al. 2020a). Briefly, AugMix stochastically applies a diverse set of image augmentations and uses a Jensen-
Divergence consistency loss. Our AugMix parameters were as follows: mixture width: 3; mixture depth: stochastic; augmentation probability: 1; augmentation severity: 2. We found that AugMix does indeed decrese the mean level of class selectivity across neurons in network (Figure A1l AugMix decreases overall levels of selectivity similarly to training with a class selectivity regularization scale of approximately a = -0.1 -02 in both ResNet18 trained on Tiny ImageNet (Figures and AAllb and ResNet20 trained on CIFARIO (Figures AlIc and ATId). These results indicate that the causal relationship between average-case perturbation robustness and class selectivity is bidirectional: not only does decreasing class selectivity cause average-case perturbation robustness to increase, but increasing average-case perturbation-rooustees, also causes class selectivity decrease.
1.0
b)
0.9
Regubrizztton
0.8
Class Selectiiiy hovex
0.8
0.7
0.6
Class Selectiviy
0.6
0.5
0.4
0.3
AugMix
0.2
0.0
8
10
12
14
16
2.0 AugMix
Layer
Regularization Scalle (a)
Regulbrization
d)
0.7
0.5
0.6
0.7
Class Selectivity Index
0.5
0.4
4.2
0.3
0.3
0.2
10 Layer
12
14
16
18
0.7 AugMix
Regularization
Figure A11: AugMix training causes class selectivity to decrease. (a) Mean class selectivity (y-axis) as function of layer (x-axis) and class selectivity regularization scale (a) when training with AugMix (hue; AugMix green) ResNet18 trained on Tiny ImageNet. Similar to (a), but mean class selectivity computed across an entire network (y-axis) for different class selectivity regularization scales (a) or when training with AugMix (x-axis; AugMix in green) for ResNet18 trained on Tiny ImageNet. The dashed line denotes the mean class selectivity for 0, the baseline for comparison to AugMix. (c) and (d), identical to (a) and (b), but for ResNet20 trained CIFAR10. Error hars denote 95% confidence intervals of the
A.5 WORST-CASE PERTURBATION ROBUSTNESS
We also confirmed that the worst-case robustness of sigh-selectivityy ResNet18 and ResNet20 net- works was not simply due gradient-masking (Athalye et al. 2018) by generating worst-case perturbations using each of the replicate models trained with no selectivity regularization (a = 0), then testing reeuuaiity-r models these samples. We found that high-selectivity models were less vulnerable to the a = samples than low-selectivity models for high-intensity perturbations (Appendix A14 indicating that gradient-masking does not fully account for the worst-case robustness of high-selectivity models.
21

---

Under review as a conference paper at ICLR 2021
b)
00
0015
2211
30.07
00.81
Figure A12: Reducing class selectivity increases worst-case perturbation vulnerability in ResNet20 trained on CIFARIO. (a) Test accuracy (y-axis) as a function perturbation intensity (c: x-axis) and class selectivity regularization scale (a; color) for the FGSM attack. (b) Test accuracy (y-axis) as function of adversarial optimization steps (X-axis) and (color) for the PGD attack. (c) Network stability, as measured with norm of the input-output Jacobian (y-axis) a function of (x-axis).
b)
6.0
305 ores nOES om ERR 21S 2ITS
5
18
20
25
28
Repraradion
Figure A13: Reducing class selectivity increases worst-case perturbation vulnerability in ResNet50 trained on Tiny ImageNet. (a) Test accuracy (y-axis) a function of perturbation intensity x-axis) class selectivity regularization scale (o; color)
FGSM attack. (b) Test accuracy (y-axis) as function of adversarial optimization steps (x-axis) and a (color) for the PGD attack. (c) Network stability, as measured with norm of the input-output Jacobian (y-axis) as a function of (x-axis).
a)
50
Accuncy
Casss Selectivit Regularization Scale (o)
Class Selectivity Reqularization Scale
0.7
10
tolffecent
"same
'sama replicate repilcate
ddiffrentt
replicate
20
60
80
100
20
40
60
80
100
Step
Step
Figure A14: Gradient masking does not fully account for worst-case perturbation robustness conferred by increased class selectivity, (a) Test Accuracy (y-axis) function adversarial optimization steps (x-axis) and class selectivity regularization scale (o: color) when tested on adversarial examples generated using a Because 20 replicate networks were trained for each value of a (see Approach 3 models trained with alpha = could be tested on adversarial examples generated from different replicate
0 network (" ("different replicate": solid line) adversarial samples generated from their parameters ("same replicate": dashed line). Data shown for ResNet18 trained Tiny ImageNet (b). Same (a), but for ResNer20 trained CTFAR10.
22

---

Under review as a conference paper at ICLR 2021
A.6 CLASS SELECTIVITY REGULARIZATION DOES NOT AFFECT ROBUSTNESS TO NATURAL ADVERSARIAL EXAMPLES
We also examined whether class selectivity regularization affects robustness to "natural" adversarial examples, images that are "natural, unmodified, real-world examples...selectee to cause a fixed model make mistake" (Hendrycks et al., 2020b). We tested robustness natural adversarial examples using ImageNet-A, dataset of natural adversarial examples that belong to ImageNet classes but consistently cause misclassification errors with high confidence (Hendryeks adapted ImageNet-A to our models trained on Tiny ImageNet (ResNet18 and ResNet50) by only testing on the 74 image classes that overlap between ImageNet-A and Tiny ImageNet (yielding total of 2957 samples), and downsampling the images to 64 64. Test accuracy was similar across all tested values of a for both ResNet18 8 (Figure A15a) and ResNet50 (Figure A indicating that class selectivity regularization may share some limitations with other methods for improving robustness, many of which also fail to yield significant robustness against ImageNet-A (Hendrycks
al. 2020b). We
al. 2020b).
a)
Tiny NetgeNet
50
40
Test Accuracy
30
Accuracy
1.9
20
1.8
10
-2.0-1.0 11.10-..7-004-033-022-0000 0.2
1.0
2.0
Regularization (o)
Regularization Scale (a)
b)
Iny_emggette
Test Accuracy
40
2.4
2.2
20
2.0
1.9
1.7
0
-1.0 1.0-----004 -04---3
0.1 0.1 0.3 0.4 0.7 1.0
-1.0 -0.7 -0.4 -0.3
-0.2
-0.1
0.0
0.1
0.2
0.3
0.4 0.7
1.0
Regularization Scale (a)
Regularization Scale (a)
Figure Class selectivity regularization (a) Test accuracy on ImageNet-A (y-axis) as function of regularization scale (a; x-axis and hue) for ResNet18 trained on Tiny ImageNet. The dashed line at the top of the plot denotes the mean test accuracy on Tiny ImageNet for models trained with a = 0, which serves trained on Tiny ImageNet. Error bars 95% confidence interval the mean. Note that confidence intervals are larger for ResNet50 in part due to a smaller sample size-only replicates per a instead 20, Chance 74 (1.35%)
not affect robustness to natural adversarial examples
bascline for comparison. (b) Identical to (a), but for ResNet50
23

---

Under review as a conference paper at ICLR 2021
A.7 THE CAUSAL RELATIONSHIP BETWEEN CLASS SELECTIVITY AND WORST-CASE ROBUSTNESS IS BIDIRECTIONAL
We observed that regularizing to increase class selectivity causes robustness to worst-case pertur- bations. But is the converse is also true? Does increasing robustness to worst-case perturbations cause class selectivity technique for improving worst-case robustness. PGD training applies the PGD method of sample perturbation (see Approach 3 to samples during training. We used the same parameters for PGD) sample generation when training our models as when testing (Approach 3). The number of PGD iterations controls the intensity the perturbation, and the degree of perturbation-roousteess
increase? We investigated this question using PGD training, a common
the trained model (Madry et al., 2018). We found that PGD training does indeed increase the mean level of class selectivity across neurons in a network, and this effect is proportional to the strength of PGD training: networks trained with more strongly strongly-perturbed perturbed samples have higher class selectivity (Figure ) Interestingly, PGD training also appear to cause units to die (Lu et al. (019), and the number of dead untis proportional the intensity of PGD training (Figures AT6bband/aa and [6e). Removing dead units, which have a class selectivity index of 0, from the calculation of mean class selectivity results in a clear, monotonic effect of PGD training intensity on class selectivity both ResNet18 trained on Tiny ImageNet (Figure 6 and ResNet20 trained on CIFARIO (Figure T These results indicate that the causal relationship between worst-case perturbation robustness and class selectivity is bidirectional: increasing class selectivity not only causes increased worst-case perturbation robustness, but increasing worst-case perturbation--obustees also causes increased class selectivity.
a)
FGD Steps
2:59
5.42
0:49
0.42
6:25
10
15
35
d)
0.20
e)
PGD
0.24
Pripetion Deed
0.22
1
84
6..0
0.19
022
FS0 Stops
15 10
PCD
Figure A16: PGD training causes class selectivity to increase. (a) Mean class selectivity (y-axis) function of PGD training steps (x-axis; hue) for ResNet18 trained Tiny ImageNet. The number PGD steps corresponds to the intensity of PGD training and worst-case robustness. (b) Proportion of dead units (y-axis) as a function of layer (x-axis) for different numbers of PGD training steps (hue). (c) Mean class selectivity (y-axis) function of PGD training steps (x-axis; hue) after removing dead units from calculation of mean. (d) (0. identical to (a) (c), but for ResNet20 trained CIFAR10.
24

---

Under review as a conference paper at ICLR 2021
A.8
STABILITY TO INPUT PERTURBATIONS IN UNITS AND LAYERS
Requbrtmatio
18-1
Standd Monterdrroaa Dreodde offeeee
Das prealee
Strnbo Divildoo
Bag aetraatttton
12 34
16
Figure A17: Mcan and standard deviation of input-unit gradient in ResNet18 trained on Tiny ImageNet. (a) unput-unit gradient (y-axis) function of layer (x-axis). (b) Standard deviation of input-unit gradient for each unit (ara: y-axis) as function layer (x-axis). (c) Standard deviation input-unit gradient across units a layer (ot y-axis) as a function of layer (x-axis). Shaded region = 95% confidence interval the mean.
a)
Class Solectvity Regularizarion
b)
101
10)
Insut Lnit Grostent Varibilyy.
200
10
15
10
Layer
Layer
Figure A18: Class selectivity causes higher variation in perturbability within and across units in ResNet20 trained on CIFAR10. (a) Coefficient of variation of input-unit gradient for each unit (CV,, see Approach y-axis) function of layer (x-axis). (b) CV of input-unit gradient across units in layer (CV; y-axis) as a function of layer (x-axis). Shaded regions 95% confidence interval of mean.
Cle begraaentio
Bhe Seleclite Scre of
SEpuRatt Grudken
12
Figure A19: Mean and standard deviation of input-unit gradient in ResNet20 trained on CIFARIO. (a) Input-unit unit gradient (y-axis) a function of layer (x-axis). (b) Standard deviation of input-unit gradient for cach unit (aru; y-axis) function layer (x-axis). (c) Standard deviation of input-umit gradient across units in layer (oys y-axis) as function layer (X-axis). Shaded region 95% confidence interval the mean.
25

---

Under review as a conference paper at ICLR 2021
A.9
REPRESENTATIONAL DIMENSIONALITY
Dlts Sefecirtt Peogaarazion Scite HO
34:-8-443442221
Lape
39-30-34062241 B2 02 24
Trandoo of
of
Figure A20: Dimensionality in early layers predicts worsf-case vulnerability in ResNet18 trained on Tiny ImageNet. Identical to Figure[] 9 but dimensionality is computed the number of principal components needed explain 90% of variance in (a) (c), and 99% of variance in (f). Fraction of dimensionality (y-axis; Appendix A.I.4) as function layer (x-axis). (b) Dimensionality difference between clean and average-case perturbation activations (y-axis) as a function of layer (x-axis). (c) Dimensionality of difference between clean and worst-case perturbation activations (y-axis) function of layer (x-axis), (d) (f), identical to (a) (c), but for 99% explained variance threshold.
b)
Cluss Sencindy Negs Regitnntton SCare
Fratton toat
22-30-66-33-2241
m
E &
14
12
Eectton Strwestoosss
0.
0.8
2.0
60
3.0
h)
66
64
1.4
Sescsiity Regaiattztion
Laye
Lager
Figure A21: Dimensionality in early layers predicts worst-case vulnerability in ResNet20 trained on CIFARIO. Identical to Figure 4 but for ResNet20 trained CIFARIO. (a) Fraction of dimensionality needed explain 90% of variance (y-axis; see Appendix A.I.4) as function of layer (x-axis). (b) Dimensionality of
difference between clean and average- case perturbation activations (y-axis) as a function of layer (x-axis), (c) Dimensionality of difference between clean and worst case perturbation activations (y-axis) as function of layer (x-axis). (d) (f) identical to (a) (c). but for 95% explained variance threshold. (g) (i), identical to (a) (c), but for 99% explained variance threshold.
26

---

Under review as a conference paper at ICLR 2021
Selectviit
2.4
2. Li cut Bu-d
nn of
20
Traction of Trae Deseen cralts nine
Selectally Regubritation Seaie In
6.6
Figure A22: Intrinsic dimensionality in early layers predicts worst-case vulnerability. (a) Fraction of function of layer (x-axis) for ResNet18 trained Tiny ImageNet. (b) Dimensionality of difference between clean and average-case perturbation activations (y-axis) a function of layer (x-axis) for R trained on Tiny ImageNet. clean and worst-case perturbation activations (y-axis) a function of layer (x-axis) for ResNet18 trained on Tiny ImageNet. (d) (f), identical to (a) (c), but for ResNet20 trained on CIFARIO.
intrinsic dimensionality (y-axis; see Appendix
(c) Dimensionality difference between
27


Reviews:
[{"review_id": "tdR-DJgJAnz", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work investigates two classes of perturbation robustness, average-case perturbations which are considered to be naturally occurring in image data, and worst-case perturbations that are perturbations generated by an adversary. Neural network susceptibility to these perturbations is evaluated with respect to a class selectivity metric and dimensionality measure. \n\nThe authors find that decreasing class selectivity will increase robustness to average-case perturbations while reducing robustness to worst-case perturbations. Simultaneously, increasing class selectivity improves robustness to worst-case perturbations while reducing average-case perturbation robustness.\n\nIn addition, the authors consider the correspondence between the dimensionality of the early layers and observe how this dimensionality corresponds to class selectivity and robustness. They find the dimensionality is inversely related to class selectivity while also positively correlated with reduced worst-case robustness. \n\nWhile the experiments are thorough in showing a relationship between class selectivity and robustness, many of the observations do not map convincingly to the conclusions. For instance, in Section 4.1, the second paragraph appears to be conjecture which is not justified by the potential observation outcomes. Having higher class selectivity does not necessarily mean fewer potent neurons is the reason for increased average-case robustness. This conclusion is not sufficiently supported by measuring robustness as a function of class selectivity. It also seems odd to hypothesize two different conclusions for opposite outcomes. The same issue of drawing a conclusion regarding the worst-case perturbation analysis in section 4.2. Why create potential conclusions for two opposite outcomes?\n\nThe relationship between the definition of an average-case perturbation and a worst-case perturbation is not well-defined. It appears to be a semantic association of what is expected to occur naturally (average-case) and what is caused by an adversary (worst-case). But without some topology or relationship between these two cases, the observations of class selectivity on each are just independent and unrelated observations. For instance, what is in between an average-case and worst-case perturbation? Where do images from datasets like ObjectNet (https://objectnet.dev/) or Natural Adversarial Examples (https://arxiv.org/abs/1907.07174) fall in the average-case to worst-case dimension?\n\nOverall, this work would benefit from a unified view of how class selectivity relations to image perturbations. As it currently stands, this paper appears to be a collection of observations in need of a clear conclusion.\n\nOther notes:\n-It would be illustrative to have some visualization of the average-case perturbations.\n-$SI_u$ in Eq 3. should also be a function of $l$\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "ZRbly7aZt9N", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##########################################################################\n\nSummary:\n\nThis work empirically studies the relationship between robustness and class selectivity, a measure of neuron variability between classes. Robustness to both adversarial (\"worst-case\") perturbations and corruptions (\"average-case\") are considered. This work builds off the recent work of Leavitt and Morcos (2020) (currently in review at ICLR 2021) who claim empirical evidence that class selectivity may be harmful for generalization.  The experiments in this paper examine the robustness (in both senses) of networks explicitly regularized for class selectivity. The main empirical claims are that (1) class sensitivity is negatively correlated with robustness to corruptions (2) class sensitivity is positively correlated with robustness to adversarial perturbations.\n\n\n\n##########################################################################\n\nReasons for score: \n \n\nOverall I vote for rejection. The authors frame the results in connection to sparsity and dimensionality. But at present the evidence for this connection appears preliminary. For example the differences in the class selectivity curves in Figure 2 appear marginal. On the other hand, Figure 1 does seem convincing for the claim reducing class selectivity improves robustness to corruptions.\n\n\n##########################################################################\n\nPros: \n* Clear results in Figure 1a\n* Important topic\n* Potentially relevant results\n\n##########################################################################\n\nCons: \n* Marginal results in Figure 2\n* Unclear results in Figure 4\n* Measurements of dimensionality limited to PCA\n* Not easy to read. The paper appears hastily written.\n\n##########################################################################\n\nQuestions during rebuttal period: \n\nPlease argue why the differences between class-selective curves in Figure 2 are significant. \n\nI am quite unclear on how to interpret Figure 4. Please clarify.\n\nThe authors use the word \"causal\" several times in the paper, which appears to me dubious. I can find no justification for a claim of causality here, since the results are correlative. Can the authors clarify this?\n\nThe authors only examine one dimensionality estimation method: the number of PCA components required to capture 95% of the data variance. Dimensionality estimation with PCA on data with non-linear (i.e. manifold) structure is problematic. Thus I have doubts on measurements of dimensionality used here. The authors may consider adding the method of Levina and Bickel [0]\n\n[0] Maximum Likelihood Estimation of Intrinsic Dimension - Levina and Bickel (Neurips 2004)\nhttps://papers.nips.cc/paper/2577-maximum-likelihood-estimation-of-intrinsic-dimension\n\n#########################################################################\n\nAdditional Feedback: \n\nIt is the prerogative of the authors to choose the words they believe best express their message. However I lament the author's choice to use \"worst-case perturbation\" and \"average-case perturbation\" to refer to \"adversarial attack\"and \"corruption\". The literature on adversarial attacks is quite large at this point, and \"adversarial\" is the de facto terminology. The authors claim their terminology is more general, however I cannot see the justification given the widespread existing usage. \n\n#########################################################################\n\nPOST-REBUTTAL RESPONSE:\n\nI read the author's rebuttal but have decided to not increase my score. I still have doubts over the claims in this paper. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "Z1wpLGlrzR9", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper studies the relationship between class selectivity and robustness. In particular, it finds that higher class selectivity tends to lead to worse average-case robustness (a.k.a. performance on naturalistically perturbed inputs) but better worst-case robustness (a.k.a. performance on adversarial inputs). \n\nIt further finds that variability of input-unit gradient across samples and units is proportional to a network’s overall class selectivity. Besides, the increase of dimensionality of activation caused by corruption is larger for worst-case perturbation and low-selectivity networks.\n\n################################################\n\nReasons for score:\nOverall, this paper makes a series of interesting observations on the relationship between class selectivity and robustness. I feel that the paper’s contribution is a bit limited to some interesting empirical observations on a particular task (image classification) and type of model (resnet) on two datasets (CIFAR-10 and Tiny ImageNet). Also, no attempt to make much theoretical connections and the practical impact of the paper is not very clear. \n\n################################################\n\nPros:\n\n+mostly well-written and easy to follow\n\n+interesting empirical findings and exploration of sensitivity on worst-case robustness and average-case robustness\n\n+experiments support the claims \n\nCons:\n\n-evaluation looks a bit limited in terms of tasks/models/datasets.\n\nAs even the authors stated in the paper “We hesitate to generalize too broadly about our findings, as they are limited to CNNs trained on image classification tasks. It is possible that the results we report here are specific to our models and/or datasets, and also may not extend to other tasks.” The results are indeed a bit limited in terms of its scope. The evaluation is limited to only ResNets (Res18 and Res20, which are similar to each other in terms of properties) for the image classification task only. I wonder if the findings can hold for deeper models since Fig 3 (b) seems to show that for deeper layers the variability of different SI tends to be increasingly similar.\n\n\n-pure empirical results and not enough theoretical justification\n\nI understand that this is the first paper trying to connect average-case robustness and worst-case robustness with class selectivity but I still would like to see a bit more discussion on the theoretical side.\n\n-the impact of the observations need more discussions.\n\nI understand that this paper focuses on empirically linking average-case robustness and worst-case robustness. However, I think it needs a bit more discussion on how other researchers/practitioners can benefit from these findings. For example, how should one leverage such observations to detect adversarial / naturally perturbed error-prone input , improve robustness, or find the optimal tradeoff between different measures (e.g. natural accuracy, average-case robustness, worst-case robustness. \n\n################################################\n\nTypo:\nIn Figure A6 caption “Figure ??”\n\n################################################\n\nQuestions:\nAt the end of sec 4.2 “indicating that distributed semantic representations are more vulnerable to worst-case perturbation because they are less stable than sparse semantic representations.” What does “distributed semantic representations” mean here? What’s the difference between “distributed semantic representations” and “sparse semantic representations“ in this sentence?\n\n################################################\n\nSuggestions:\nThe paper can be improved by addressing at least one of the three cons I mentioned.\n\n\n################################################\n\nPost-Rebuttal:\nThanks the authors for their detailed responses! \nThe authors responses addressed most of my concerns. In particular,1. the authors showed that their results generalize to Tiny-ImageNet and ResNet50 with additional experiments. 2. the authors added some discussions on the theoretical side. Besides, I also appreciate the addition of experiments on AugMix and PGD which imply the bidirectional causality of  class selectivity and perturbation robustness. Overall, I think this work will potentially be a good addition to the existing understanding of trade-off between worst-case robustness and average-case robustness for the community. Therefore, I decide to increase my score to 6.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "EKpqhBPFcBP", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a new finding that class selectivity is negatively correlated with average-case corruptions (natural visual distortions) while positively correlated with worst-case corruptions (adversarial attacks). The authors then try to explain this phenomenon from two aspects: variability of input-unit gradient and dimensionality in early layers.\n\nPros:\n1. Class selectivity is previously used as a metric to indicate model generalization or memorization. It is good to see the authors generalize its usage to measuring model robustness to natural corruptions and adversarial attacks.\n2. The finding that robustness to average-case corruptions are negatively correlated with class selectivity is no surprise to me, since it has been previously shown that low class selectivity indicates better generalization ability [1].\nThe intriguing finding is that robustness to adversarial images is positively correlated with class selectivity. This is contour intuitive to me at first glance, but the explanations provided in Section 4.3 and 4.4 convince me. \n3. If the conclusions in this paper holds, I think this is one step further from the well-know accuracy-robustness tradeoff [5,6]: not only do we have tradeoff between clean accuracy and adversarial accuracy [5,6], but also a (possibly inherent) tradeoff between average- and worst-case adversarial accuracy.\n\nCons:\nThe general idea to use class selectivity as an indicator for model robustness in this paper is very interesting. \nBut I do think more experiments and discussions are needed to explain these phenomenons, especially the different behaviors of average-and worst-case corruptions. \n1. Adversarial training (e.g., [2]) can greatly improve adversarial robustness. If the conclusions in this paper holds, we would expect adversarially trained models to have much better class selectivity than normally trained models. I'm wondering whether the authors could kindly provide these results?\n2. Similarly, there are some methods (e.g., AugMix [4]) improving model robustness to average-case corruptions. Does AugMix model has significantly lower class selectivity than normally trained models.\n3. In my point of view, one possible explanation for the different behaviors between average-case and worst-case corruptions could be that normal adversarial images (generated by PGD, FGSM as in your paper) are out-of-distribution samples [3], while average-case corruptions are likely to be on-distribution. So I'm a bit curious about the behaviors of the \"on–\nmanifold adversarial examples\" defined in [3]: do they behave more like normal adversarial images (e.g., causing a larger increase in early-layer dimensionality) or more like average-case corruptions (causing a smaller increase in early-layer dimensionality )?\n\n[1] On the importance of single directions for generalization. ICLR, 2018.\n\n[2] Towards Deep Learning Models Resistant to Adversarial Attacks. ICLR, 2018.\n\n[3] Disentangling Adversarial Robustness and Generalization. CVPR, 2019.\n\n[4] AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty. ICLR, 2020.\n\n[5] Robustness May Be at Odds with Accuracy. ICLR, 2019.\n\n[6] Theoretically Principled Trade-off between Robustness and Accuracy. ICML, 2019.\n\nUpdate:\nThanks the authors for their response. All my concerns are addressed and I decide to increase my score from 6 to 7.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers had raised a number of concerns which were mostly addressed during the discussion phase thanks to the additional experiments/explanations that the authors provided. However, some of the reviewers are not yet convinced about the main claims of the paper. While the paper provides a number of interesting/important/novel observations, which are obtained using an extensive set of carefully designed experiments, there are still some ambiguities about the main claims that may need further investigation/ justification. 

One general issue with the current version is lack of clarity. I recommend that the authors revise the writing of the paper and make an effort to better explain/justify the main claims, ideas, and setups in the paper. Perhaps it would help to add a section early in the paper and define/review the basic concepts/definitions.  I also encourage the authors to reason about their main claims, choice of loss function (and why it is the right choice), and their experimental setups.   As an example, the authors should further investigate the impact of their proposed regularization on semantic representations.  The authors interpret the (distribution of) class information as “semantic” content/features. Indeed, when considering data with a variety of average-case perturbations,  one could argue that semantic features like brightness or snow are actually uniform over classes, which might means that class selectivity could not appropriately capture the effect of those features. To evaluate semantic robustness,   it seems necessary to find a way to isolate specific semantic features in the data, i.e. by changing from snow to rain in corresponding images, rather than looking at the performance on the same brightness subset for different levels of their regularization parameter alpha.  Hence there is a need for further investigation on the effectiveness of class selectivity. 

Some of the reviewers have indicated that differences in the class selectivity curves in Figures 1,2 appear marginal (e.g. at most 3% difference). Hence, additional experiments (with other data sets) could be beneficial in this regard. 





Decision: Reject