Linking average- and worst-case perturbation robustness via class selectivity and dimensionality

1 INTRODUCTION . Methods for understanding deep neural networks ( DNNs ) often attempt to find individual neurons or small sets of neurons that are representative of a network ’ s decision ( Erhan et al. , 2009 ; Zeiler and Fergus , 2014 ; Karpathy et al. , 2016 ; Amjad et al. , 2018 ; Lillian et al. , 2018 ; Dhamdhere et al. , 2019 ; Olah et al. , 2020 ) . Selectivity in individual units ( i.e . variability in a neuron ’ s activations across semantically-relevant data features ) has been of particular interest to researchers trying to better understand deep neural networks ( DNNs ) ( Zhou et al. , 2015 ; Olah et al. , 2017 ; Morcos et al. , 2018 ; Zhou et al. , 2018 ; Meyes et al. , 2019 ; Na et al. , 2019 ; Zhou et al. , 2019 ; Rafegas et al. , 2019 ; Bau et al. , 2020 ; Leavitt and Morcos , 2020 ) . However , recent work has shown that selective neurons can be irrelevant , or even detrimental to network performance , emphasizing the importance of examining distributed representations for understanding DNNs ( Morcos et al. , 2018 ; Donnelly and Roegiest , 2019 ; Dalvi et al. , 2019b ; Leavitt and Morcos , 2020 ) . In parallel , work on robustness seeks to build models that are robust to perturbed inputs ( Szegedy et al. , 2013 ; Carlini and Wagner , 2017a ; b ; Vasiljevic et al. , 2016 ; Kurakin et al. , 2017 ; Gilmer et al. , 2018 ; Zheng et al. , 2016 ) . Hendrycks and Dietterich ( 2019 ) distinguish between two types of robustness : corruption robustness , which measures a classifier ’ s performance on low-quality or naturalistically-perturbed inputs—and thus is an `` average-case '' measure—and adversarial robustness , which measures a classifier ’ s performance on small , additive perturbations that are tailored to the classifier—and thus is a `` worst-case '' measure.1 Research on robustness has been predominantly focused on worst-case perturbations , which is affected by weight and activation sparsity ( Madry et al. , 2018 ; Balda et al. , 2020 ; Ye et al. , 2018 ; Guo et al. , 2018 ; Dhillon et al. , 2018 ) and representational dimensionality ( Langeberg et al. , 2019 ; Sanyal et al. , 2020 ; Nayebi and Ganguli , 2017 ) . But less is known about the mechanisms underlying average-case perturbation robustness and its common factors with worst-case robustness . Some techniques for improving worst-case robustness also improve average-case robustness ( Hendrycks and Dietterich , 2019 ; Ford et al. , 2019 ; Yin et al. , 2019 ) , thus it is possible that sparsity and representational dimensionality also contribute to average-case robustness . Selectivity in individual units can be also be thought of a measure of the sparsity with which semantic information is represented.2 And because class selectivity regularization provides a method for controlling selectivity , and class selectivity regularization has been shown to improve test accuracy on unperturbed data ( Leavitt and Morcos , 2020 ) , we sought to investigate whether it could be utilized to improve perturbation robustness and elucidate the factors underlying it . In this work we pursue a series of experiments investigating the causal role of selectivity in robustness to worst-case and average-case perturbations in DNNs . To do so , we used a recently-developed class selectivity regularizer ( Leavitt and Morcos , 2020 ) to directly modify the amount of class selectivity learned by DNNs , and examined how this affected the DNNs ’ robustness to worst-case and average-case perturbations . Our findings are as follows : • Networks regularized to have lower levels of class selectivity are more robust to average-case perturbations , while networks with higher class selectivity are generally less robust to average-case perturbations , as measured in ResNets using the Tiny ImageNetC and CIFAR10C datasets . The corruption robustness imparted by regularizing against class selectivity was consistent across nearly all tested corruptions . • In contrast to its impact on average-case perturbations , decreasing class selectivity reduces robustness to worst-case perturbations in both tested models , as assessed using gradient-based white-box attacks . • The variability of the input-unit gradient across samples and units is proportional to a network ’ s overall class selectivity , indicating that high variability in perturbability within and across units may facilitate worst-case perturbation robustness . • The dimensionality of activation changes caused by corruption markedly increases in early layers for both perturbation types , but is larger for worst-case perturbations and low-selectivity networks . This implies that representational dimensionality may present a trade-off between worst-case and average-case perturbation robustness . Our results demonstrate that changing class selectivity , and hence the sparsity of semantic representations , can confer robustness to average-case or worst-case perturbations , but not both simultaneously . They also highlight the roles of input-unit gradient variability and representational dimensionality in mediating this trade-off . 2 RELATED WORK . 2.1 PERTURBATION ROBUSTNESS . The most commonly studied form of robustness in DNNs is robustness to adversarial attacks , in which an input is perturbed in a manner that maximizes the change in the network ’ s output while 1We use the terms `` worst-case perturbation '' and `` average-case perturbation '' instead of `` adversarial attack '' and `` corruption '' , respectively , because this usage is more general and dispenses with the implied categorical distinction of using seemingly-unrelated terms . Also note that while Hendrycks and Dietterich ( 2019 ) assign specific and distinct meanings to `` perturbation '' and `` corruption '' , we use the term `` perturbation '' more generally to refer to any change to an input . 2Class information is semantic . And because class selectivity measures the degree to which class information is represented in individual neurons , it can be considered a form of sparsity . For example , if a network has high test accuracy on a classification task , it is necessarily representing class ( semantic ) information . But if the mean class selectivity across units is low , then the individual units do not contain much class information , thus the class information must be distributed across units ; the semantic representation in this case is not sparse , it is distributed . attempting to minimize or maintain below some threshold the magnitude of the change to the input ( Serban et al. , 2019 ; Warde-Farley and Goodfellow , 2017 ) . Because white-box adversarial attacks are optimized to best confuse a given network , robustness to adversarial attacks are a `` worst-case '' measure of robustness . Two factors that have been proposed to account for DNN robustness to worst-case perturbations are particularly relevant to the present study : sparsity and dimensionality . Multiple studies have linked activation and weight sparsity with robustness to worst-case perturbations . Adversarial training improves worst-case robustness Goodfellow et al . ( 2015 ) ; Huang et al . ( 2016 ) and results in sparser weight matrices ( Madry et al. , 2018 ; Balda et al. , 2020 ) . Methods for increasing the sparsity of weight matrices ( Ye et al. , 2018 ; Guo et al. , 2018 ) and activations ( Dhillon et al. , 2018 ) likewise improve worst-case robustness , indicating that the weight sparsity caused by worst-case perturbation training is not simply a side-effect . Researchers have also attempted to understand the nature of worst-case robustness from a perspective complementary to that of sparsity : dimensionality . Like sparsity , worst-case perturbation training reduces the rank of weight matrices and representations , and regularizing weight matrices and representations to be low-rank can improve worst-case perturbation robustness ( Langeberg et al. , 2019 ; Sanyal et al. , 2020 ; Nayebi and Ganguli , 2017 ) . Taken together , these studies support the notion that networks with low-dimensional representations are more robust to worst-case perturbations . Comparatively less research has been conducted to understand the factors underlying averagecase robustness . Certain techniques for improving worst-case perturbation robustness also help against average-case perturbations ( Hendrycks and Dietterich , 2019 ; Geirhos et al. , 2018 ; Ford et al. , 2019 ) . Examining the frequency domain has elucidated one mechanism : worst-case perturbations for `` baseline '' models tend to be in the high frequency domain , and improvements in averagecase robustness resulting from worst-case robustness training are at least partially ascribable to models becoming less reliant on high-frequency information ( Yin et al. , 2019 ; Tsuzuku and Sato , 2019 ; Geirhos et al. , 2018 ) . But it remains unknown whether other factors such as sparsity and dimensionality link these two forms of robustness . 2.2 CLASS SELECTIVITY . One technique that has been of particular interest to researchers trying to better understand deep ( and biological ) neural networks is examining the selectivity of individual units ( Zhou et al. , 2015 ; Olah et al. , 2017 ; Morcos et al. , 2018 ; Zhou et al. , 2018 ; Meyes et al. , 2019 ; Na et al. , 2019 ; Zhou et al. , 2019 ; Rafegas et al. , 2019 ; Bau et al. , 2020 ; Leavitt and Morcos , 2020 ; Sherrington , 1906 ; Kandel et al. , 2000 ) . Evidence regarding the importance of selectivity has mostly relied on single unit ablation , and has been equivocal ( Radford et al. , 2017 ; Morcos et al. , 2018 ; Amjad et al. , 2018 ; Zhou et al. , 2018 ; Donnelly and Roegiest , 2019 ; Dalvi et al. , 2019a ) . However Leavitt and Morcos ( 2020 ) examined the role of single unit selectivity in network performance by regularizing for or against class selectivity in the loss function , which sidesteps the limitations of single unit ablation and correlative approaches and allowed them to investigate the causal effect of class selectivity . They found that reducing class selectivity has little negative impact on—and can even improve—test accuracy in CNNs trained on image recognition tasks , but that increasing class selectivity has significant negative effects on test accuracy . However , their study focused on examining the effects of class selectivity on test accuracy in unperturbed ( clean ) inputs . Thus it remains unknown how class selectivity affects robustness to perturbed inputs , and whether class selectivity can serve as or elucidate a link between worst-case and average-case robustness . 3 APPROACH . A detailed description of our approach is provided in Appendix A.1 . Models and training protocols Our experiments were performed on ResNet18 and ResNet50 ( He et al. , 2016 ) trained on Tiny ImageNet ( Fei-Fei et al. , 2015 ) , and ResNet20 ( He et al. , 2016 ) trained on CIFAR10 ( Krizhevsky , 2009 ) . We focus primarily on the results for ResNet18 trained on Tiny ImageNet in the main text for space , though results were qualitatively similar for ResNet50 , and ResNet20 trained on CIFAR10 . Experimental results were obtained with model parameters from the epoch that achieved the highest validation set accuracy over the training epochs , and 20 replicate models ( ResNet18 and ResNet20 ) or 5 replicate models ( Resnet50 ) with different random seeds were run for each hyperparameter set . Class selectivity index Following ( Leavitt and Morcos , 2020 ) . A unit ’ s class selectivity index is calculated as follows : At every ReLU , the activation in response to a single sample was averaged across all elements of the filter map ( which we refer to as a `` unit '' ) . The class-conditional mean activation was then calculated across all samples in the clean test set , and the class selectivity index ( SI ) was calculated as follows : SI = µmax − µ−max µmax + µ−max ( 1 ) where µmax is the largest class-conditional mean activation and µ−max is the mean response to the remaining ( i.e . non-µmax ) classes . The selectivity index ranges from 0 to 1 . A unit with identical average activity for all classes would have a selectivity of 0 , and a unit that only responds to a single class would have a selectivity of 1 . As Morcos et al . ( 2018 ) note , the selectivity index is not a perfect measure of information content in single units . For example , a unit with a litte bit of information about many classes would have a low selectivity index . However , it identifies units that are class-selective similarly to prior studies ( Zhou et al. , 2018 ) . Most importantly , it is differentiable with respect to the model parameters . Class selectivity regularization We used ( Leavitt and Morcos , 2020 ) ’ s class selectivity regularizer to control the levels of class selectivity learned by units in a network during training . Class selectivity regularization is achieved by minimizing the following loss function during training : loss = − C∑ c yc· log ( ŷc ) − αµSI ( 2 ) The left-hand term in the loss function is the standard classification cross-entropy , where c is the class index , C is the number of classes , yc is the true class label , and ŷc is the predicted class probability . The right-hand component of the loss function , −αµSI , is the class selectivity regularizer . The regularizer consists of two terms : the selectivity term , µSI = 1 L L∑ l 1 U U∑ u SIu , l ( 3 ) where l is a convolutional layer , L is number of layers , u is a unit , U is the number of units in a given layer , and SIu is the class selectivity index of unit u . The selectivity term of the regularizer is obtained by computing the selectivity index for each unit in a layer , then computing the mean selectivity index across units within each layer , then computing the mean selectivity index across layers . Computing the mean within layers before computing the mean across layers ( as compared to computing the mean across all units in the network ) mitigates the biases induced by the larger numbers of units in deeper layers . The other term in the regularizer is α , the regularization scale , which determines whether class selectivity is promoted or discouraged . Negative values of α discourage class selectivity in individual units and positive values encourage it . The magnitude of α controls the contribution of the selectivity term to the overall loss . During training , the class selectivity index was computed for each minibatch . The final ( logit ) layer was not subject to selectivity regularization or included in our analyses because by definition , the logit layer must be class selective in a classification task . Measuring average-case robustness To evaluate robustness to average-case perturbations , we tested our networks on CIFAR10C and Tiny ImageNetC , two benchmark datasets consisting of the CIFAR10 or Tiny ImageNet data , respectively , to which a set of naturalistic corruptions have been applied ( Hendrycks and Dietterich , 2019 , examples in Figure A1 ) . We average across all corruption types and severities ( see Appendix A.1.2 for details ) when reporting corrupted test accuracy . Measuring worst-case robustness We tested our models ’ worst-case ( i.e . adversarial ) robustness using two methods . The fast gradient sign method ( FGSM ) ( Goodfellow et al. , 2015 ) is a simple attack that computes the gradient of the loss with respect to the input image , then scales the image ’ s pixels ( within some bound ) in the direction that increases the loss . The second method , projected gradient descent ( PGD ) ( Kurakin et al. , 2016 ; Madry et al. , 2018 ) , is an iterated version of FGSM . We used a step size of 0.0001 and an l∞ norm perturbation budget ( ) of 16/255 . Computing the stability of units and layers To quantify variation in networks ’ perturbability , we first computed the l2 norm of the input-unit gradient for each unit u in a network . We then computed the mean ( µu ) and standard deviation ( σu ) of the norm across samples for each unit . σu/µu yields a ) Shot Noise Brightness Pixelate Glass Blur Motion Blur Impulse Noise Frost Jpeg Compression Contrast Defocus Blur Elastic Transform Snow Fog Gaussian Noise Zoom Blur 0 5 10 15 20 25 30 35 Te st A cc ur ac y Class Selectivity Regularization Scale ( α ) -2.0 Low Selectivity 2.0 High Selectivity 0 Baseline Selectivity -1.0 -0.7 -0.4 -0.2 0.2 0.4 0.7 1.0 b ) -2.0 -1.0 -0.7 -0.4 -0.2 -0.0 0.2 0.4 0.7 1.0 2.0 Regularization Scale ( α ) 13 14 15 16 17 Te st A cc ur ac y Figure 1 : Reducing class selectivity improves average-case robustness . Test accuracy ( y-axis ) as a function of corruption type ( x-axis ) , class selectivity regularization scale ( α ; color ) , and corruption severity ( ordering along y-axis ) . Test accuracy is reduced proportionally to corruption severity , leading to an ordering along the y-axis ; corruption severity 1 ( least severe ) is at the top , corruption severity 5 ( most severe ) at the bottom . ( b ) Mean test accuracy across all corruptions and severities ( y-axis ) as a function of α ( x-axis ) . Results shown are for ResNet18 trained on Tiny ImageNet and tested on Tiny ImageNetC . Error bars = 95 % confidence intervals of the mean . See Figure A6 for CIFAR10C results . the coefficient of variation ( Everitt , 2002 ) for a unit ( CVu ) , a measure of variation in perturbability for individual units . We also quantified the variation across units in a layer by computing the standard deviation of µu across units in a layer l , σ ( µu ) = σl , and dividing this by the corresponding mean across units µ ( µu ) = µl , to yield the CV across units σl/µl = CVl .

Summary: This work empirically studies the relationship between robustness and class selectivity, a measure of neuron variability between classes. Robustness to both adversarial ("worst-case") perturbations and corruptions ("average-case") are considered. This work builds off the recent work of Leavitt and Morcos (2020) (currently in review at ICLR 2021) who claim empirical evidence that class selectivity may be harmful for generalization.  The experiments in this paper examine the robustness (in both senses) of networks explicitly regularized for class selectivity. The main empirical claims are that (1) class sensitivity is negatively correlated with robustness to corruptions (2) class sensitivity is positively correlated with robustness to adversarial perturbations.