Selectivity considered harmful: evaluating the causal impact of class selectivity in DNNs
Authors: Matthew L Leavitt
Ari S. Morcos
Venue: ICLR 2021 Poster (2021)
Keywords: interpretability
explainability
empirical analysis
deep learning
selectivity

Abstract: The properties of individual neurons are often analyzed in order to understand the biological and artificial neural networks in which they're embedded. Class selectivity—typically defined as how different a neuron's responses are across different classes of stimuli or data samples—is commonly used for this purpose. However, it remains an open question whether it is necessary and/or sufficient for deep neural networks (DNNs) to learn class selectivity in individual units. We investigated the causal impact of class selectivity on network function by directly regularizing for or against class selectivity. Using this regularizer to reduce class selectivity across units in convolutional neural networks increased test accuracy by over 2% in ResNet18 and 1% in ResNet50 trained on Tiny ImageNet. For ResNet20 trained on CIFAR10 we could reduce class selectivity by a factor of 2.5 with no impact on test accuracy, and reduce it nearly to zero with only a small (~2%) drop in test accuracy. In contrast, regularizing to increase class selectivity significantly decreased test accuracy across all models and datasets. These results indicate that class selectivity in individual units is neither sufficient nor strictly necessary, and can even impair DNN performance. They also encourage caution when focusing on the properties of single units as representative of the mechanisms by which DNNs function.

Published as a conference paper at ICLR 2021
SELECTIVITY CONSIDERED HARMFUL: EVALUATING THE CAUSAL IMPACT OF CLASS SELECTIVITY IN DNNS
Matthew L. Leavitt, Ari S. Morcos Facebook Al Research
Menlo Park, CA, USA (ito, e fb.. com
ABSTRACT
The properties of individual neurons are often analyzed in order to understand the biological and artificial neural networks in which they're re embedded. Class dleetivity-typiccul
defined as how different neuron's ' responses are across different classes of stimuli or data samples is commonly used for this purpose. However, it remains an open question whether is necessary and/or sufficient for deep neural networks (DNNs) to learn class selectivity in individual units. We investigated the causal impact of class selectivity on network function by directly regularizing for against class selectivity. Using this regularizer to reduce class selectivity across units in convolutional neural networks increased test accuracy by over 2% in R8 and 1% in ResNet50 trained on Tiny ImageNet. For ResNet20 trained on CIFARIO 10 we could reduce class selectivity by factor of 2.5 with no impact on test accuracy, and reduce it nearly zero with only a small (~2%) drop in test accuracy. In contrast, regularizing to increase class selectivity significantly decreased test accuracy across all models and datasets. These results indicate that class selectivity in individual units is neither sufficient nor strictly necessary, and can even impair DNN performance. They also encourage caution when focusing on the properties of single units as representative of the mechanisms by which DNNs function.
INTRODUCTION
Our ability to understand deep learning systems lags considerably behind our ability to obtain practical outcomes with them. breadth of approaches have been developed in attempts to better understand deep learning systems and render them more comprehensible to humans Yosinski et al. [015 Bau (et al. 2017 Olah et al.) 2018 Hooker et al. 2019). Many of these approaches examine the properties of single neurons and treat them representative of the networks in which they're embedded (Erhan
cet al. 2009: Zeiler and Fergus 2014 Karpathy et al. 2016 Amjad et al. 2018 Lillian et al. 2018
Dhamdhere al. 2019: Olah al.) 2020).
The selectivity of individual units (i.c. the variability in a neuron's 's responses across data classes or dimensions) is one property that has been of particular interest researchers trying better
understand deep neural networks (DNNs) (Zhou et al. 2013] Olah et al. 2017 Morcos et al.) 2018b Zhou al. 2018 Meyes al. 2019 Na et al. 2019 Zhou al. 2019 Rafegas et al.] 2019 Bau et al.] 2020 This tocus on individual neurons makes intuitive sense, as the tractable, semantic nature of selectivity is extremely alluring: some measure of selectivity in individual units is often provided
as an explanation of "what" network is "doing". One notable study highlighted a neuron selective for sentiment in an LSTM network trained on a word prediction task (Radford et al. 2017). Another attributed visualizable, semantic features to the activity of individual neurons across GoogLeNet trained on ImageNet (Olah al.) (017]. Both of these examples influenced many subsequent studies, demonstrating the widespread, intuitive appeal of "selectivity" (Amjad et al.) [2018] Meyes et al.
2019
Morcos et al. 2018b Zhou et al. [2015] 2018 Bau al. 2017 Karpathy et al. 2016 Na el al.
2019
Radford et al. 2017 Rafegas al. 2019 Morcos et al. 2018b Olah et al. 201 7 2020).
Work performed as part of the Facebook AI Residency

---

Published as a conference paper at ICLR 2021
Finding intuitive ways of representing the workings of DNNs is essential for making them understand- able and accountable, but we must ensure that our approaches are based on meaningful properties of the system. Recent studies have begun to address this issue by investigating the relationships between selectivity and measures of network function such as generalization and robustness to perturbation (Morcos et al., 2018b] Zhou al. 2018 Dalvi al. 2019). Selectivity has also been used the basis for targeted modulation of neural network function through individual units (Bau et al. 2019a|b) b).
However there is also growing evidence from experiments in both deep learning (Fogg and Vedaldi 2018; Morcos et al. 2018b Gale et al. 2019; Donnelly and Roegiest 2019) and neuroscience (Leavitt al., 2017 Zylberberg 2018] Insanally et al. 2019) that single unit selectivity may not be as important as once thought. Previous studies examining the functional role of selectivity in DNNs have often measured how selectivity mediates the effects of ablating single units, or used indirect, correlational approaches that modulate selectivity indirectly (e.g. batch norm) (Morcos et al.) [018b] Zhou et al., 2018: Lillian et al., 2018 Meyes et al., 2019 Kanda et al.) 2020). But single unit ablation in trained networks has two critical limitations: it cannot address whether the presence of selectivity is beneficial, nor whether networks need to learn selectivity to function properly. It can only address the effect of removing neuron from network whose training process assumed the presence of that neuron. And even then, the observed effect might be misleading. For example, a property that is critical to network function may be replicated across multiple neurons. This redundancy means that ablating any one of these neurons would show little effect, and could thus lead to the erroneous conclusion that the examined property has little impact on network function.
We were motivated by these issues to pursue series of experiments investigating the causal impor- tance of class selectivity in artificial neural networks. To do so, we introduced a term to the loss function that allows us to directly regularize for or against class selectivity, giving us a single knob to control class selectivity in the network. The selectivity regularizer sidesteps the limitations of single unit ablation and other indirect techniques, allowing us to conduct a series of experiments evaluating the causal impact of class selectivity on DNN performance. Our findings are as follows:
Performance can be improved by reducing class selectivity, suggesting that naturally- -earrnely levels of class selectivity can be detrimental. Reducing class selectivity could improve test accuracy by over 2% in R and 1% in ResNet50 trained on Tiny ImageNet.
Even when class selectivity isn't an't detrimental to network function, it remains largely unnec- essary. We reduced the mean class selectivity of units in ResNet20 trained on CIFARIO by factor of 2.5 with no impact on test accuracy, and by a factor of ~20-nearly nearly to a mean
of w-with only a 2% change in test accuracy.
Our regularizer does not simply cause networks to preserve slass-veleytivity by rotating it off of unit-aligned axes (i.e. by distributing selectivity linearly across units), but rather seems to suppress selectivity more generally, even when optimizing for high-selectivity basis sets
This demonstrates the viability of low-selectivity representations distributed across units.
We show that regularizing to increase class selectivity, even by small amounts, has significant negative effects on performance. Trained networks seem to be perched precariously at performance cliff with regard to class selectivity. These results indicate that the levels of class selectivity learned by individual units in the absence of explicit regularization are at the limit of what will impair the network.
Our findings collectively demonstrate that class selectivity in individual units is neither necessary nor sufficient for convolutional neural networks (CNNs) perform image classification tasks, and in some cases can actually be detrimental. This alludes to the possibility of class selectivity regularization as a technique for improving CNN performance. More generally, our results encourage caution when focusing on the properties of single units as representative of the mechanisms by which CNNs function, and emphasize the importance of analyses that examine properties across neurons (i.e. distributed representations). Most importantly, our results are a reminder to verify that the properties we do focus on are actually relevant to CNN function.
RELATED WORK
2.1
SELECTIVITY IN DEEP LEARNING
Examining some form of selectivity in individual units constitutes the bedrock of many approaches to understanding DNNs. Sometimes the goal is simply to visualize selectivity, which has been pursued
2

---

Published as a conference paper at ICLR 2021
using breadth of methods. These include identifying the imput sample(s) (e.g. images) or sample subregions that maximally activate a given neuron (Zhou et al.] 2013: Rafegas et al. 2019). and numerous optimization- saaddd techniques for generating samples that maximize unit activations (Erhan et al. 2009, ZZiler and Fergus, 2014, SSionnyan et al. 2014 Yosinski et al. 2015 Nguyen et al.
2016, Olah al., 2017 2018). While the different methods for quantifying single unit selectivity often conceptually quite similar (measuring how variable are a neuron's 's responses across different
classes of data samples), they have been applied across broad range of contexts (Amjad et al.) 2018
Meyes et al. 2019; Morcos et al. 2018b Zhou et al.) 2015: 2018
Bau et al. [017]] Karpathy et al.
2016] Na et al. 2019; Radford et al., 2001 Rafegas et al. 2019), For example, Bau al. (2017)
quantified single unit selectivity for "concept"" (as annotated by humans) in networks trained for object and scene recognition. Olah et al. 2018 2020) have pursued a research program examining single unit selectivity as a building block for understanding DNNs. And single units in models trained to solve natural language processing tasks have been found to exhibit selectivity for syntactical and semantic features (Karpathy al. 2016 Na et al., (019)), of which the "sentiment-selective neuron" reported by Radford et al. (017)) is particularly recognized example.
The relationship between individual unit selectivity and various measures of DNN performance have been examined in prior studies, but the conclusions have not been concordant. Morcos et al.) (2018b), using single unit ablation and other techniques, found that network's test set generalization is negatively correlated (or uncorrelated) with the class selectivity of its units, a finding replicated by Kanda et al. (2020). In contrast, though (amjadd et al. (2018) confirmed these results for single unit ablation, they also performed cumulative ablation analyses which suggested that selectivity beneficial, suggesting that redundancy across units may make it difficult interpret single unit
ablation studies.
In a follow-up study, /Zhuu et al. (2018) found that ablating class- selective units impairs classification accuracy for specific classes (though interestingly, not always the same class the unit was selective for), but a compensatory increase in accuracy for other classes can often leave overall accuracy unaffected. (201g found that orientation selectivity in individual units is correlated with generalization performance in convolutional neural networks (CNNs), and that ablating highly orientation sellectvv units impairs classification accuracy more than ablating units with low iientation-selectivii while orientation selectivity and class selectivity can both be considered types of feature selectivity, orientation selectivity is far less abstract and focuses specific properties of the image (e.g., oriented edges) rather than semantically meaningful concepts and classes. Nevertheless, this study still demonstrates the importance of some types of selectivity.
But
Results are also variable for models trained on NLP tasks. Dalvi et al. (2019) found that ablating units selective for linguistic features causes greater performance deficits than ablating less-selective units, while Donnelly and Roegiest (2019) found that ablating the "sentiment neuron" of Radford al. 2017) has equivocal effects on performance. These findings seem challenging to reconcile.
All of these studies examining class selectivity in single units are hamstrung by their reliance on single unit ablation, which could account for their conflicting results. As discussed earlier, single unit ablation can only address whether class selectivity affects performance in trained networks, and not whether individual units to need to learn class selectivity for optimal network function. And even then, the conclusions obtained from single neuron ablation analyses can be misleading due to redundancy across units (Amjad et al. 2018 Meyes et al. 2019
2.2 SELECTIVITY IN NEUROSCIENCE
Measuring the responses of single neurons a relevant set of stimuli has been the canonical first-order
approach for understanding the nervous system (Sherrington 1906; Adrian 1926, Granit [955]
Hubel and Wiesel 1959; [Ballow] 1972, Kandel et al. 2000); its application has yielded multiple
Nobel Prizes (Hubel and Wiesel 1959 1962 1962, Hubel 1982 Wiesel 1982 OeKeee and Dostrovsky
1971: Fyhn et al./ 2004). But recent experimental findings have raised doubts about the necessity of
selectivity for high-fidelity representations in neuronal populations (Leavitt et al.] 2017 7 Insanally
ct al.
2019 Zylberberg, 2018), and neuroscience research seems to be moving beyond characterizing
neural systems at the level of single neurons, towards population-level phenomena (Shenoy et al.
2013
Raposo et al. 2014 Fusi et al. 2016||Mrrco, and Harvey 2016: Pruszynski and Zylberberg
2019
Heeger and Mackey
2019 Saxena and Cunningham 2019)
3

---

Published as a conference paper at ICLR 2021
Single unit selectivity-based approaches ubiquitous in attempts to understand artificial and biological neural systems, but growing evidence has led to questions about the importance of focusing on selectivity and its role in DNN function. These factors, combined with the limitations of prior approaches, lead to the question: is class selectivity necessary and/or sufficient for DNN function?
3 APPROACH
Networks naturally seem to learn solutions that result in class-selective individual units (Zhou et al.
|013| OOlah et al. 2017]] Morcos et al. 2018b] Zhou et al. 20118) Meyes et al. 2019 Na et al. 2019
Zhou et al.] 2019 Rafegas et al., 2019; Amjad et al. 22018] Meyes et al. 2019 Bau et al. 2017
Karpathy et al. 2016 Radford et al. 201 Olah et al. 2018 2020). We examined whether learning
class-selective representations in individual units is actually necessary for networks to function
properly. Motivated by the limitations of single unit ablation techniques and the indirectness of using batch norm or dropout modulate class selectivity (e.g. [Mocoss al.] (2018b); /Zhou et al. (2018); Lillian et al. (2018); Meyes et al. (2019), ) we developed an alternative approach for examining the necessity of class selectivity for network performance. By adding a term the loss function that serves as a regularizer to suppress (or increase) class selectivity, we demonstrate that it is possible to directly modulate the amount of class selectivity in all units in aggregate. We then used this approach as the basis for series of experiments in which we modulated levels of class selectivity across individual units and measured the resulting effects on the network. Critically, the selectivity regularizer sidesteps the limitations of single unit ablation-based approaches, allowing us to answer debeewise-inacceseee questions such as whether single units actually need to learn class selectivity, and whether increased levels of class selectivity are beneficial.
Unless otherwise noted: all experimental results were derived from the test set with the parameters from the epoch that achieved the highest validation set accuracy over the training epochs; 20 replicates with different random seeds were run for each hyperparameter set; error bars and shaded regions denote bootstrapped 95% confidence intervals; selectivity regularization (logits), nor was the output included in any of our analyses because by definition the output must be class selective in classification task. Selectivity regularization was only applied to intermediate (hidden) layers with non-linearities
not applied to the output
3.1 MODELS AND DATASETS
Our experiments were performed on ResNet18 and ResNet50 (He et al., 2016) trained on Tiny ImageNet (Fei-Fei al. 2015), and ResNet20 (He al. 2016)) and a VGGI6 like network (Simonyan and Zisserman 2015), both trained on CIFARIO (Krihhvskyy) 2009). Additional details about hyperparameters. data, training, and software are in Appendix A.I We focus on ResNet18 trained on Tiny ImageNet in the main text, but results were qualitatively similar across models and datasets except where noted.
3.2 DEFINING CLASS SELECTIVITY
There are a breadth of approaches for quantifying class selectivity in individual units (Moody et al. 1998 Zhou et al. 2015 Li et al. 2015 Zhou et al. 2018; Gale et al. (019). We chose the neuroscience-inspree. approach of Morcos et al. 2h) because it similar to many widely-used metrics, easy to compute, and most importantly, differentiable (the utility of this is addressed in the next section). We also confirmed the efficacy of our regularizer on a different, non-differentiable selectivity metric (see Appendix 13).). For single convolutional feature map (which we refer to as a "unit"), we computed the mean activation across elements of the filter map in response to a single sample, after the non-lineari... Then the class-conditional mean activation (i.e. the mean activation for each class) was calculated across all samples in the test set, and the class selectivity index (ST) was calculated follows:
SI = Pmar 14 max
(1)
Pmaz + p-max +
where Pmaz is the largest class- conditioll mean activation, M-moz is the mean response to the remaining (i.e. non-Hmax) classes, and e is a small value to prevent division by zero (we used 10 10-7) in the case of a dead unit. The selectivity index can range from 0 to 1. A unit with identical average
4

---

Published as a conference paper at ICLR 2021
b)
50
53.0
Regularization
80
Scale (r)
-0.7
53.5
10
50
-0.
1.5
0.1
-1.0 0.0 (baselinet
50.0
0.10
0.15 0.20 0.25
0.30
0.35
Regularization Scale (at
Iagularizationn Scale (o)
Mean Class Selectivity
Figure 1: Effects of reducing class selectivity on test accuracy in ResNet18 trained on Tiny Imagenet. (a) Test accuracy (y-axis) function of regularization scale (a, x-axis and intensity of blue). (b) Identical to (a), but for subset of a values. The center of each violin plot contains boxplot, in which the darker central lines denote the central two quartiles. (c) Test accuracy (y-axis) as function of mean class selectivity (x axis) for different values of a. Error bars denote 95% confidence intervals. "p < 0.01, **p < x 10 10 difference from = 0, t-test, conferroni-correcte See Appendix A.4 and A.12 for ResNet20 and VGG results, respectively.
activity for all classes would have selectivity of O, and unit that only responded to single class would have a selectivity of 1.
As Morcos al. (2118) note, this selectivity index is not perfect measure of information content in single units. For example, a unit with some information about many classes would have a low selectivity index. But it achieves the goal of identifying units that are class-selective in similarly intuitive way as prior studies (Zhou et al. 2018), while also being differentiable with respect to the model parameters.
3.3 A SINGLE KNOB TO CONTROL CLASS SELECTIVITY
Because the class selectivity index is differentiable, we can insert it into the loss function, allowing us to directly regularize for or against class selectivity. Our loss function, which we seek to minimize, thus takes the following form:
C loss Σ Be log(ge) - apsi
The left-hand term in the loss function is the traditional cross-entropy between the softmax of the output units and the true class labels, where e is the class index, C' is the number of classes, ye is the true class label, and ye is the predicted class probability. We refer to the right-hand component of the loss function, -aUsI, as the class selectivity regularizer (or regularizer, for brevity). The regularizer consists of two terms: the selectivity term.
U
MSi 1E
E ST
(3)
where is convolutional layer. L is number of layers, is a unit (i.c. feature map), U is the number of units in a given layer, and ST, is the class selectivity index of unit u. The selectivity term of the regularizer is obtained by computing the selectivity index for each unit in layer, then computing the mean selectivity index across units within each layer. then computing the mean selectivity index across layers. Computing the mean within layers before computing the mean across layers (as compared to computing the mean across all units in the network) mitigates the biases induced by the larger numbers of units in deeper layers. The remaining term in the regularizer is a, the regularizer scale. The sign of a determines whether class selectivity is promoted or discouraged. Negative values of a discourage class selectivity in individual units, while positive values promote it. The magnitude of a controls the contribution of the selectivity term to the overall loss. a thus serves single knob with which we can modulate class selectivity across all units in the network in aggregate. During training, the class selectivity index was computed for each minibatch. For the results presented here, the class selectivity index was computed across the entire test set. We also tried restricting regularization to the first or final three layers (Appendix 151 and warming up the class selectivity regularization over the initial training epochs (Appendix A. 16), all of which yielded qualitatively similar results.
5

---

Published as a conference paper at ICLR 2021
0:35
0.200
0:30
Reguderteernn
0:175
0.25
00030
Disrance
0.20
0.000
0.15
0.00
0.10
1:41
0.090
0.05
0.023
Opdi bed Inquation Lasper bound
0.00
10 16
Siall
Figure 2: Checking for off-axis selectivity. (a) Mcan CCA distance (p. y-axis) function of layer (x-axis) between pairs of replicate ResNet18 networks (sce Section 4.2 or Appendix A.3.2] trained with a = 2 (i.c. 2. light purple), and between pairs of networks trained with a = 2 and a = 0 (i.e. p(a- (k)) dark purple). (b) For each layer, we compute the ratio of p(a-2, (x)) p(a-11 a-2), which we refer to as the CCA distance ratio. We then plot the mean CCA distance ratio across layers (y-axis) as a function of (x-axis, intensity of blue). Example from panel a (a -2) circled in purple. p << 1.3 10-6, paired t-test, for all except -0.1. (c) Mean class selectivity (y-axis) as a function of regularization scale (a; x-axis) for ResNet18 trained on Tiny ImageNet. Diamond-shaped data points denote the upper bound on class selectivity for a linear projection of activations (see Section 442 or Appendix A.7). while circular points denote the amount of axis-aligned class selectivity for the corresponding values of o. Error bars shaded region 95% confidence intervals. ResNet20 results are in Appendix A.6 (CCA) and A.7 (selectivity upper bound).
4 RESULTS
4.1 TEST ACCURACY IS IMPROVED OR UNAFFECTED BY REDUCING CLASS SELECTIVITY
Prior research has yielded equivocal results regarding the importance of class selectivity in individual units. We sidestepped the limitations of previous approaches by regularizing against selectivity directly in the loss function through the addition of the selectivity term (see Approach 3.3), giving us knob with which to causally manipulate class selectivity. We first verified that the regularizer works as intended (FggureAAT). Indeed, class selectivity across units in a network decreases as a becomes more negative. We also confirmed that our class selectivity regularizer has similar effects when measured using a different class selectivity metric and mutual information (see Appendix [A.13). and when regularizing control d'-a measure of class discriminability- individual units (Appendix A. The consistency of our observations across metrics of selectivity indicates that our results are not unique to the metric used in our regularizer. The regularizer thus allows us to to examine the causal impact of class selectivity on test accuracy.
Regularizing against class selectivity could yield three possible outcomes: If the previously-reporte
anti-correlation between selectivity and generalization is causal, then test accuracy should increase. But if class selectivity is necessary for high-fidelit class representations, then we should observe a decrease in test accuracy. Finally, if class selectivity is an emergent phenomenon and/or irrelevant to network performance, test accuracy should remain unchanged.
Surprisingly. we observed that reducing selectivity significantly improves test accuracy in ResNet18
trained on Tiny ImageNet for all examined values of a E [-0.1, 2.5] (Figure 1 p < 0.01, cnnfeerroiicorrecte t-test). Test accuracy increases with the magnitude of a, reaching maximum at a = -10 (test accuracy at Q-1.0 = 53.60 = 0.13, ag (i.c. no regularization) = 51.57 + 0.18), at which point there is a 1.6x reduction in class selectivity (mean class selectivity Q-1.0 0.22 + 0. 0.0009, ag = 0.35 + 0.0007). Test accuracy then begins to decline; at 0x-3.0 test accuracy is statistically indistinct from ag, despite a 3x decrease in class class selectivity (mean class selectivity at a-3.0 = 0.12 0.0007, ag 0.35 + 0.0007). Further reducing class selectivity beyond a = -3.5 (mean class selectivity = 0.10 + 0.0007) has increasingly detrimental effects on test accuracy. These results show that the amount of class selectivity naturally learned by network (i.e. the amount learned in the absence of explicit regularization) can actually constrain the network's 's performance.
ResNet20 trained on CIFARIO also learned superfluous class selectivity. Although reducing class se- lectivity does not improve performance, causes minimal detriment, except at extreme regularization scales (a < -30; Figure Increasing the magnitude of a decreases mean class selectivity across the network, with little impact on test accuracy until mean class selectivity reaches 0.003 = 0.0002 at a= (Figure Ald) Reducing class selectivity only begins to have statistically significant effect on performance at 1-1.0 (Figure A2a), at which point mean class selectivity across the network has decreased from 0.22 = 0.002 at og (1.c. no regularization) to 0.07 + 0.0013 at a-100 factor of -iceerooi-coorectee t-test). This implics that ResNet20 learns more than three times the amount of class selectivity required for maximum test accuracy.
more than 3 (Figure A2c; p = 0.03,

---

Published as a conference paper at ICLR 2021
20
50
40
Actursty
Regularsation Scale
Tex
0.0
1.0
10
41
10
0.0 0.1 0.2 0.5 DA 0.7 1.0 2.0
Regulbrizatin Scale
Regularizatin Scale (o)
Class Selectivity
Figure 3: Effects of increasing class selectivity on test accuracy in ResNet18 trained on Tiny ImageNet. (a) Test accuracy (y-axis) as a function of regularization scale (a; x-axis, intensity of red). (b) Identical to (a), but for subset of values. Each violin plot contains a boxplot in which the darker central lines denote the central two quartiles. (e) Test accuracy (y-axis) as function of mean class selectivity (x-axis) across values. Error bars denote 95% confidence intervals. *p- < 10 ** < 8 10 difference from a 0, t-test, Conferroni-correctt See Appendix A.9 and A.12 for ResNet20 and VGG results, respectively.
We observed qualitatively similar results for VGGI6 (see Appendix A.12). Although the difference significant at = -0.1 (p= = 0.004, Bonferroni-corrected t-test), possible reduce mean class selectivity by factor of 5 with only 0.5% decrease in test accuracy, and by a factor of 10 with only a ~1% drop in test aceuracy. These differences may be due to VGG16's naturally higher levels of class selectivity (see Figure[A16]an comparisons between VGG16 and ResNet20). We also observed qualitatively similar results when using our regularization approach to decrease d', a measure of class discriminability. in ResNet20 trained on CIFAR1O 10 and ResNet18 8 trained on Tiny ImageNet (Appendix 14). Furthermore, we also find that regularizing to reduce class selectivity improves test accuracy in ResNet50 trained on Tiny ImageNet (Appendix A.. 18.18. Together, these results demonstrate that class selectivity in individual units is largely unnecessary for optimal performance in CNNs trained on image classification tasks.
4.2 DOES SELECTIVITY SHIFT TO A DIFFERENT BASIS SET?
We were able to reduce mean class selectivity in all examined networks by a factor of at least three with minimal negative impact on test accuracy (~1% at worst, for VGG16). However, one trivial solution for reducing class selectivity is for the network to "hide" it from the regularizer by rotating it off of unit-aligned axes or performing some other linear transformation. In this scenario the selectivity in individual units would be reduced, but remain accessible through linear combinations of activity across units. In order test this possibility, we used CCA (see Appendix App.. which is invariant to rotation and other invertible affine transformations, to compare the representations in regularized (i.e. low-selectivity) networks to the representations unregularized networks.
We first established a meaningful baseline for comparison by computing the CCA distances between each pair of 20 replicate networks for given value of a (we refer to this set of distances as p(a,, a.,)). )). If regularizing against class selectivity causes the network to move selectivity off-axis, the CCA distances between regularized and unregularized networks -which we term p(a,, ag)- should be similar to p(ar, a.)) ). Alternatively, if class selectivity is suppressed via some non-affine transforma- tion of the representation, p(a,, 00) should exceed p(a,,a,)
Our analyses confirm the latter hypothesis: we find that p(ar, 0o) significantly exceeds p(ar, Ory) for all values of a except a = -0.1 in ResNet18 trained on Tiny ImageNet (Figure < 1.3 x 10-5, paired t-test). The effect even more striking in ResNet20 trained on CIFAR 10; all tested values of a are significant (Figure A3 p < 5 x 10t, paired I-test). Furthermore, the size of the effect is proportional to a in both models; larger a values yield representations that are more dissimilar to unregularized representations. These results support the conclusion that our regularizer doesn's 't just cause class selectivity to be rotated off of unit-all aligned axes, but also suppresses it.
As an additional control to ensure that our regularizer did not simply shift class selectivity to off-axis directions in activation space, we calculated an upper bound on the amount of class selectivity that could be recovered by finding the linear projection of unit activations that maximizes class selectivity (see Appendix A..7 for methodological details). For both ResNet18 trained on Tiny ImageNet (Figure 2cb and ResNet20 trained on CIFARIO 10 (Figure A4b). the amount of class selectivity in the optimized projection decreases as a function of increasing (a), indicating that regularizing against class selectivity does not simply rotate the selectivity off-axis. Interestingly, the upper bound on class selectivity is very similar across regularization scales in the final two convolutional layers in both models (Figure A4a A4c), indicating that immediate proximity to the logits (output) may mitigate
7

---

Published as a conference paper at ICLR 2021
b)
Acturne
Test
-200
3.5
signian
20
100.0
0.0
0.1
5.3
0.3
0.4 0.7 1.0 2.0
0.0 Nean Class Selectivity
0.6
Regularrvation Staile [a1)
Figure 4: Increasing class selectivity has deleterious effects test accuracy compared to reducing class selectivity, (a) Test accuracy (y-axis) as a function regularization scale magnitude (al) for negative (blue) vs positive (red) values of a. Solid line in distributions denotes mean, dashed line denotes central two quartiles. **p < 6 x 10 difference hetween o < O and a > 0, Wilcoxon rank-sum test, conferroni-corrected. (b) Test accuracy (y-axis) a function of mean class selectivity (x-axis). All results shown are for ResNet18.
the effect of class selectivity regularization. While we also found that the amount of class selectivity in the optimized projection is consistently higher than the observed axis-aligned class selectivity, we consider this to be an expected result, as the optimized projection represents an upper bound on the amount class selectivity that could be recovered from the models' representations. However, the decreasing upper bound as a function of increasing |a| indicates that our class selectivity regularizer decreases selectivity across all basis sets, and not just along unit-aligned axes.
4.3 INCREASED CLASS SELECTIVITY CONSIDERED HARMFUL
We have demonstrated that class selectivity can be significantly reduced with minimal impact on test accuracy. However, we only examined the effects of reducing selectivity. What are the effects of increasing selectivity? We examined this question by regularizing for class selectivity, instead of against it. This is achieved quite easily, as it requires only a change in the sign of a. We first confirmed that changing the sign of the scale term in the loss function causes the intended effect of increasing class selectivity in individual units (see Appendix(A.3. (A.)).
Despite class selectivity not being strictly necessary for high performance, its ubiquity across biological and artificial neural networks leads us to suspect it may still be sufficient. We thus expect that increasing it would either improve test accuracy yield no effect. For the same reason, would consider it unexpected if increasing selectivity impairs test accuracy.
we observe the latter outcome: increasing class selectivity negatively impacts network performance in ResNet18 8 trained on Tiny ImageNet (Figure Ba). Scaling the regularization has an immediate effect: a significant decline in test accuracy is present even at the smallest tested value of a (p S 6 x 10-5 for all a, Bonferroni-corree == 5.0. The effect proceeds even more dramatically in ResNet20 trained on CIFARIO (Figure AAaa Note that we observed a correlation between the strength of regularization and the presence of dead units in ResNet20 (but not ResNet18), however further analyses ruled this out as an explanation for the decline in test accuracy (see Appendix 10).
Surprisingly,
t-test) and falls catastrophically to ~25% by
One solution to generate a very high selectivity index if a unit is silent for the vast majority of inputs and has low activations for remaining set of inputs. If this were the case, we would expect that regularizing to increase selectivity would cause units to be silent for the majority of inputs. However, we found that the majority of units were active for 880 of inputs even significant performance deficits have emerged in both ResNet18 and ResNet20 (Appendix I1) These findings rule out sparsity regularizing to increase class selectivity could discourage individual neurons from changing their preferred class during training, even if changing their preferred class would improve performance. If regularizing to increase class selectivity did indeed lock units in to their initial preferred class, this could impose a constraint on performance. We tested for this possibility in two ways (Appendix A.16): by examining the statistics of units' changes in preferred class during training (Figures A28
0.7, after
potential explanation for our results. It is also possible that
A299, and slowly warming up class selectivity regularization over the initial training epochs (Figures A30 A31. Approximately 100% of units across all examined models and regularization scales change their preferred class at least once during training (Figure A28). and the relationship between class selectivity regularization and the number of changes in unit's 's preferred class over training is inconsistent (Figure A29). Furthermore, warming up the regularization has qualitatively similar effects as using constant a (Figure[A31). None of these analyses indicate that inability to change preferred locations can fully explain the class indectivity-inducae test accuracy impairment (see Appendix A. 1. for additional details).
8

---

Published as a conference paper at ICLR 2021
The effects of regularizing to increase class selectivity qualitatively similar for VGG16 (see Appendix (2.12) and ResNet50 (Appendix ( we observed across all models that increasing class selectivity beyond the levels that are learned naturally (i.e. without regularization, 0) impairs network performance.
Recapitulation We directly compare the effects of increasing vs. decreasing class selectivity in Figure 4 and Appendix AA.7] The effects diverge immediately at |ar| = 0.1, and suppressing class
selectivity yields 6% increase in test accuracy relative to increasing class selectivity by |a| 2.0.
5 DISCUSSION
We examined the causal role of class selectivity in CNN performance by adding term the loss function that allows us to directly manipulate class selectivity across all neurons in the network. We found that class selectivity is not strictly necessary for networks to function, and that reducing can even improve test accuracy. In ResNet18 trained on Tiny Imagenet, reducing class selectivity by 1.6x improved test accuracy by over 2%. In ResNet20 trained on CIFAR1O, we could reduce the mean class selectivity of units in a network by factor of ~2.5 with impact on test accuracy, and by a factor of ~20 ~20-nearly to a mean O-with only a 2% change in test accuracy. We confirmed that regularizer seems to suppress class selectivity, and not simply cause the network to rotate it off of unit-aligned axes. We also found that regularizing a network to increase class selectivity in individual units has negative effects performance. These results resolve questions about class selectivity that remained inaccessible to previous approaches: class selectivity in individual units is neither necessary constrain CNN performance.
sufficient for and can sometimes
One caveat to our results is that they are limited to CNNs trained to perform image classification. It's possible that our findings are due idiosyncracies of benchmark datasets, and wouldn't 't generalize to more naturalistic datasets and tasks. Given that class selectivity is ubiquitous across DNNs trained on different tasks and datasets, future work should examine how broadly our results generalize, and the viability of class selectivity regularization as a general-purpor tool to improve DNN performance.
The presence of non-sclective units in a network trained perform image classification could appear puzzling. It may be difficult to envision how non-selective units could shape representations in manner that useful for a classification task. One possibility that the ctass-conditional joint distribution of activations across units facilitates readout. Put another way. the correlations between units' activations can help separate the distributions of activations for different classes. Indeed, there is evidence that correlated variability between neurons can facilitate information readout in the brain (Zylberberg et al. 2016 Leavitt et al. 2017 Zylberberg, 2018: Nogueira et al. 2020).
We know that class selectivity in individual units naturally emerges over the course of learning. The single unit ablation studies show that class selectivity can have an effect on the performance of trained networks. And while it is possible that networks trained with selectivity regularization learn different solutions from networks trained without it, our results show that class selectivity is not strictly necessary for networks to learn representations that result in class-selective units. This finding naturally leads to a compelling question: if class selectivity is unnecessary, why does it emerge?
Our results also make broader point about the potential pitfalls of focusing the properties of single units when trying to understand DNNs, emphasizing instead the importance of analyses that focus on distributed representations. While consider it essential find tractable, intuitive approaches for understanding complex systems, it's critical to empirically verify that these approaches actually reflect functionally relevant properties of the system being examined.
ACKNOWLEDGEMENTS
We would like to thank Tatiana Likhomanenko, Tiffany Cai, Eric Mintun, Janice Lan, Mike Rabbat, Sergey Edunov, Yuandong Tian, and Lyndon Duong for their productive scrutiny and insightful feedback.

---

Published as a conference paper at ICLR 2021
REFERENCES
Alessandro Achille, Matteo Rovere, and Stefano Soatto. Critical Learning Periods in Deep Networks. September 2018. UtL.ittess//peeeee....aaadeeeere BckstssscKKeeK
E. D. Adrian. The impulses produced by sensory nerve endings. The Journal of Physiology, 61 (1):49-72, March 1926. ISSN 0022-3751. URL http:://www.r.......mmnm....///.
Rana Ali Amjad, Kairen Liu, and Bernhard C. Geiger. Ind Neuron Importance Using Information Theory. April 2018. URL /RRLittpp///art/oor//////1110000
Barlow. Single Units and Sensation: A Neuron Doctrine for Perceptual Psychology? Perception,
1(4):371-394, December 1972. ISSN 0301-0066. doi: 10.1068/p010371. URL https://
doi 2ol.org/10.1068/pbll
Anthony Bau, Yonatan Belinkov, Hassan Sajjad, Nadir Durrani, Fahim Dalvi, and James Glass. Identifying and Controlling Important Neurons Neural Machine Translation, In International Conference on Learning Representations, 2019a. URL https://openrevieww
Forum?id-Hi. PsR5KX
David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network Dissec- tion: Quantifying Interpretability of Deep Visual Representations. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3319-3327, Honolulu, HI. July 2017. IEEE. ISBN 978-1 178-1-5386-040 1-5386-0455-
doi: 10. 11..///V...00...555 URL http:// beeege/eoorr.ieeeerrrr///eeeeees
David Bau, Jun- Yan Zhu, Hendrik Strobelt, Bolei Zhou, Joshua B. Tenenbaum, William T. Freeman, and Antonio Torralba. GAN Dissection: Visualizing and Understanding Generative Adversarial Networks. In Proceedings of the International Conference on Learning Representations (ICLR), 2019b.
David Bau, Jun- Yan Zhu, Hendrik Strobelt, Agata Lapedriza, Bolei Zhou, and Antonio Tor- ralba. Understanding the role of individual units in deep neural network. Proceedings of the National Academy of Sciences, September 2020. ISSN 0027-8424, 1091-6490, doi: 10. 1073/aapss 1L11073/paas.1003751177 URL BRLLttps::/ http://wwwwww....t./ .rgg wwnteet/onnttttooeeeeeeoo wereeearryyyy/e 31/1907375117
/ttpteeeeeeeeerr//////////// 20200008
Publisher: National Academy of Sciences Section: Physical Sciences.
Thomas M Cover. Elements of information theory. John Wiley & Sons, 1999.
Fahim Dalvi, Nadir Durrani, Hassan Sajjad, Yonatan Belinkov, Anthony Bau, and James Glass. What Is One Grain of Sand the Desert? Analyzing Individual Neurons Deep NLP Models. Proceedings of the AAAI Conference on Artificial Intelligence, (3(01):6309-631) July 2019. doi: 10.1660//aii.v33330 33300030000000000000003 URL
ISSN 2364-3468,
index.php/ AAAI http//aaiiieeeeeeeee5/////////
Kedar Dhamdhere, Mukund Sundararajan, and Qiqi Yan. How Important is a Neuron. In International Conference on Learning Representations, 2019. URL httpa://ogenreview.ees
Forum? 1d= 1 Koo0cKm.
Jonathan Donnelly and Adam Roegiest. On Interpretability and Feature Representations: An Analysis of the Sentiment Neuron. In Leif Azzopardi, Benno Stein, Norbert Fuhr, Philipp Mayr, Claudia Hauff, and Djoerd Hiemstra, editors, Advances in Information Retrieval, Lecture Notes in Computer Science, pages 795-802, Cham, 2019. Springer International Publishing. ISBN 978-3-030-15511-88 doi: 10. 00.1077/778-3000-5557223
Dumitru Erhan, Yoshua Bengio, Aaron C. Courville, and Pascal Vincent. Visualizing Higher-Layer Features of Deep Network. 2009.
Li Fei-Fei, Andrej Karpathy, and Justin Johnson. Tiny imagenet visual recognition challenge, 2015. URL htw.:hhcp..//Eimmmimneeeeee.ee
10

---

Published as a conference paper at ICLR 2021
Ruth Fong and Andrea Vedaldi. Vec: Quantifying and Explaining How Concepts are Encoded by Filters in Deep Neural Networks. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8738, Salt Lake City, UT. June 2018. IEEE. ISBN 978- 1-53666--0-
9. doi: 10.1 2090//CV.2008.000911 URL htt.ltttps:::/eeeplloooo http:///oo.eeee....ee/////ooeeee
85790087
Jonathan Frankle, David J. Schwab, and Ari S. Morcos. The Early Phase of Neural Network Training.
arXiv:2002. 10365 [cs, stat], February 2020. URL /ttphh/ttr///aaiioorr//a/////220 arXiv: 2002.
Stefano Fusi, Earl K. Miller, and Mattia Rigotti. Why neurons mix: high dimensionality for higher cognition. Current opinion in neurobiology, 37:66-74, 2016. doi: 10.10166j.comb.2216.011.
Marianne Fyhn, Sturla Molden, Menno P. Witter, Edvard I. Moser, and May-Britt Moser. Spatial Representation in the Entorhinal Cortex. Science, (05(5688)11258-2225 August 2004. ISSN 0036-8075, 1095-9203. doi: 10.1126/science.
10126//scccnce.1999911 1099901. URL https:// Publisher: American Associ- ation for the Advancement of Science Section: Research Article.
science.sciennemeg.orr. cwtt:/nonteeeeeeeeeeeoceeceecctt /305 /ttmtetttt05//////2222222 5155/5688/1255
Ella Gale, Ryan Blything, Nicholas Martin, Jeffrey S. Bowers, and Anh Nguyen. Selectivity metrics provide misleading estimates of the selectivity of single units in neural networks. In Ashok K. Goel, Colleen M. Seifert, and Christian Freksa, editors, Proceedings of the 41th Annual Meeting of the Cognitive Science Society, CogSci 2019: Creativity + Cognition Computation, Montreal, Canada, July 24-27, 2019, pages 18888-88444 cognifivesciencsociiett..ggg 2019. URL
Juan A. Gallego, Matthew G. Perich, Stephanie N. Naufel, Christian Ethier, Sara A. Solla, and Lee E. Miller. Cortical population activity within a preserved neural manifold underlies multiple motor behaviors. Nature Communications, 9(1::2233, December 2018. ISSN 2041-1723. doi:
10. 10.1088/41447--11-66333 06560-z
Ragnar Granit. Receptors and sensory perception. Receptors and sensory perception. Yale University Press, New Haven, CT, US, 1955.
Guy Gur-Ari, Daniel A. Roberts, and Ethan Dyer. Gradient Descent Happens in Tiny Subspace. [cs, stat], December 2018. URL UKL/ttpp///axxvocrr////////C
arXiv: 1812.04754.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
David J. Heeger and Wayne E. Mackey. Oscillatory recurrent gated neural integrator circuits (organ- ics), unifying theoretical framework for neural dynamics. Proceedings of the National Academy of Sciences, 116445)227833-22794 2019. ISSN 0027-8424. doi: 10. 10.10731/ URL
1911633116..
htttpss//ww.paassorr//ooooo wtw.aaaa...rrooooee.ee.oaeeeee/e /0/66/5522283
Sara Hooker, Dumitru Erhan, Pieter-Jan Kindermans, and Been Kim. A Bench- mark for Interpretability Methods in Deep Neural Networks. In H. Wallach, H. Larochelle, A. Beygelzimer, F. Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages ciates, Inc., 2019. interpretabl htg.r//aee.l.atteccccceeeeeeeeee neural www.uuraallntttttrss.oo.om
Curran Asso- 0006-3444. doi: 10.2307/2333955. http:////w....aaaooo/o/////aaaa/
Harold Hotelling. Relations Between Two Sets of Variates, Biometrika 28333333333333)3377777777 BBt.s//oeoooaaaaaa88333333331110 ISSN
D. H. Hubel and T. N. Wiesel. Receptive fields of single neurones in the cat's striate cor- tex. The Joumaa of Physiology, 148()):574-91,, 1959. ISSN 1469-7793. doi: 10.1113)
1pby:ioo..9559..000333 URL abbs

---

Published as a conference paper at ICLR 2021
D. H. Hubel and T. N. Wiesel. Receptive fields, binocular interaction and functional architecture in the cat's 's visual cortex. The Journal of Physiology, 160(1):106 1962. ISSN 1469-7793. doi:
10. 10.1113/pyysioo.1962..00066.
URL hitps: RtRLaa/Dee//bbyoc..ooaaaaaaaaaa/
David H. Hubel. Exploration of the primary visual cortex, 1955-78. Nature, 299683)5229
October 1982. ISSN 1476-4687. doi: 10.1038/299515a0. URL httt::::www.tttueee.oom
articles/2999
Yerlan Idelbayev. wwwpe//eeaaaaeeeeeyyoeeeeeeeeeee ofnn
cifarle January 2020. URL Utt.httss://gitub.ooo original-date: 2018-01-15T09:500555
akamaster/pytorcch http:orchh_eeset_ccaaaaaee
Michele Insanally, Ioana Carcea, Rachel E Field, Chris Rodgers, Brian DePasquale, Kanaka Rajan, Michael R DeWeese, Badr F Albanna, and Robert C Froemke. Spike-timing-dependaar ensemble encoding by non-classically responsive cortical neurons. eLife. 8:e42409, January doi: 10.7554/eLife.424000 URL https://6ol.og//101115
2019. ISSN 2050-084X.
eLi fLife.42409
Yuta Kanda, Kota S. Sasaki, Izumi Ohzawa, and Hiroshi Tamura. Deleting object selective units in a fully-connected layer deep convolutional networks improves classificcation performanccc.. arXiv:2001.0788 107811 [q-bio], January 2020. URL http:///axxivorr//a//20000155350
arXiv: 2001,07811.
E R Kandel, H Schwartz, and Jessica Chao. Principles of neural science. McGraw-Hill, New York, 2000.
Andrej Karpathy, Justin Johnson, and Li Fei-Fei. Visualizing and Understanding Recurrent Networks. In International Conference on Learning Representations, page 11, 2016.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http://arxiv.orrg/ htt.:v/..rr/.b//oeaaoooooo 1122/.611.688
Alex Krizhevsky. Learning Multiple Layers of Features from Tiny Images. Technical report, 2009.
Matthew I. Leavitt, Florian Pieper, Adam J Sachs, and Julio C Martinez--Truilll, Correlated variability modifies working memory fidelity in primate prefrontal neuronal ensembles. Proceedings of the
National Academy of Sciences of the United States of America, 114(12)122444E25500 2017. doi: 10. 1073/paps (tt.///99..6199991101 URL http: httpp///www.ppsssccc////o/oo///p
pnas.1619949111
Mario Lezcano-Casado. Trivializations for gradient-based optimization on manifolds. In Advances in Neural Information Processing Systems, NeurlPS, pages 9154-9164, 2019.
Yixuan Li, Jason Yosinski, Jeff Clune, Hod Lipson, and John Hopcroft. Convergent learning: Do different neural networks learn the same representations? In Dmitry Storcheus, Afshin Rostamizadeh, and Sanjiv Kumar, editors, Proceedings of the 1st International Workshop on Feature Extraction: Modern Questions and Challenges NIPS 2015, volume 44 of Proceedings of Machine Learning Research, pages 196-212 Montreal, Canada, 11 Dec 2015. PMLR. URL bttp://ppoeeiingg.mLpppssss/sss /vtes////444//// httpa//........a/////////eooo/// htt.///11111laaveee.......m
Peter E. Lillian, Richard Meyes, and Tobias Meisen. Ablation of a Robot's Brain: Neural Networks abs/////////// 1atbb//111.008.77
Under Knife. December 2018.
Lu Lu, Yeonjong Shin, Yanhui Su, and George Em Kariiadakis Dying ReLU and Initialization: Theory and Numerical Examples. arXiv: 1903.06733 [cs, math, stat], November 2019. URL http:://rxiia www.g/ 1911106333333333
arXiv: 1903.06733.
Andrew L. Maas, Awni Y. Hannun, and Andrew Y. Ng. Rectifier nonlinearities improve neural network acoustic models. In ICML Workshop on Deep Learning for Audio, Speech and Language Processing, 2013.
12

---

Published as a conference paper at ICLR 2021
Neil A Macmillan and C Douglas Creelman. Detection theory: A user's guide. Psychology press, 2004.
Richard Meyes, Melanie Lu, Constantin Waubert de Puiseau, and Tobias Meisen. Ablation Studies in Artificial Neural Networks. arXiv:1901.0864: [cs, q-bio], February 2019. URL http://
arXiv: 1901.08644.
Sohie Lee Moody, Steven P. Wise. Giuseppe di Pellegrino, and David Zipser. A Model That Accounts for Activity in Primate Frontal Cortex during Delayed Braching-to-Sampee Task. Journal of Neuroscience, 18(1):399-44 -10, January 1998. ISSN 0270-6474, 1529-2401. doi:
(0.522//EUROSCC.BB-0000000 1998.
18/1/399
Ari Morcos, Maithra Raghu, and Samy Bengio. Insights on representational simi- larity in neural networks with canonical correlation. In Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems 31, pages 5727-5736 Curran Associates, Inc., 2018a. URL hrgp::/rraaarrttteoooeaaaaaraarr
Ari S. Morcos and Christopher D. Harvey. History variability in population dynamics during evidence accumulation in cortex. Nature Neuroscience, 19(12):6672 1681, December 2016. ISSN 1546-1726.
doi: 10.10388nn.4400 4403. URL http:p:::/www.aaaturrr..o//////a
4n34
Ari S. Morcos, David G. T. Barrett, Neil C. Rabinowitz, and Matthew Botvinick. On the importance of single directions for gemeralization. In International Conference on Learning Representations, 2018b. URL httpppss///ppeeeeeeeeeeeerraaaap
Seil Na, Yo Joong Choe. Dong- Hyun Lec, and Gunhee Kim. Discovery of Natural Language Concepts in Individual Units of CNNs. In International Conference on Learning Representations, 2019. URL htt.:tttpss//opeeeeeeeee// httpee/eeeeoooeeeeeessseeee.ee
Anh Nguyen, Alexey Dosovitskiy, Jason Yosinski, Thomas Brox, and Jeff Clune. Synthesizing the preferred inputs for neurons in neural networks via deep generator networks. In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Gamett, editors, Advances in Neural Information Processing Systems 29, pages 3387-3395. Curran Associates, Inc., 2016. URL http'
the theeeeesiiinngtttte-eeeeeeeeee inputs neuroeu in neural networks via deep -eneratoo--
Ramon Nogueira, Nicole E. Peltier. Akiyuki Anzai, Gregory C. DeAngelis, Julio Martínez- Martinez-Truilll, and Rubén Moreno-Bote. The Effects of Population Tuning and Trial-by-Triall Variability on Information Encoding and Behavior. The Journal of Neuroscience, 40(5): 1066-1083, January 2020. ISSN 0270-6474, 1529-2401. /ttp://///....111///.//2 2 0006-8993. doi: 10. http:////....ao.ooo.o...././//..
doi: 10. 152./INEUROSCIOOSSS5 2019. URL http:
/www.
J. O'Keefe and J. Dostrovsky. The hippocampus as spatial map. Preliminary evidence from unit activity in the freely-moving rat. Brain Research, 34(1):171-175, November 1971. ISSN
Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Feature Visualization, Distill, 2(11):e7, 2011
doi: 00.23915/distill.0000 URUhttps://distiipppp 2017/featuxe-vvesaallaaaaaa
Chris Olah, Arvind Satyanarayan, Ian Johnson, Shan Carter, Ludwig Schubert, Katherine Ye, and Alexander Mordvintsev. The Building Blocks of Interpretability. Distill, 3(3) March 2018. ISSN 2476-0757. doi: (0.23995/diistill.00000 URL /
bui Building-blocks
Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom In: An Introduction to Circuits. Distill, 50),000005))00000000,0, March 2020. ISSN 2476-0757. doi:
Atdistiil.0002..000.. h|t/o//0///////////////////////0
13

---

Published as a conference paper at ICLR 2021
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin. Natalia Gimelshein, Luca Antiga, Alban Desmaison, An- dreas Kopf. Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chil- amkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An Im- perative Style, High-Performance Deep Learning Library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'textquotesinglc Alché-Bus E. Fox. and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 8024-8035. Curran Associates, Inc..
2019. http:///ecccco/eeeeeaccao///////
J Andrew Pruszynski and Joel Zylberberg. The language of the brain: real-world neural popula- tion codes. Current Opinion in Neurobiology, 58:30-36, October 2019. ISSN 09594388. doi: 10.101 conb 2019.06.005. URL
pi1 p11/5954381180022
Alec Radford, Rafal Jozefowicz, and llya Sutskever. Learning to Generate Reviews and Discovering Sentiment. arXiv: 1104::100401144 [cs], April 2017. /ttpp/tpe//aartttooo////////1000
arXiv: 1704.01444.
Ivet Rafegas, Maria Vanrell, Luis A. Alexandre, and Guillem Arias. Understanding trained CNNs by indexing neuron selectivity. Pattern Recognition Letters, page 50167865519300209, October 2019. ISSN 01678655. doi: 10. (e..1///jppatrc..220 patte 10.011. URt htt.:tttp:///rxii.or////// 1702.00382 arXiv: 1702.00382.
Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstenn SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 6076-6085. Curran Associates,
Inc., 2017.
David Raposo, Matthe T Kaufman, and Anne K Churchland, A category-free neural population sup- ports evolving demands during mecision Nature Neuroscience, 17(12:1784-1779, 2014. doi: 10.1038/nn.3865. URL
e
Shreya Saxena and John P Cunningham. Towards the neural population doctrine. Current
Opinion in Neurobiology, 55:103-111, April 2019. ISSN 0959-4388. doi: 10.1016/
URL http:p//ww..sieeeeeeeeeeee..ooos
011/80959338183093
Krishna V Shenoy, Maneesh Sahani, and Mark M Churchland. Cortical con- trol of arm movements: dynamical systems perspective. Annual Review of
Neuroscience, 36:337-359, 2013.
doi: 1011166aammeerrrrum--0001111---- URL
Charles S. Sherrington. The integrative action of the nervous system. The integrative action of the nervous system. Yale University Press, New Haven, CT, US 1906. doi: 10. 10.1037/17788-000
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale im- age recognition. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps. arXiv:1312.608 [cs], April 2014. URL WWRLLtt.pp///arrii.rr///./...
2.6044 arXiv: 1312.6034.
Stephen M. Smith, Thomas E. Nichols, Diego Vidaurre, Anderson M. Winkler, Timothy E. J. Behrens, Matthew F. Glasser, Kamil Ugurbil, Deanna M. Barch, David C. Van Essen, and Karla L. Miller. A positive-negative mode of population covariation links brain connectivity, demographics and behavior. Nature Neuroscience,
November 2015. ISSN 1546-1722 doi:
10. 10.1038/nn.4133 4125. httph//pps.//...tttrrcc///a/artt
14

---

Published as a conference paper at ICLR 2021
Harold Stanislaw and Natasha Todorov. Calculation of signal detection theory measures. Behavior Research Methods, Instruments, & Computers, 31(1):137-149, March 1999. ISSN 1532-5970. doi: 10.3758/BF03207704 UKLhtttpss//boiiorg//..33333 /t1p:///000.331///222200000
David Sussillo, Mark M Churchland, Matthew T Kaufman, and Krishna V Shenoy. A neural network that finds a naturalistic solution for the production of musele activity. Nature Neuroscience, 18())1025-1033, 2015. doi: 10. 10.1038/mn.402.
URL pponoaac.ee777763eeeeeeeeeeeaadr
Jumpei Ukita. Causal importance of wrwentttiio selectivity for generalization in image recognition.
September 2018.
Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Courna- peau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nel- son, Eric Jones, Robert Kern, Eric Larson, C. J. Carey, Ilhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quin- tero, Charles R. Harris, Anne M. Archibald, Antônio H. Ríbeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy O Contributors. SciPy 1. J.O-Fundamental Algorithms for Scientific Com- puting in Python. arXiv: 1907.10111 [physics], July 2019. URL
1907.10121 arXiv: 1907.10121.
Michael Waskom, Olga Botvinnik, Drew O'Kane, Paul Hobson, Saulius Lukauskas, David C. Gemperline. Tom Augspurger, Yaroslav Halchenko, John B. Cole, Jordi Warmenhoven, Ju- lian de Ruiter, Cameron Pye, Stephan Hoyer, Jake Vanderplas, Santi Villalba, Gero Kunter, Eric Quintero, Pete Bachant, Marcel Martin, Kyle Meyer, Alistair Miles, Yoav Ram, Tal Yarkoni, Mike Lee Williams, Constantine Evans, Clark Fitzgerald, Brian, Chris Fonnesheck, Antony Lee, and Adel Qalich. mwaskom//saboon:: v0.8.1 (September 2017), September 2017. URL https:/7bo.oog//0088///eo688888
T. N. Wiesel. Postmatal development of the visual cortex and the influence of environment. Nature, 299(5884):5833-51, October 1982. ISSN 0028-0836. doi: 10. 10.1038/2995.3a..
Jason Yosinski, Jeff Clune, Anh Nguyen, Thomas Fuchs, and Hod Lipson. Understanding Neural Networks Through Deep Visualization. In ICML Workshop on Visualization for Deep Learning, 2015.
arviv: 1506.06579.
Matthew D. Zeiler and Rob Fergus. Visualizing and Understanding Convolutional Networks. In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars, editors, Computer Vision ECCV 2014, pages 818 0833, Cham, 2014. Springer International Publishing. ISBN 778-3-311--05900
B. Zhou, D. Bau, A. Oliva, and A. Torralba. Interpreting Deep Visual Representations via Network Dissection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 4199:2213122145, September 2019. ISSN 1939-3539. doi: 10.1109/TPAMI. 20122222112222 2858559.
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Object Detectors Emerge in Deep Scene CNNs. In International Conference on Learning Representations, April 2015.
attpaabb//111111555111 arxiv: 1412.6856.
Bolei Zhou, Yiyou Sun, David Bau, and Antonio Torralba. Revisiting the Importance of Individual Units in CNNs via Ablation. 1eXiv:1806.00 02891 [cs], June 2018. URL http::/arxxi..oor
abs / 1886.02861111 arXiv: 1806.02891.
Joel Zylberberg. The role of untuned neurons in sensory information coding, bioRxiv, page 134379, May 2018. doi: 10.1101//34379 URL
134379v6
Joel Zylberberg, Jon Cafaro, Maxwell H Turner, Eric Shea-Brown, and Fred Rieke. Direction- Selective Circuits Shape Noise to Ensure a Precise Population Code. Neuron, 89(2):69--33,, 2016 doi: 1ww.0////..nuoo.2000000 11.019. http:./.........................
pr inkinks
15

---

Published as a conference paper at ICLR 2021
A APPENDIX
A.1 MODELS, TRAINING, DATASETS, AND SOFTWARE
Our experiments were performed on ResNet18 and ResNet50 He et al., 2016) trained on Tiny Imagenet (Fei-Fei al., 2015), and ResNet20 (He al. (2016); code modilied from [delbayev (2220) and a VGG VGGl6-likk network (Simonyan and Zisserman, 2015). both trained on CIFARIO Krizhevsky 2 All models were trained using stochastic gradient descent (SGD) with momentum 0.9 and weight decay= = 0.0001.
The maxpool layer after the first batchnorm layer (see He et al. (2016) was removed because of the smaller size of Tiny Imagenet images compared to standard ImageNet images (64x64 vs. 256x256, respectively). ResNet18 were trained for 90 epochs with minibatch size of 4096 samples with a learning rate of 0.1, multiplied (annealed) by 0.1 at epochs 35, 50, 65, and 80. ResNet50 was trained identically, except with a batch size of 1 1400 samples. Tiny Imagenet (Fei-Fei et al. 2015) consists of 500 training images and 50 images for each of its 200 classes. We used the validation set for testing and created a new validation set by taking 50 images per class from the training set, selected randomly for each training run.
The VGG16-like network is identical to the batch norm VGG16 in Simonyan and Zisserman (2013). except the final two colne-connected layers of 4096 units each were replaced with single 5 512-unit layer. ResNet20 and VGG16 were trained for 200 epochs using a minibatch size of 256 samples. ResNet20 were trained with learning rate of 0.1 and VGG16 with a learning rate of 0.01, both annealed by 10-1 at epochs 100 and 150. We split the 50k CIFARIO 10 training samples into a 45k sample training set and 5k validation set, similar to our approach with Tiny Imagenet.
All experimental results were derived from the test set with the parameters from the epoch that achieved the highest validation set accuracy over the training epochs. 20 replicates with different random seeds were run for each hyperparameter set, except for ResNet50, which only used 5 replicates per hyperparameter set. Selectivity regularization was not applied to the output (logit) layer, nor was the output layer included any of our analyses.
Experiments were conducted using PyTorch (Paszke et al. 2019) analyzed using the SciPy ecosystem (Virtanen et al., 2019), and visualized using Seaborn Waakomm et al. 2017).
A.2 EFFECT OF SELECTIVITY REGULARIZER ON TRAINING TIME
We quantified the number of training epochs required to reach 95% of maximum test accuracy (t95). The 895 without selectivity regularization (ta=n) for ResNet20 is 45+15 epochs (median + IQR). in [-2, 0.7] had overlapping IQRs with = 0. For ResNet18, 1 35 + 1, while 895 for in [-2, 1] was as high as 1.5. Beyond these ranges, the f exceeded 1.5 x 95 variable.
and/or was highly
A.3 CCA
A.3.1 AN INTUITION
We used Canonical Correlation Analysis (CCA) to examine the effects of class selectivity regulariza- tion hiden layer representations. CCA a statistical method that takes two sets of multidimensional variates and finds the linear combinations of these variates that have maximum correlation with
each other (Hotelling 1936). Critically, CCA is invariant to rotation and other invertible affine transformations. CCA has been productively applied to analyze and compare representations in (and between) biological and neural networks (Sussillo et al. 2015 Smith et al. 2015 Raghu et al. 2017]
Morcos et al. 2018a; Gallego et al. 2018).
We use projection-weighted CCA (PWCCA), a variant of CCA introducted in Morcos et al. (20 8a) that has been shown to be more robust to noise than traditional CCA and other CCA variants (though for brevity we just use the term "CCA" in the main text). PWCCA generates a scalar value, P. that can be thought of as the distance
dissimilarity between the two sets of multidimensional variates, L, and Lz. For example, if L2 = L,, then Lt, = O. Now let R be a rotation matrix. Because CCA is invariant to rotation and other invertible affine transformations, if T-g RI, (i.e. if Ta2
16

---

Published as a conference paper at ICLR 2021
is rotation of L1). then PL1.L2 0. In contrast, traditional similarity metries such Pearson's Correlation and cosine similarity would obtain different values if Lg = L, compared to L = RL1- We use the PWCCA implementation available at httttss/gghbbb.oo//ggogll///ae as provided in Morcos et al. 2018a).
A.3.2 OUR APPLICATION
As an example for the analyses in our experiments, L1 is the activation matrix for a layer in a network that was not regularized against class selectivity (i.e. 0), and L2 is the activation matrix for the same layer a network that was structured and initialized identically, but subject to regularization against class selectivity (i.e. representations to be rotated (or to undergo to some other invertible affine transformation), then due differences in random seeds and/or other stochastic factors
0). If regularizing against class selectivity causes the network's
Pla. In practice Pla. in the training process, so we can determine a threshold value and say PLI.L2 S e. If regularizing against class selectivity instead causes non-affine transformation to the network's representations, then PL1.L2 E.
In our experiments we empirically establish a distribution of e values by computing the PWCCA distances between PLza where Lza and L2s are two networks from the set of 20 replicates for given hyperparameter combination that differ only in their initial random seed values (and thus have the same a). This gives (2) 190 values of e. We then compute the PWCCA distance between each [[,, L2) replicate pair, yielding distribution of 20 20 400 values of PL,, which compare to the distribution ef
A.3.3 FORMALLY
For the case of our analyses. let us start with a dataset X, which consists of M data samples {Z1,...IM}) J. Using the notation from Raghu et al. (2017). the scalar output (activation) of a single neuron i on layer & in response to each data sample collectively form the vector
2 = (s)((((()))...
We then collect the activation vector 2 of every neuron in layer into matrix L {2f ..., 2My of size N x M, N is the number of neurons in layer 4, and M is the number data samples. Given two such activation matrices L1. of size Na M, and L2, of size No x M, CCA finds the vectors (in Na) and s (in R RN,,, such that the inner product
(w L1, ST L2)
|w Lill- 1l. || L2]
is maximized,
17

---

Published as a conference paper at ICLR 2021
A.4
REGULARIZING TO DECREASE CLASS SELECTIVITY IN RESNETI8 AND RESNET20
0.45
Regularization Scale (o)
b)
0.35
0.40
-100.0
-0.7
Cass Selectivity Index
0.30
Oass Selectvity Index
0.35
-30.0
-0.4
-10.0
-0.3
0.30
-5.0
-0.2
-0.1
-1.0
0.0
0.15
0.20
0.10
0.15
0.10
0.05
0.05
00
10
10.0
30.0
100
Regularization Stale (a)
Layer
d)
0.5
Regularization Scall (o)
Class Selectivity Index
100.00
-0.7
Closs Selectivtsy Index
0.4
-3.0
-0.4
-0.3
0..15
0.3
5.0
-0.2
0.10
0.2
-1.0
0.1
0.0
0.0
10
12
3000
30.0
100.
10
12 1.5
Layer
Regulariation Scale (0)
Figure Al: Manipulating class selectivity by regularizing against the loss function. (a) Mean class selectivity index (y-axis) as a function of layer (x-axis) for different regularization scales (ar; denoted by intensity of blue) for ResNet18. (b) Similar to (a), but mean is computed across all units in a network instead of per layer. (b) Similar to (a), but mean is computed across all units in network instead of per layer. (e) and (d) identical to and (b), respectively, but for ResNer20. Error bars denote bootstrapped 95% confidence intervals.
A.5 DECREASING CLASS SELECTIVITY WITHOUT DECREASING TEST ACCURACY IN RESNET20
a)
b)
90.75
91.0
Tel. Accuracy
Test Accurace
90.5
Tes Accurary
90.25
90.00
Regularivation
88.5
89.75
(o)
-5.0
-0.3
20
88.5
89.25
-0.1
0.7
0.0
% 30 100
Meen Class Selectivity
Regularization Scale (a)
Regularization
Figure A2: Effeets of redueing class selectivity test accuracy in ResNet20 trained CIFARIO. (a) Test accuracy (y-axis) as function of regularization scale (a, x-axis intensity of blue). (b) Identical to (a), but for subset of a values. The center of each violin plot contains boxplot, in which the darker central lines denote the central two quartiles. different values of a. Error bars denote 95% confidence intervals. "p 0.05, **p 10 difference from t-test, Ronferroni-corrected..
(c) Test accuracy (y-axis) as function of mean class selectivity (x-axis) for
18

---

Published as a conference paper at ICLR 2021
A.6 CCA RESULTS FOR RESNET20
a)
CCA Distance Ratio
1.74
1.52
1.15
10.2
Regularization Scale (a)
Figure A3: Using CCA to check whether class selectivity is rotated off-axis in ResNet20 trained on CIFARIO. Similar to Figure we plot the average CCA distance ratio (y-axis) function of a (x-axis, intensity of blue). The distance ratio is significantly greater than the baseline for values of a (p < x 10-9. paired (-test). Error bars = 95% confidence intervals.
A.7 CALCULATING AN UPPER BOUND FOR OFF-AXIS SELECTIVITY
As an additional control to ensure that our regularizer did not simply shift class selectivity to off-axis directions in activation space, we calculated an upper bound on the amount of class selectivity that could be recovered by finding the linear projection of unit activations that maximizes class selectivity, To do so, we first collected the validation
activation vector 2 of every neuron in layer into matrix Aeat {-í:-- ..... $4 of size M x N, where M is the number of data samples in validation
set and N is the number of neurons in layer We then found the projection matrix W E IR NxN that minimizes the loss
loss = (1-SS(Apa
such that
| W - 111
i.e. W is orthonormal, where SI is the selectivity index from Equation 1 We constrained W to be orthonormal because the non-orthonormal solution to maximizing selectivity is degenerate: project all axes onto the single direction activation space with the highest class selectivity. We used Lezcano-Casado 2019)'s 's toolbox to constrain W to be orthonormal. Because ST requires inputs 2 0, we shifted the columns of AW by subtracting the columnwise mininum value before computing ST. The optimization was performed using Adam (Kingma and Ba (015) with learning rate of 0.001 for 3500 steps or until the magnitude of the change loss was less than 10-6 for 10 steps. W was then used to project the activation matrix for the test set Atest, and the selectivity index was calculated for each axis of the new activation space (i.e. each column Atest W) after shifting the columns of Atest W to be 2 0. A separate W was obtained for each layer of each model and for each replicate and value of o.
19

---

Published as a conference paper at ICLR 2021
0.45
Regulanzation
Class Selectivity Index
0.40
(Upeer Bound)
-2.
0.2
0.30
-10
4..1
0.20
0.15
0.10
0.05
Layer
b)
0.200
Regularization
0:375
Class Selectivity Index
0.4
-04
0.150
(Upper Bound)
-20
-0.2
Soleziitty
0.100
0.075
0.050
Optimized Projection (Upper bound)
0.025
0.0
12
14
16
18
Regularization Scale
Layer
Figure A4: An upper bound for off-axis class selectivity. (a) Upper bound on class selectivity (y-axis) as function of layer (x-axis) for different regularization scales (a; denoted by intensity of blue) for ResNet18 trained on Tiny ImageNet.
(b) Mean class selectivity (y-axis) function of regularization scale (o; x-axis) the upper bound class selectivity
for ResNet20 trained CIFARIO. Diamond-shaped data points for a linear projection of activations as described in Appendix A.7 while circular points denote the amount of axis-aligned class selectivity for the corresponding values of a. (c) (a), but for ResNet20 trained on CIFARIO. Error bars 95% confidence intervals.
20

---

Published as a conference paper at ICLR 2021
A.8
REGULARIZING TO INCREASE CLASS SELECTIVITY RESNET18 AND RESNET20
1.0
b) 1.0
Class Class Selectivity Index
0.9
Class Selezivity Index
0.9
0.8
0.0
0.7
Regularization
0.7
0.8
0.0
1.0
0.5
0.2
5.0
0.5
0.4
0.3
10.0
0.3
0.7
100.0
0.2
0.0
10.0
30,0
100.0
Layer
Regularization Scale
d)
0.6
Regularization Scale
0.0
0.1
2.0
Class Selectivity Index
0.2
0.5
0.3
10.0
Seleclivity
0.5
0.7
100.0
0.4
0.4
0.3
10.0
30,0
100
0.0
10
16
18
Regularization Scale (o)
Layer
Figure A5: Regularizing to increase class selectivity (a) Mcan class selectivity index (y-axis) as function of layer (x-axis) for different regularization scales (o; denoted by intensity of red) for ResNet18, (b) Similar to (a), but mcan is computed across all units in network instead per layer. (c) and (d) are identical (a) and (b), respectively, but ResNet20. Note the inconsistent effect larger a values (c) (d) addressed Error bars denote bootstrapped 95% confidence intervals.
Appendix A.10
21

---

Published as a conference paper at ICLR 2021
A.9
ADDITIONAL EFFECTS OF CLASS SELECTIVITY REGULARIZATION ON TEST ACCURACY IN RESNET20
100
90
80
Text Accuracy
Test Accuracy
60
80
60
Regudarivation
50
Scale (o)
75
1.0
70
30
0.2
2.0
5.0
65
20
10.0
20
0.2 0.7
0.2
0.3
0.4
0.5
0.6
Regularization Scale
Regularization Scale (a)
Moan Class Selectivity
Figure A6: Effects of increasing class selectivity on test accuracy on ResNet20 trained on CIFARIO. (a) Test accuracy (y-axis) as function of regularization scale (a; x-axis, intensity of red). (b) Identical to (a), but for subset of values. The center of each violin plot contains a boxplot, in which the darker central lines denote the central two quartiles. (c) Test accuracy (y-axis) as function of mean class selectivity (x-axis) for different values of a. Error bars denote 95% confidence intervals. "p < x 10-4 **p < x 10 difference from a = 0, t-test, correctee.......
A.10 SINGLE UNIT NECROMANCY
Lethal ReLUs The inconsistent relationship between a and class selectivity for larger values of led to question whether the performance deficits were due alternative factor, such as the optimization process, rather than class selectivity per se. Interestingly, we observed that ResNet20 regularized to increase selectivity contained significantly higher proportions of dead units, and the number of dead units is roughly proportional to alpha (see Figure A8a). Regularizing to increase class selectivity did not cause units to die in ResNet18 trained on Tiny ImageNet except very extreme values of a (a 2 30), though even at these values the proportion of dead units never exceeded 0.03 (Figure A7p. Removing the dead units ResNet20 makes the relationship between regularization and selectivity in ResNet20 more consistent at large regularization scales (see Appendix A8).
The presence of dead units is not unexpected, as units with the ReLU activation function are known to suffer from the "dying ReLU problem" (Lu et al. [019): If, during training, a weight update causes a unit to cease activating response to all training samples, the unit will be unaffected by subsequent weight updates because the ReLUU gradient at x S O is zero, and thus the unit's activation will forever remain zero The dead units could explain the decrease in performance from regularizing to increase selectivity as simply decrease in model capacity.
Fruitless resuscitation One solution to the dying ReLU problem is to use a leaky-Rel.U activation function (Maas al. 2013), which has non-zero slope, b (and thus non-zero gradient) for x S O. Accordingly, we re-ran the previous experiment using units with leaky-ReLU activation in an attempt to control for the potential confound of dead units. Note that because the class selectivity index assumes activations 2 0, we shifted activations by subtracting the minimum activation when computing selectivity for leaky-ReLUs. If the performance deficits from regularizing for selectivity are simply due to dead units, then using leaky-ReLUs should rescue performance. Alternatively, if dead units are not the cause of the performance deficits, then leaky-ReL.Us should not have an effect.
We first confirmed that using leaky-Rel.Us solves the dead unit problem. Indeed, the proportion of dead units is reduced to in all networks across all tested values of Despite complete recovery of the dead units, however, using leaky-ReLUs does not rescue class relectivityinnduce performance deficits (Figure A9). While the largest negative slope value improved test accuracy for larger values of a, the improvement confirm that dead units cannot explain the rapid and catastrophic effects of increased class selectivity on performance.
minor, and increasing a still had catastrophic effects. These results
22

---

Published as a conference paper at ICLR 2021
Proportion of Dead Units
0.8
0.6
0.2
0.0
-0.0
0.1
0.2
0.3
0.4
0.7
1.0
2.0
5.0 10.0 30.0 100.0
Regularization Scale
Figure A7: Regularizing to increase class selectivity does not cause units to die in ResNet18 trained on Tiny ImageNet. (a) Proportion of dead units (y-axis) for different for different regularization scales (a; x-axis, intensity red). Error bars denote 95% confidence intervals.
10
Roguio lot Scale
02-
0.
Test Accuncy
06-
Regularoation Scale fot
100.0
10
04
40
0.0
300
0.4 Noan Selectivity
Regularizatin Scale
Figure A8: Removing dead units partially stabilizes the effects of large positive regularization scales in ResNet20. (a) Proportion of dead units (y-axis) as function of layer (x-axis) for different regularization scales (o, intensity of red). (b) Mean class selectivity index (y-axis) as function of regularization scale (a; x-axis and intensity of red) after removing dead units. Removing dead units from the class selectivity calculation establishes a more consistent relationship between and the mean class selectivity index (compare to Figure A5d). (c)
Test accuracy (y-axis) as a function of mean class selectivity (x-axis) for different values a after removing dead units from the class selectivity calculation. Error bars denote 95% confidence intervals. *p < x 10 **p 5 10 difference from 0 difference from 0, t-test, Bonferroni-comeetee All results shown for ResNet20.
23

---

Published as a conference paper at ICLR 2021
b)
90
80
80
Taxt Accuracy
70
Accuracy
60
negatvve dlppe
Regulerization Scale (a)
0.0
50
40
0001
0.1
1.0
30
01
0.3
5.0
10.0
30.0
100.0
20
0.4
0.5
0.6
0.7
0.8
Regularization
Mean Class Selectivity
Figure A9: Reviving dead units does not rescue the performance deficits caused by increasing selectivity in ResNet20. (a) Test accuracy (y-axis, as a function of regularization scale (a: x-axis) for different leaky- ReLU negative slopes (intensity of red). Leaky-Rel.Us completely solve the dead unit problem but do not fully rescue test accuracy for networks with 0. (b) Mean class selectivity index (y-axis) as function of regularization
scale (a; x-axis and intensity of red) for leaky-Rel.U negative slope 0.5. *p 0.001, **p < 2 10 10-4 *tpp < 5 10 difference from a 0, t-test, conferroni-coreette Error bars denote bootstrapped 95% confidence intervals.
A.11 RULING OUT A DEGENERATE SOLUTION FOR INCREASING SELECTIVITY
One degenerate solution to generate a very high selectivity index for unit to be silent for the vast majority of inputs, and have low activations for the small set of remaining inputs. We refer to this scenario as "activation minimization". We verified that activation minimization does not fully account for results by examining the proportion of activations which elicit non-zero activations in the units in our models. our regularizer is indeed using activation minimization to generate high selectivity in individual units, then regularizing to increase class selectivity should cause most units to have non-zero activations for only very small proportion of samples. In ResNet18 trained on Tiny ImageNet, we found that sparse units, defined units that do not respond at least half of the data samples, only constitute more than 10% of the total population at extreme positive regularization scales (a 2 10; Figure A well after large performance deficits >4% emerge (Figure 3) In ResNet20 trained on CIFARIO, networks regularized to have higher class selectivity (i.e. positive a) did indeed have more sparse units (Figure A10b). However, this effect does not explain away our findings: by 0.7, the majority of units respond to over 80% of samples (i.e. they are not sparse), but test accuracy has already decreased by 5% (Figure (6). These results indicate that activation minimization does not explain class selectivity-relate changes in test accuracy.
Propertion of Samoles
Elicting Non-2wre Act
0.9
Elicking Samplcs
0.7
0.6
0.4
0.5
0.2
0.3
Regularizatin Scale (c)
Regullrizaaionn Scale (o)
Figure A10: Activation mininization does not explain selectivity performance changes. (a) Pro- portion samples eliciting a non-zero activation (y-axis) vs. regularization scale (a: x-axis) in ResNet18 trained on Tiny ImageNet. Data points denote individual units. Boxes denote IQR, whiskers extend 2 2x 2x1QR. Note that the boxes are very compressed because the distribution is confined almost entirely to y=1.0 for all values of x. (b) Identical to (a), but for ResNer20 trained on CIFAR10.
24

---

Published as a conference paper at ICLR 2021
A.12 RESULTS FOR VGG16
Modulating class selectivity in VGGI6 yielded results qualitatively similar to those we observed in ResNet20. The regularizer reliably decreases class selectivity for negative values of a (Figure ATT], and class selectivity can be drastically reduced with little impact on test accuracy (Figure A12 Although test accuracy decreases significantly at a -0.1 (p- 0.004, conferroni-correcter t-test). the effect is small: it is possible to reduce mean class selectivity by a factor of 5 with only a 0.5% 1% drop test accuracy.
decrease in test accuracy, and by a factor of 10 to 0.03 with only ~1%
Regularizing to increase class selectivity also has similar effects in VGG16 and ResNer20. Increasing causes class selectivity to increase, and the effect becomes less consistent at large values of (Figure A5). Although the class selectivity-induced collapse in test accuracy does not emerge quite as rapidly in VGG16 as it does in ResNet20, the decrease in test accuracy is still significant at the smallest tested value of a (a- 0.1, p 0.02, cnnfferroni-correcttd t-test), and the effects test accuracy of regularizing to promote vs. discourage class selectivity become significantly different at == 0.3 (p 10 - Wilcoxon rank-sum test; Figure A15). Our observations that class selectivity is neither necessary nor sufficient for performance in both VGG16 and ResNet20 indicates that this is likely general property of CNNs.
is worth noting that VGG16 exhibits greater class selectivity than ResNet20. In the absence of regularization (i.e. a 0), mean class selectivity in ResNet20 is 0.22, while in VGG16 it is 0.35, 1.6x increase. This could explain why positive values of a seem to have a stronger effect on class selectivity in VGG16 relative ResNet20 (compare Figure AS and Figure A13 also see Figure A16
a)
Regularization Scale (o)
b)
0.35
Class Selectivity Index
0.5
-100.0
0.7
0.30
-30.0
-0.4
0.4
-10.0
-03
Closs Selectiiity
0.25
-5.0
0.2
0.20
0.3
-011
0.15
-1.0
0.0
0.10
0.2
0.05
0.1
0.00
10.0
30.0
300
10
12
Layer
Regularication Scale (e)
Figure A11: Regularizing to decrease class selectivity in VGG16. (a) Mean class selectivity index (y-axis) as a function of layer (x-axis) for different regularization scales (o; denoted by intensity of blue) for VGG16. (b) Similar to (a), but mean is computed across all units in network instead of per layer. Error bars denote hootstrapped 95% confidence intervals.
a)
91.0
90
80
85
Test Accuracy
Accurecy
Accuracy
80
Regularization
60
96.0
Scale (0) -0.4
75
88.5
2.0
40
70
1100
-0.1
88.0
05
-0.7
0.0
20
88.5
60
-0.3 -044 -0.7
0.1 0.2
Regularization Scale (a)
Mean Class Selectivity
Reqularization Scale
Figure A12: Effeets of reducing class selectivity on test accuracy in VGG16, (a) Test accuracy (y-axis) as a function of regularization scale (a, x-axis and intensity of bluc). (b) Identical to (a), but for subset of values. The center of cach violin plot contains boxplot, in which the darker central lines denote the central two quartiles. (c) Test accuracy (y-axis) as function of mean class selectivity (x (x-axis) for different values a. Error bars denote 95% confidence intervals. *p < 0.005, **p < 5 x 10-6, ***p < 5 x 10 60 difference from = O, t-test, Bonfereomi-oorrected. All results shown are for VGG16.
25

---

Published as a conference paper at ICLR 2021
1.0
b)
Cless Selectivity Index
Class Selectvity Index
0.7
0.6
Reguiorization Scale
0.6
0.0
1.0
0.5
5.0
0.2
0.4
30.0
100.0
0.0-1
100
30.0
100.
Layer
Regularization Scale
Figure A13: Regularizing increase class selectivity in VGG16. (a) Mean class selectivity index (y-axis) function of layer (x-axis) for different regularization scales (a; denoted by intensity of red) for VGGI6. (b) Similar to (a), but mean is computed across all units in a network instead of per layer. Error bars denote bootstrapped 95% confidence intervals.
a)
90.5
90
80
90.0
Test Accurac3
60
89.5
89.0
88
Scale (m)
Reqularization
0.3
0.1
0.7
88.0
86
0.
1.0
87.0
0.0
0.1
0.2
0.3
0.4
0.4 0.5 0.6
0.7 Moan Class Selectivity
0.8
Regularization Scale
Requlartzation
Figure A14: Effects increasing class selectivity on accuracy VGG16. (a) Test accuracy (y-axis) as a function of regularization scale (a, x-axis and intensity of red). (b) Identical to (a), but for a subset of a values. The center of cach violin plot contains boxplot, in which the darker central lines denote the central quartiles. (c) Test accuracy (y-axis) as function of mean class selectivity (x-axis) for different values of Error bars denote 95% confidence intervals. "p < 0.05, **p 5 10 ***p < 10 difference from a 0, t-test, Bonferromi-corrected. All results shown are for VGG16.
b)
92
91
90
Test Accuracy
Ted Accurcy
89
80
signtar)
8 [penalize selectivity)
87
(promote selectivity)
86
signial (penelize selectivity) fpromote selectivity)
8
85
10.0
30.0
100.0
84
0.0
0.1
0.2
0.3
0.4
|Regularization Scale
|egularization Scale
Figure A15: Regularizing to promote vs. pemalize class selectivity in VGG16. (a) Test accuracy (y-axis) as function of regularization scale magnitude ([a]; x-axis) when promoting (a: > 0, red) penalizing (a: blue) class selectivity in VGG16. Error bars denote bootstrapped 95% confidence intervals. Identical (a), but for subset |ar] values. *p < 0.05, *ap < 10-3 **3 < 10-6 difference between a < Wilcoxon rank-sum test, Bonferroni-corrected..
26

---

Published as a conference paper at ICLR 2021
100
b) 1.0
Madel Reshec20 VGG16
Model ResNet20
Test Accuncy
80
Class Selectivity
VGG16
60
40
0.4
20
0.2
0.0
100.0
0.3 0.2
0.0
Regularization Scale (c)
|egullarizaioon Scale [o]]
Figure A16: Effects of class selectivity regularization in ResNet20 vs VGG16. (a) Test accuracy (y-axis) function of regularization scale (a x-axis) for ResNet20 (cyan) and VGG16 (orange), Error bars denote bootstrapped 95% confidence intervals. (b) Class selectivity (y-axis) as function of regularization scale (a for Resnet20 (cyan) and VGGI6 (orange).
A.13 ADDITIONAL MEASURES OF CLASS INFORMATION IN SINGLE UNITS
In order to confirm that the effect of the regularizer is not unique to our chosen class selectivity metric, we also examined the effect of our regularizer on two different measures of class information in single units: the "precision" metric for class selectivity (Zhou et al.) 2015 2018 Gale et al. 2019)) and mutual information (Cover 1999).
The precision metric is calculated by finding the N images that most strongly activate given unit, then finding the image class C, that constitutes the largest proportion of the N images. Precision is defined as this proportion. For example, if N = 200, and the "cats" " class, with 74 samples, constitutes the largest proportion of those 200 activations for a unit, then the precision of the unit is 74 0.34. 200
Note that for given number of classes C. precision bounded by [ 1]. thus in our experiments the lower bound on precision is 0.1. Zhou et al. (2015) used N = 60, while Gale et al. (2019) used N 100. We chose to use the number of samples per class in the test set data and thus the largest possible sample size. This yielded N 1000 for CIFARIO 10 and N 50 for Tiny Imagenet.
The class selectivity regularizer has similar effects on precision as it does on the class selectivity index. Regularizing against class selectivity has a consistent effect on precision (Figure (Figure[A17). while regularizing to promote class selectivity has a consistent effect in ResNet18 trained on Tiny ImageNet and for smaller values of a in ResNet20 trained on CIFARIO 10. However, the relationship between precision and the class selectivity index becomes less consistent for larger positive values of a in ResNet20 trained on CIFARIO. One explanation for this is that activation sparsity is valid solution for maximizing the class selectivity index but not precision. For example, a unit that responded only to ten samples from the class "cat" and not all to the remaining samples would have class selectivity index of 1, but a precision value of 0. 11. This seems likely given the increase in sparsity observed for very large positive values of (see Appendix I..II.
The effects of the class selectivity regularizer on mutual information are nearly identical to its effects on precision. Regularizing for and against class selectivity has a consistent effect on mutual information in ResNet18 trained on Tiny ImageNet (Figure (18). Regularizing to reduce class selectivity also has a consistent effect in ResNer20 trained on CIFAR1O. but the relationship between mutual information and the class selectivity index becomes less consistent for larger positive values of a. However, given that class selectivity and mutual information are very different quantities, and that Morcos et al. (2018b) did not observe clear relationship between mutual information, class selectivity, and individual unit importance (as measured by impact of ablation), we hesitate to make strong conclusions about the role of mutual information in selectivity regularizer.
While there are additional class selectivity metrics that we could have used to further assess the effect of our regularizer, many of them are based relating the activity of neuron to the accuracy of the network's 's output(s) (e.g. top class selectivity Gale et al. (2019) and class correlation Li et al. (2013): Zhou al. (2018)), confounding classification accuracy and class selectivity. Accordingly, these metrics are unfit for use in experiments that examine the relationship between class selectivity and classification accuracy, which exactly what we do here.
27

---

Published as a conference paper at ICLR 2021
a) 0.18
Regularization Scale lo)
b)
-100.0
-0.7
0.13
0.16
30.0
-0.4
0.12
0.14
5.0
-0.2
Prediion
Procision
2.0
-0.1
0.10
0.12
0.09
0.10
0.08
0.07
0.06
80
12
10,0
100.0
12
14
Layer
Regularization Scale (a)
0.40
0.40
0.35
0.35
Regularzation Scale ta)
Precision
Precision
0.30
0.1
2.0
0.25
5.0
10.0
0.20
0.4
30.0
0.20
0.7
100.0
0.15
0.15
0.10
10.0
30,0
100.0
Layer
Regularization Scale (a)
0.6
Regularization Scale (a)
100.0
0.7
30.0
0.4
0.300
10.00
0.3
0.275
Precision
Precision
2.0
0.1
0.250
-1.0
0.0
0.225
0.3
0.200
0.175
0.2
0.150
00
30,0
100
10
12
18
Layer
Regularzation Scale (o)
g
0.7
Regulariration Scale
0.0
1.0
0.45
0.40
Precsion
0.5
0.3
10.0
30.0
0.35
0.25
0.3
0.20
0.15
10.0
100.0
Layer
Regularization Scale (a)
Figure A17: Class selectivity regularization has similar effects when measured using a different class selectivity metric. (a) Mean precision (y-axis) function of layer (x-axis) for different regularization scales (a: denoted by intensity of blue) when regularizing against class selectivity in ResNet18. Precision an alternative class selectivity metric (see Appendix(A.I3
(b) Similar to (a), mean is computed across all units network instead per layer. (c) and (d) are identical (a) and (b), respectively, but when regularizing to promote class selectivity. (e-h) are identical to (a-d), respectively, hut for ResNet20. Error bars denote bootstrapped 95% confidence intervals.
28

---

Published as a conference paper at ICLR 2021
a) 0.0225
b) 0.175
Requirization
Regulbrization
0.0200
Scale (0)
0.0
5.0
0.1
0.150
Mutual Information
0.0175
2.0
1.0
0.3
0.0150
-0.
044
07
0.100
0.0125
-0.3
-03
2.0
1.0
Mutual
0.075
-0.2
2.0
0.0100
3.0
0.1
5.0
0.050
0.0075
0.0050
0.000
10
122 14 36
Layer
Layer
Figure A18: Effects of class selectivity regularization mutual information. (a) Mean mutual information (y-axis) as function layer (x-axis) for different regularization scales (a: negative values denoted by intensity of blue, positive values denoted by intensity of red) when regularizing to control class selectivity in ResNet18. (b) Identical to (a), but for ResNet20. Error bars denote bootstrapped 95% confidence intervals.
A.14 REGULARIZING TO CONTROL CLASS DISCRIMINABILITY
We further confirmed that our results are not unique to our chosen class selectivity metric by examining related, but distinct metric: class discriminability. d' is measure of the discriminability between two distributions (Stanislaw and Todorov 1999, Macmillan and Creelman 2004), defined as
d'
Pmoz p--ax
(4)
Vi 02 man) +
The numerator of d' is identical the numerator of the selectivity index (Equation 1 the sum of Mmaz, the largest class-conditional mean activation, and jm-max the mean response to the remaining (i.e. non-Hmmaz) classes but the denominator differs. o² denotes the variance across activations is a small value to prevent division by zero (we used 10-7).
Because d' is differentiable with respect the model parameters, we can control the amount of class discriminability learned by individual units using the same approach as when regularizing to control class selectivity. Instead of calculating the class selectivity index for each unit, we calculated d'. This leads to the following loss function:
loss
Σ Me- log((e) aftaa
(5)
This loss is identical to Equation
2 except the selectivity term, psr, is replaced with part
Pe'' 1 П Σ d'
As when computing jusz (Equation 3. is convolutional layer, T. is the number of layers, u is unit (i.e. feature map), U is the number of units a given layer, and u is unit. The procedure for regularizing d' is otherwise identical regularizing class selectivity (Approach 3.3
Regularizing to control d' has the intended effect in both ResNet18 trained on Tiny ImageNet (Figure Al9a) and ResNet20 trained on CIFARIO (Figure ATc)); the amount of class discriminability learned by individual units varies as a function of the sign and scale of regularization (a). d' also correlates strongly with class selectivity across units (Spearman's p 0.83 for ResNet18 trained on Tiny ImageNet; p = 0.90 for ResNet20 trained on CIFARIO; p < 10-5 for both). We observe very similar effects test accuracy as when manipulating class selectivity: increasing d' has rapid negative effects on test accuracy, while decreasing has more modest effects. Though we do not observe an improvement in test accuracy we did when regularizing to decrease class selectivity in ResNet18 trained on Tiny ImageNet, the asymmetry in effect between increasing vs. decreasing d' is nevertheless consistent, indicating that neither class selectivity nor discriminability are sufficient nor strictly necessary for CNN performance.
Traditional signal detection terminology describes d' as being computed for a "signal" distribution (Heignal, )signat) and "noise" distribution (fnoise, Anoise). In our setting, max is the signal distribution and mar is the noise distribution.
29

---

Published as a conference paper at ICLR 2021
b)
3.5
Requariation Scale
3.0
2.0
Test Accuracy
2.5
-10
6.5
0.7
0.4
b
2.0
-044
30
1.5
0.1
5.0
1.0
0.5
Layer
Regularization Scale
d)
Requlattzatoon
-00
0.1
-20
0.2
Test Accuracy
80
3
0.3
100
70
65
Layer 14
16
18
.5.0
200
10
8o
Regularization Scale (a)
Figure A19: Regularizing control class discriminability (d' in individual units. (a) Mean class selectivity (y-axis) as function of layer (x-axis) for different values of (intensity of blue) when class selcctivity regularization restricted to the first three network layers in ResNet18 trained on Tiny ImageNet. (b) Mean class selectivity in the first three layers (y-axis) as function of a (x-axis) in ResNet1 8 trained on Tiny ImageNct. (c) and (d) are identical to (a) and (b), respectively, but for ResNet20 trained on CIFARIO. Error bars 95% confidence intervals.
A.15 RESTRICTING CLASS SELECTIVITY REGULARIZATION THREE LAYERS
FIRST THREE FINAL
To investigate the layer-specificity of the effects class selectivity regularization, we also examined the effects of restricting class selectivity regularization to the first three or last three layers of the networks. Interestingly, we found that much of the effect of regularizing for or against selectivity on test accuracy replicated even when the regularization For example, reducing class selectivity in the first three layers either improves test accuracy- in has little-to-no effect test accuracy- in ResNet20 trained on CIFAR10 (Figures (A20 and A2T]. Likewise, regularizing to increase class selectivity the first three layers had immediate negative impact on test accuracy in both models (Figures A22 and
restricted to the first final three layers.
ResNet18 trained Tiny ImageNet-
A23 Regularizing against class selectivity in the final three layers (Figures A24 and A25) caused
modest increase in test accuracy over narrow range of in ResNet18 trained Tiny ImageNet: less
than half a percent gain at most (at a = -022), and no longer present by a -0.4 (Figure A255). In
ResNet20, regularizing against class selectivity in the final three layers actually causes decrease in test accuracy (Figures A21c Af2c and (21d). Given that the logit (output) layer of CNNs trained for image classification are by definition class-selective, we thought that regularizing to increase class selectivity in the final three layers could improve performance, but surprisingly it causes immediate drop in test accuracy in both models (Figures A226 and A27). Our observation that regularizing to decrease class selectivity provides greater benefits (in the case of ResNet18) less impairment (in the case of ResNet20) in the first three layers compared to the final three layers leads to the conclusion that class selectivity is less necessary (or more detrimental) in early layers compared to late layers.
30

---

Published as a conference paper at ICLR 2021
h)
Class Selectiitty Index
Cless Selectivity Index
(first3 Lapers),
0.90
Regularization
0.15
-04
0.2
-5.0
0.3
0.10
-2.0
0.1
-1.0
-0.1
0.00
0.0
12
Regularication Scale
Layer
d)
0.14
Reqularization
Cless Selectivity Inder
0.4
0.10
Selectivity
2200
-0.2
0.00
-0.7
0.0
5.004
0.1
0.2
40
Layer
Regularizatonn Scale (a)
Figure A20: Regularizing to decrease class selectivity in the first three network layers. (a) Mean class selectivity (y-axis) as function of layer (x- axia for different values of a (intensity of blue) when class selectivity regularization restricted to the first three network layers in ResNet18 trained Tiny ImageNet. (b) Mean class selectivity in first three layers (y-axis) as function a (x-axis) ResNet18 trained on Tiny ImageNet. (c) and (d) are identical to (a) and (b), respectively, but for ResNet20 trained on CIFAR1O. Error bars 95% confidence intervals.
a)
b)
52.0
Regulatization
Scale
-0.4
51.8
Py
51.6
-01
51.4
-0.7
0.0
51.2
51.2
$1.0
50.8
50.6
0.20 Meen Class Selectivity (First Layers)
0.05
Regularization Scale (o)
d)
90.0
90.7
Accuracy
Test Accuracy
00.6
90.5
Regularization
(e)
90.4
2.0
0.2
90.3
50.3
1.0
0.1
-0.7
0.0
0.000 0.025 0.050 0.075 0.100 0.125
Regu/arizztion Scale (d)
Mean Class Selectiiity (First Layers)
Figure A21: Effects on test accuracy when regularizing to decrease class selectivity in the first three network layers. (a) Test accuracy (y-axis) function of (x-axis) when class selectivity regularization is restricted to the first three network layers in ResNet18 trained on Tiny ImageNet. (b) Test accuracy (y-axis) function of mean class selectivity in the three layers (x-axis) ResNet18 trained Tiny ImageNet. (c) (d) are identical to (a) and (b), respectively, but for ResNet20 trained on CIFARIO. Error bars 95% contidence intervals.
31

---

Published as a conference paper at ICLR 2021
a)
b)
Regularization
class Selectvity Index
0.9
Scale (a)
Class Selectivity Index
0.8
0.0
0.7
(First Layers)
0.7
0.2
2.0
0.6
0.6
0.5
0.5
0.4
0.4
0.3
0.3
0.2
100
40
2
12
14
16
Layer
Regularization Scale to)
0.5
d)
Regularization (o]
Cless Selectivity Indox
0.4
0.0
0.7
[fiist Layers
0.335
Class Selectvitty
0.3
0.2
2.0
20030
0.3
5.0
0.25
0.2
0.1
0.15
0.0
10
12
144
166
18
Layer
Regularization Scale
Figure A22: Regularizing to increase class selectivity in the first three network layers. (a) Mean class selectivity (y-axis) as a function layer (x-axis) for different values of (intensity of red) when class selectivity regularization is restricted to the first three network layers in ResNet18 trained on Tiny ImageNet. (b) Mean class selectivity in the first three layers (y-axis) as a function of a (x-axis) in ResNet18 18 trained on Tiny ImageNet. (c) and (d) identical to (a) and (b), respectively, but for ResNet20 trained on CIFARIO. Error bars 95% confidence intervals.
a) 51.0
b)
50.5
Text Accuracy
50.0
50.0
49.5
Scale (cl
Regdatization
0.4
48.5
0.3
5.0
0.3
0.5
0.7
0.9
Regularzation Scale (o)
Mean Class Selectivity (First Layers)
01
d)
91
90
89
Tex: Accuracy
Regularization Scale
88
88
0.0
0.1
1.0
87
0.2
0.1
0.2
0.3
0.4
Regularization
Mean Class Selectivity (First Layers)
Figure A23: Effects test accuracy when regularizing to increase class selectivity in the first three network layers. (a) Test accuracy (y-axis) function of (x-axis) when class selectivity regularization restricted to first three network layers in ResNet18 trained on Tiny ImageNet. function of mean class selectivity in the first three layers (x-axis) in ResNet18 trained on Tiny ImageNet. (c) and (d) are identical to (a) and (b), respectively, but for ResNet20 trained on CIFARIO 10. Error bars = 95% confidence intervals.
(b) Test accuracy (y-axis) as a
32

---

Published as a conference paper at ICLR 2021
a)
b)
0.4
Claxs Selectivity index
Cless Selectivity Index
(Final 3 Layers)
0.3
(Final 3 Layers)
0.3
0.2
Regulariation
Scale ld]
004
-5.0
-033
0.1
-0.2
0.1
-1.0
-0.3
0.0
0.0
10
144
16
Layer
Regularization Scale to)
d)
Regularization
Cless Selectiilt Index
-5.0
-0.3
(Fine) 3 Layens)
0.3
Selectivity
0.3
-2.0
0.2
-0.7
0.0
0.2
0.2
0.1
0.0
100
Layer
Regularization
Figure A24: Regularizing to decrease class selectivity in the last three network layers. (a) Mean class selectivity (y-axis) as a function of layer (x-axis) for different values of a (intensity of blue) when class selectivity regularization is restricted to the last three network layers in ResNet18 trained on Tiny ImageNet. (b) Mean class selectivity in the last three layers (y-axis) as function of (x-axis) in ResNet18 8 trained on Tiny ImageNet. and (d) are identical (a) and (b), respectively, but for ResNet20 trained on CIFAR1O. Error bars 95% confidence intervals.
50
b)
C) 50
40
Accurscy
30
50
30
Reqularization
-0.4
20
20
2.0
0.2
-1.0
-0.1
10
0.1
0.2
0.3
0.4
Regulariaation
Regularbattonn Scale
Mean Class Selectivity (Final Layers)
90.8
90.8
Tost Actursty
90.6
Test Acaracy
90.6
90.4
Regular zation
90.2
Scale (c)
0.4
-5.0
-0.3
90.0
90.0
-0.7
83
0.0
0.1
0.2
0.3
0.4
Regularization
Moan Class Selectivity (Rinal Layers)
Figure A25: Effeets test accuracy when regularizing to decrease class selectivity in the last three network layers. (a) Test accuracy (y-axis) a function (x-axis) when class selectivity regularization restricted to the last three network layers in ResNet18 trained on Tiny ImageNet. (b) Similar to (a), but for a subset of a values. (c) Test accuracy (y-axis) as function of mean class selectivity in the last three layers (x-axis) in ResNet18 trained Tiny ImageNet. (d) and (e) are identical (a) and (c), respectively, but for ResNet20 trained CIFARI0. Error bars 95% confidence intervals.
33

---

Published as a conference paper at ICLR 2021
1.0
Regularization
Scale (a)
Class Selectivity Index
0.0
Class Selectivity Index
(Final Lay 17
0.1
1.0
(Rina) Layers
0.3
5.0
0.5
0.4
0.3
0.2
0
10
14
16
10.0
Layer
Regularization Scale le)
d)
Regularization
0.0
Scale (ol
0.4
Chocs Selectivity Indox
0.9
0.1
1.0
680.8
Class Selectiitty
0.6
0.2
2.0
3Final
0.4
0.0
0.2
04
0.0
0
2
10
12 144 16
18
Regularizztion Scale
Figure A26: Regularizing increase class selectivity in the last three network layers. (a) Mean class selectivity (y-axis) a function layer (x-axis) for different values of (intensity of red) when class selectivity regularization is restricted to the last three network layers in ResNet18 trained on Tiny ImageNet. (b) Mean class selectivity in the last three layers (y-axis) as function of a (x-axis) in ResNet18 trained on Tiny ImageNet. (c) and (d) are identical to (a) and (b), respectivoly, but for ResNet20 trained on CIFARIO. Error bars 95% confidence intervals.
50
50
51
40
35
T
Accuracy
50
Accuracy
33
Regularzation
0.4
30
0.7
0.1
1.0
25
2.0
20
5.0
15
46
10
Regularization
9
-0.0
Regularization
0.1
0.2
0.3
0.4
0.7
0.4
Mean Cless Selectivity (Final Layers)
0.6
0.8
1.0
d)
91.5
90
80
Regularization
Text Accurryy
90.5
Test Accuracy
Scale (o)
0.4
80.0
0.2
2.0
89.5
50
40
40
88.5
30
30
88.0
20
0.0
0.1
0.2
0.3
0.4
0.7
Regularbattion Scale
0.8
Regularization Scale (o)
Mean Closs Selectivity (Final 3 Layers)
accuracy when regularizing to increase class selectivity in the last three network layers. (a) Test accuracy (y-axis) as a function a (x-axis) when class selectivity regularization is restricted to the last three network layers in ResNet18 trained on Tiny ImageNet. (b) Similar to (a), but for subset of a values. (c) Test accuracy (y-axis) as function of mean class selectivity in the last three layers (x-axis) in ResNet18 trained Tiny ImageNet. (d-f) are identical to (a-c), respectively, but for ResNet20 trained on CIFAR10. Error bars 95% confidence intervals.
Figure A27: Effects
34

---

Published as a conference paper at ICLR 2021
A.16 AN INABILITY CHANGE PREFERRED CLASSES DURING TRAINING DOES NOT EXPLAIN SELECTIVITY-INDUGEE PERFORMANCE DEFICITS
One potential limitation of selectivity regularizer is that regularizing to increase class selectivity could discourage individual neurons from changing their preferred class during training: units would be locked in the class they preferred initialization. If changing preferred classes is necessary to improve test accuracy, then regularizing to increase class selectivity could impose a constraint on performance. We controlled for this possibility in two ways. First, we analyzed the statistics of preferred class changes that occur as function of the class selectivity regularization scale. We examined the proportion of units that change classes at least once during training when a 2 0. The mean proportion is one, for all examined regularization scales and models (Figure A28), indicating that regularizing to increase selectivity does not lock units into their initial preferred class. We also examined whether regularizing to increase selectivity affects the number of times unit changes its preferred class during training. In ResNet18 trained on Tiny ImageNet, we found that the mean number of preferred class changes decreases we note that the number of preferred class changes is roughly equal for a = {0, O.T, 0.2), even though test accuracy for a {0.1, 0.2) is significantly lower than for a O, indicating that preferred class changes cannot fully account for decreased test accuracy. Furthermore. in ResNet20 trained on CIFARIO there is no clear relationship between a and the number of preferred class changes (Figure A29bp. These results indicate that the test accuracy impairment caused by regularizing to increase class selectivity cannot be fully explained by the regularizer preventing units from changing their preferred class during training.
function of a for 2 0.3 (Figure A2aa.. However,
We also controlled for the possibility of preferred class "lock-in" by warming up selectivity regu- larization during training, which would allow units' preferred classes to change during the early period of training, when learning-induced changes are most significant (Achille et al. 2018 Gur-Ari al. 2018 Frankle et al. 2020). We implemented selectivity regularization warmup in analogous manner to learning rate warmup by scaling a (see Equation 2 linearly over the interval [0,1] across the epochs first five epochs of training. We observed qualitatively similar results as when we did not warm selectivity regularization (Figure [A3 Figure A3T). Regularizing to reduce class selectivity either has minimal negative effects (ResNet20 trained CIFARIO) or improves test accuracy 18esNet18 trained on Tiny ImageNet), while regularizing to increase class selectivity impairs test accuracy in all examined models (Figure AA3I), Interestingly, the severity of the test accuracy impairment is slightly reduced when using regularization warmup. This indicates that some of the test accuracy deficit from regularizing to increase class selectivity is attributable to the regularizer forcing the network into suboptimal solutions early in training. Nevertheless, the test accuracy deficit imparted by increased class selectivity remains even when the class selectivity regularizer is warmed up, further supporting the claim that that class selectivity is not sufficient for network performance.
Proportion of Units 1 Change
Proportion of Units w/ 1 Change
1.04
1.04
1.02
1.00
1.00
0.98
0.98
0.96
0.0
1.0
2.0
20
20
50
Regularization Scale (a)
Regularization Scale (a)
Figure A28: Units change preferred classes during training even when regularizing to increase class selectivity. (a) Mean proportion of units that change preferred classes least once during training (y-axis) as function of regularization scale (x-axis: hue) for ResNet18 trained on Tiny ImageNet. (b) Identical (a), but for ResNer20 trained on CIFARIO. Error bars = 95% confidence intervals.
35

---

Published as a conference paper at ICLR 2021
b)
16
Preferred Class Changes
Preferred Changes
30
25
8
5
2
0
0.0
2
Regularization (a)
Regularization Scale (a)
Figure A29: Regularizing increase class selectivity has inconsistent effects the number of times units change their preferred class during training. (a) Mean number preferred class changes across units (y-axis) as function of regularization scale (x-axis; hue) for ResNet18 trained on Tiny ImageNet. (b) Identical to (a), but for ResNer20 trained on CIFARIO. Error bars 95% confidence intervals.
a)
1.0
b)
Regulariation
Class Selectivity Index
0.8
Scale
Class Selectiviy Index
0.8
-5.0
0.1
0.7
0.4
0.4
-03
0.4
-02
2.0
0.2
0.2
0
2
8
10
12
14
16
Regularization Scale (a)
Layer
d)
0.6
0.8
Regulenzation Scale (a)
Class Selectiity Index
0.6
0.4
1.0
0.3
Closs Selectivity
4.77
0.4
0.4
0.4
0.7
0.2
0.2
5.0
Layer
Regularization Scale la)
Figure A30: Effeets of warming up class selectivity regularization. (a) Mean class selectivity (y-axis) as function of layer (x-axis) for different values of (hue) when class selectivity regularization is warmed up the first 5 epochs in ResNet18 trained Tiny ImageNet. (b) Mean class selectivity function a (X-axis) when warming up selectivity regularization in ResNet18 trained on Tiny ImageNet. (c) and (d) are identical to (a) and (b), respectively, but for ResNet20 trained on CIFARIO. Error bars 95% confidence intervals.
all layers (y-axis)
36

---

Published as a conference paper at ICLR 2021
54
52
Accuracy
Test Accuracy
50
Regulerizat on
Scale (o)
0.0
48
48
2.0
0.2
-10
0.3
46
-0.4
0.3
1.0
-0.1
42
8
0.4
0.6
Regularization Scale
Mean Class Selectivity
90
80
80
Regularination Scale (n)
Test Accuracy
-3.0
0.1
70
70
-200
0.2
-07
0.4
0.7
0.3
1.0
40
0.0
0.1
0.2
0.4 Mean Class Selectivity
0.3
0.5
0.6
0.7
Regularization Scale (a]
Figure A31: Effects of class selectivity regularization warmup on test accuracy. (a) Test accuracy (y-axis) as a function of class selectivity regularization scale (a: x-axis and hue) when class selectivity regularization is warmed up over the first 5 epochs in ResNet18 trained on Tiny ImageNet. (b) Test accuracy (y-axis) as function of mean class selectivity (X-axis) when warming up selectivity regularization in ResNet18 trained on Tiny ImageNet. Each data point denotes single network. and are identical (a) and (b), respectively, but for ResNet20 trained on CIFAR 10. Error bars 95% confidence intervals.
A.17 DIRECTLY COMPARING THE EFFECTS OF REGUIARIZING TO INCREASE VS. DECREASE CLASS SELECTIVITY
Scale toi
0.0
-10.00
0.1
80
-500
022
-1..0
0.3
70
550
DA
-2.
0.7
60
-10
1.0
-0.0
2.0
30
-044
5.0
.40
0.3
0.2
10.0
30,0
30
0.1
100.0
2
10
Mean Class Selectivlty
Figure A32: Increasing class selectivity has rapid and deleterious effects on test accuracy compared to reducing class selectivity in ResNet20 trained on CIFAR1O. Test accuracy (y-axis) as function of mean class selectivity (x-axis). Each data point denotes a network.
37

---

Published as a conference paper at ICLR 2021
b)
50
Accuracy
Accuracy
70
20
signla]
0 (penalize selectivity)
30
(promote selectivity)
sign(a)
(penalize selectivity!
30
(promate selectivity)
5.0
16.0
30.0
100.0
10
5.0
10.0
30.0
100.0
|Regularization Scale (d))
|Regularization Scale (ai)
Figure A33: Directly comparing promoting vs penalizing class selectivity, (a) Test accuracy (y-axis) as
function of regularization scale (a; x-axis) when promoting (a > 0, red) or penalizing (a < 0, blue) class selectivity in ResNet18 8 trained Tiny Imagenet. **p < 2 x 10 difference hetween penalizing vs. promoting selectivity, Wilcoxon rank-sum test, Ronferromi-corrrcted.. (b)
(a) hut for ResNet20 trained on CIFAR10. 10 9 difference. Wilcoxon rank-sum test, Bonferroni-correce Error bars denote bootstrapped 95% confidence intervals.
"p 0.05, *sp
A.I8 EFFECTS OF CLASS SELECTIVITY REGULARIZATION IN RESNET5O TRAINED ON TINY IMAGENET
We also examined the effect regularizing to control class selectivity in ResNet50 trained on Tiny ImageNet. Our results were qualitatively similar to those observed in ResNet18: class selectivity in trained networks correlates with a (Figure/A34a) A34a). and regularizing decrease class selectivity can improve test accuracy, while regularizing to increase class selectivity impairs test accuracy (Figure A34b
b)
Class Seleciiiit Index
57
556
0.4
54
0.3
0.2
1.00
10
10
1 20
Regularization (a]
Regularization Scale (a)
Figure A34: Effeetts of class selectivity regularization in ResNet50 trained on Tiny ImageNet (a) Mean class selectivity (y-axis) penalizing (a blue) class selectivity. (b) Test accuracy (y-axis) function of (x-axis). Note that the increased size the 95% confidence intervals at least partiall-attributable to training only 5 replicates per a compared to the 20 replicates per a used for other models.
function of regularization scale (a; x-axis) when promoting (a 0, red) or
38


Reviews:
[{"review_id": "NfGMzcxMtYJ", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper asks the interesting question of whether you need individual neuron (or even population level) class selectivity at intermediate stages in order to have good classification performance. The authors introduce a regularization term to the loss that controls the amount of selectivity in the units of the network. They find that the selectivity of the units in standard networks can be reduced while maintaining classification performance. \n\nOverall, this paper presents a compelling story. Although it could be strengthened by increasing the clarity in some of the presented results, the experimental results seem detailed and rigorous leading to a recommended accept. \n\nPositives: \n* The question of whether individual units or populations of units are the right domain to study is currently topical for both the machine learning community and the neuroscience community. \n* The question of necessity of selectivity is similarly topical for both ML and neuroscience. \n* The paper presents a nice step back from much of the single unit interpretability literature, highlighting that it is unclear what some of these results mean if the selective units are not actually necessary for the task performance. \n* The experimental results (other than the optimization concerns below) seem sound, particularly including the multiple replicates + error bars to verify the experimental findings. \n\nConcerns and Suggestions: \n* The stability of optimization when including the regularization scale is somewhat suspicious, specifically at high values of $|\\alpha|$ where there is not a direct relationship between the test accuracy and the observed class selectivity (ie figure 4b). If optimization is working correctly, wouldn’t one suspect that at high $\\alpha$ the mean class selectivity should be close to 1, while at low $\\alpha$ the mean class selectivity should be close to 0? Instead the mean accuracy seems to behave erratically, suggesting a potential instability in the optimization. Although these values are perhaps uninteresting, it raises the concern that there could be an interaction happening at intermediate values. \n* Related to the above, the paper may be strengthened if the main text results were from the leaky ReLU models that are presented in the appendix, as it seems like the story is the same but the confound of individual unit selectivity is removed. \n* The authors note that, by definition, the final layer must be class selective in a classification task (which means that all networks studied have class selective units). It thus seems like the question that the authors are addressing is whether selectivity in *intermediate* units is beneficial for selectivity in downstream units (rather than investigating whether selectivity is generally helpful). This should be clarified in the introduction. \n* The investigation using CCA to test whether the selectivity is simply rotated off axis is quite compelling, however providing some intuition as to how a network could use non-selective features in order to perform a task necessitating selectivity downstream would strengthen the story. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "6RiDy2YPDc", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "[Update after authors' reply]\n\nIn light of the authors' reply, I have updated my review to favor acceptance. I appreciate the additional experiments. It will be up to the readers to determine how to interpret those additional results.\n\n---\nThe authors attempt to ascertain whether single-neuron class selectivity is beneficial or harmful to overall network performance. They do this by adding a regularization term to the training loss, which measures how selective network units are. They observe that discouraging selectivity can have a small benefit, and generally doesn't harm performance even at very high values. Conversely, encouraging selectivity dramatically decreases performance. They perform additional analyses, e.g. checking that selectivity is not simply masked by linearly dividing it across units.\n\nThe paper is interesting, well-written, and the experiments are thorough  (minor quibble: I would have liked a bit more emphasis on layer-by-layer analyses, which only seem to occur in the last few figures of the appendix?). \n\nMy main problem is with the regularizer (Eq. 1). in my (possibly flawed) understanding, it seems to have additional effects, that are not discussed in the paper (unless I have missed it) and complicate the interpretation of results.\n\nIIUC, ascending the regularizer value does not simply increase \"selectivity\" in general, but *current* selectivity. That is, it encourages the neuron to increase its preference for whatever it's preferring *right now*, and discourages / prevents it from ever switching preferences, even if that might be beneficial to overall network training.\n\nThus, it is possible that the dramatic drop in performance is not due simply to increased selectivity in general, but to locking the preferences of individual units early on, when they are essentially random and likely sub-optimal.\n\nIt is not obvious how to control for such a possibility. At the very least, one might check how often units switch preference over training, with and without the regularizer. Perhaps only turning on the regularizer when switches become rare (if they ever do!) might help?\n\nIt seems that this problem does not occur in the reverse direction: decreasing selectivity does not seem to lock up a specific preference, but rather to reduce preference in general, which is the expected interpretation (IIUC). As such, the results for negative alpha seem unaffected.\n\nAgain, I may have missed something in the paper. Otherwise, the problem needs to be at the very least discussed, and the authors should find some way to estimate it and possibly control for it, for the paper to be accepted. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "01prQ1ivwci", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper examines the impact of forcing units in a CNN to be more or less “class-selective” – i.e. respond preferentially to one image class compared to another.  The approach taken is to include a regularizer in the loss that directly penalizes or encourages class selectivity in individual units. They report that penalizing class selectivity at intermediate layers has little-to-no effect on classification performance, and in some cases mildly improves performance. They authors conclude that class selectivity is not an essential component of successful performance in CNNs, and that methods which use class selectivity to interpret CNNs should be approached with caution. \n\n--\n\nPros:\n\nThe authors address a basic/important question – whether class selectivity in individual units of a deep network is related to task performance on a classification task.\n\nThe writing is overall clear. I found the motivation, experiments, and results easy to understand. \n\nThe analyses are fairly extensive. \n\nOverall, I found the results compelling.\n\n--\n\nLimitations and questions:\n\nThe authors results show that class selectivity, as they define it, is not critical to good performance, in the sense that networks can perform well without showing class selectivity. However, this fact does not demonstrate that class selectivity is unimportant to a network trained in the normal way without regularization. It is of course possible that networks trained with different losses could learn different strategies for solving the same task. -- Update: In my opinion, this fact is not highlighted prominently enough, including in the revised version. -- \n\nI didn’t find the off-axis selectivity analyses very convincing. Simply using CCA to show that the regularization has an effect on the network subspace doesn’t strike me as very convincing, since it’s unclear how the subspace differs. It seems like the key question is whether there are linear projections that exhibit class selectivity, which could be addressed in a more straightforward fashion using linear classifiers, trained and tested in the usual way. \n\nRelated to the above point, I would have liked to see a measure of class selectivity that is more directly related to discrimination – i.e. the ability to discriminate exemplars of one class from another. A natural choice in my opinion would be a measure like d-prime that measures the within vs. between class distance. How would the results look if they used linear discriminant analysis to project the activations of each layer onto a low-dimensional subspace that maximized class discriminability and then measured the overall within vs. between class separation in this space? -- Update: The authors have repeated their analysis with d-prime. They do not observe an improvement when regularizing against d-prime, but report an asymmetry, where promoting class selectivity leads to a greater decrement than down-weighting class selectivity --\n\nOne of the ways that class selectivity has been used is to examine selectivity for objects in a network trained to recognize scenes. This paper does not examine this type of selectivity for sub-classes. Does scene recognition require first classifying objects? \n\n---\n\nMy rating remains unchanged. The authors have addressed some but not all of the issues raised, as noted above.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper has received three positive reviews. In general, the reviewers have commented on the importance of the question related to how much selectivity is needed from units of a neural network for good classification -- from both the neuroscience and ML perspectives. The reviewers also commented on the thoroughness of the experiments and the general readability of the paper. This paper should be accepted if possible.
Decision: Accept (Poster)