Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper asks the interesting question of whether you need individual neuron (or even population level) class selectivity at intermediate stages in order to have good classification performance. The authors introduce a regularization term to the loss that controls the amount of selectivity in the units of the network. They find that the selectivity of the units in standard networks can be reduced while maintaining classification performance. 

Overall, this paper presents a compelling story. Although it could be strengthened by increasing the clarity in some of the presented results, the experimental results seem detailed and rigorous leading to a recommended accept. 

Positives: 
* The question of whether individual units or populations of units are the right domain to study is currently topical for both the machine learning community and the neuroscience community. 
* The question of necessity of selectivity is similarly topical for both ML and neuroscience. 
* The paper presents a nice step back from much of the single unit interpretability literature, highlighting that it is unclear what some of these results mean if the selective units are not actually necessary for the task performance. 
* The experimental results (other than the optimization concerns below) seem sound, particularly including the multiple replicates + error bars to verify the experimental findings. 

Concerns and Suggestions: 
* The stability of optimization when including the regularization scale is somewhat suspicious, specifically at high values of $|\alpha|$ where there is not a direct relationship between the test accuracy and the observed class selectivity (ie figure 4b). If optimization is working correctly, wouldn’t one suspect that at high $\alpha$ the mean class selectivity should be close to 1, while at low $\alpha$ the mean class selectivity should be close to 0? Instead the mean accuracy seems to behave erratically, suggesting a potential instability in the optimization. Although these values are perhaps uninteresting, it raises the concern that there could be an interaction happening at intermediate values. 
* Related to the above, the paper may be strengthened if the main text results were from the leaky ReLU models that are presented in the appendix, as it seems like the story is the same but the confound of individual unit selectivity is removed. 
* The authors note that, by definition, the final layer must be class selective in a classification task (which means that all networks studied have class selective units). It thus seems like the question that the authors are addressing is whether selectivity in *intermediate* units is beneficial for selectivity in downstream units (rather than investigating whether selectivity is generally helpful). This should be clarified in the introduction. 
* The investigation using CCA to test whether the selectivity is simply rotated off axis is quite compelling, however providing some intuition as to how a network could use non-selective features in order to perform a task necessitating selectivity downstream would strengthen the story. 

review_title: an interesting story about whether selectivity is necessary at intermediate representations

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper examines the impact of forcing units in a CNN to be more or less “class-selective” – i.e. respond preferentially to one image class compared to another.  The approach taken is to include a regularizer in the loss that directly penalizes or encourages class selectivity in individual units. They report that penalizing class selectivity at intermediate layers has little-to-no effect on classification performance, and in some cases mildly improves performance. They authors conclude that class selectivity is not an essential component of successful performance in CNNs, and that methods which use class selectivity to interpret CNNs should be approached with caution. 

--

Pros:

The authors address a basic/important question – whether class selectivity in individual units of a deep network is related to task performance on a classification task.

The writing is overall clear. I found the motivation, experiments, and results easy to understand. 

The analyses are fairly extensive. 

Overall, I found the results compelling.

--

Limitations and questions:

The authors results show that class selectivity, as they define it, is not critical to good performance, in the sense that networks can perform well without showing class selectivity. However, this fact does not demonstrate that class selectivity is unimportant to a network trained in the normal way without regularization. It is of course possible that networks trained with different losses could learn different strategies for solving the same task. -- Update: In my opinion, this fact is not highlighted prominently enough, including in the revised version. -- 

I didn’t find the off-axis selectivity analyses very convincing. Simply using CCA to show that the regularization has an effect on the network subspace doesn’t strike me as very convincing, since it’s unclear how the subspace differs. It seems like the key question is whether there are linear projections that exhibit class selectivity, which could be addressed in a more straightforward fashion using linear classifiers, trained and tested in the usual way. 

Related to the above point, I would have liked to see a measure of class selectivity that is more directly related to discrimination – i.e. the ability to discriminate exemplars of one class from another. A natural choice in my opinion would be a measure like d-prime that measures the within vs. between class distance. How would the results look if they used linear discriminant analysis to project the activations of each layer onto a low-dimensional subspace that maximized class discriminability and then measured the overall within vs. between class separation in this space? -- Update: The authors have repeated their analysis with d-prime. They do not observe an improvement when regularizing against d-prime, but report an asymmetry, where promoting class selectivity leads to a greater decrement than down-weighting class selectivity --

One of the ways that class selectivity has been used is to examine selectivity for objects in a network trained to recognize scenes. This paper does not examine this type of selectivity for sub-classes. Does scene recognition require first classifying objects? 

---

My rating remains unchanged. The authors have addressed some but not all of the issues raised, as noted above.
review_title: Class selectivity in individual units is not critical for CNN classification performance

final_score: rating: 6: Marginally above acceptance threshold
review_content: [Update after authors' reply]

In light of the authors' reply, I have updated my review to favor acceptance. I appreciate the additional experiments. It will be up to the readers to determine how to interpret those additional results.

---
The authors attempt to ascertain whether single-neuron class selectivity is beneficial or harmful to overall network performance. They do this by adding a regularization term to the training loss, which measures how selective network units are. They observe that discouraging selectivity can have a small benefit, and generally doesn't harm performance even at very high values. Conversely, encouraging selectivity dramatically decreases performance. They perform additional analyses, e.g. checking that selectivity is not simply masked by linearly dividing it across units.

The paper is interesting, well-written, and the experiments are thorough  (minor quibble: I would have liked a bit more emphasis on layer-by-layer analyses, which only seem to occur in the last few figures of the appendix?). 

My main problem is with the regularizer (Eq. 1). in my (possibly flawed) understanding, it seems to have additional effects, that are not discussed in the paper (unless I have missed it) and complicate the interpretation of results.

IIUC, ascending the regularizer value does not simply increase "selectivity" in general, but *current* selectivity. That is, it encourages the neuron to increase its preference for whatever it's preferring *right now*, and discourages / prevents it from ever switching preferences, even if that might be beneficial to overall network training.

Thus, it is possible that the dramatic drop in performance is not due simply to increased selectivity in general, but to locking the preferences of individual units early on, when they are essentially random and likely sub-optimal.

It is not obvious how to control for such a possibility. At the very least, one might check how often units switch preference over training, with and without the regularizer. Perhaps only turning on the regularizer when switches become rare (if they ever do!) might help?

It seems that this problem does not occur in the reverse direction: decreasing selectivity does not seem to lock up a specific preference, but rather to reduce preference in general, which is the expected interpretation (IIUC). As such, the results for negative alpha seem unaffected.

Again, I may have missed something in the paper. Otherwise, the problem needs to be at the very least discussed, and the authors should find some way to estimate it and possibly control for it, for the paper to be accepted. 
review_title: Interesting paper with a problematic methodology

Meta-review: This paper has received three positive reviews. In general, the reviewers have commented on the importance of the question related to how much selectivity is needed from units of a neural network for good classification -- from both the neuroscience and ML perspectives. The reviewers also commented on the thoroughness of the experiments and the general readability of the paper. This paper should be accepted if possible.
Decision: Accept (Poster)