Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This work investigates two classes of perturbation robustness, average-case perturbations which are considered to be naturally occurring in image data, and worst-case perturbations that are perturbations generated by an adversary. Neural network susceptibility to these perturbations is evaluated with respect to a class selectivity metric and dimensionality measure. 

The authors find that decreasing class selectivity will increase robustness to average-case perturbations while reducing robustness to worst-case perturbations. Simultaneously, increasing class selectivity improves robustness to worst-case perturbations while reducing average-case perturbation robustness.

In addition, the authors consider the correspondence between the dimensionality of the early layers and observe how this dimensionality corresponds to class selectivity and robustness. They find the dimensionality is inversely related to class selectivity while also positively correlated with reduced worst-case robustness. 

While the experiments are thorough in showing a relationship between class selectivity and robustness, many of the observations do not map convincingly to the conclusions. For instance, in Section 4.1, the second paragraph appears to be conjecture which is not justified by the potential observation outcomes. Having higher class selectivity does not necessarily mean fewer potent neurons is the reason for increased average-case robustness. This conclusion is not sufficiently supported by measuring robustness as a function of class selectivity. It also seems odd to hypothesize two different conclusions for opposite outcomes. The same issue of drawing a conclusion regarding the worst-case perturbation analysis in section 4.2. Why create potential conclusions for two opposite outcomes?

The relationship between the definition of an average-case perturbation and a worst-case perturbation is not well-defined. It appears to be a semantic association of what is expected to occur naturally (average-case) and what is caused by an adversary (worst-case). But without some topology or relationship between these two cases, the observations of class selectivity on each are just independent and unrelated observations. For instance, what is in between an average-case and worst-case perturbation? Where do images from datasets like ObjectNet (https://objectnet.dev/) or Natural Adversarial Examples (https://arxiv.org/abs/1907.07174) fall in the average-case to worst-case dimension?

Overall, this work would benefit from a unified view of how class selectivity relations to image perturbations. As it currently stands, this paper appears to be a collection of observations in need of a clear conclusion.

Other notes:
-It would be illustrative to have some visualization of the average-case perturbations.
-$SI_u$ in Eq 3. should also be a function of $l$

review_title: Many interesting observations regarding robustness, but unclear what conclusions or how to unify these observations

final_score: rating: 7: Good paper, accept
review_content: This paper presents a new finding that class selectivity is negatively correlated with average-case corruptions (natural visual distortions) while positively correlated with worst-case corruptions (adversarial attacks). The authors then try to explain this phenomenon from two aspects: variability of input-unit gradient and dimensionality in early layers.

Pros:
1. Class selectivity is previously used as a metric to indicate model generalization or memorization. It is good to see the authors generalize its usage to measuring model robustness to natural corruptions and adversarial attacks.
2. The finding that robustness to average-case corruptions are negatively correlated with class selectivity is no surprise to me, since it has been previously shown that low class selectivity indicates better generalization ability [1].
The intriguing finding is that robustness to adversarial images is positively correlated with class selectivity. This is contour intuitive to me at first glance, but the explanations provided in Section 4.3 and 4.4 convince me. 
3. If the conclusions in this paper holds, I think this is one step further from the well-know accuracy-robustness tradeoff [5,6]: not only do we have tradeoff between clean accuracy and adversarial accuracy [5,6], but also a (possibly inherent) tradeoff between average- and worst-case adversarial accuracy.

Cons:
The general idea to use class selectivity as an indicator for model robustness in this paper is very interesting. 
But I do think more experiments and discussions are needed to explain these phenomenons, especially the different behaviors of average-and worst-case corruptions. 
1. Adversarial training (e.g., [2]) can greatly improve adversarial robustness. If the conclusions in this paper holds, we would expect adversarially trained models to have much better class selectivity than normally trained models. I'm wondering whether the authors could kindly provide these results?
2. Similarly, there are some methods (e.g., AugMix [4]) improving model robustness to average-case corruptions. Does AugMix model has significantly lower class selectivity than normally trained models.
3. In my point of view, one possible explanation for the different behaviors between average-case and worst-case corruptions could be that normal adversarial images (generated by PGD, FGSM as in your paper) are out-of-distribution samples [3], while average-case corruptions are likely to be on-distribution. So I'm a bit curious about the behaviors of the "on–
manifold adversarial examples" defined in [3]: do they behave more like normal adversarial images (e.g., causing a larger increase in early-layer dimensionality) or more like average-case corruptions (causing a smaller increase in early-layer dimensionality )?

[1] On the importance of single directions for generalization. ICLR, 2018.

[2] Towards Deep Learning Models Resistant to Adversarial Attacks. ICLR, 2018.

[3] Disentangling Adversarial Robustness and Generalization. CVPR, 2019.

[4] AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty. ICLR, 2020.

[5] Robustness May Be at Odds with Accuracy. ICLR, 2019.

[6] Theoretically Principled Trade-off between Robustness and Accuracy. ICML, 2019.

Update:
Thanks the authors for their response. All my concerns are addressed and I decide to increase my score from 6 to 7.
review_title: Interesting Idea but More Experiments are Needed

final_score: rating: 4: Ok but not good enough - rejection
review_content: ##########################################################################

Summary:

This work empirically studies the relationship between robustness and class selectivity, a measure of neuron variability between classes. Robustness to both adversarial ("worst-case") perturbations and corruptions ("average-case") are considered. This work builds off the recent work of Leavitt and Morcos (2020) (currently in review at ICLR 2021) who claim empirical evidence that class selectivity may be harmful for generalization.  The experiments in this paper examine the robustness (in both senses) of networks explicitly regularized for class selectivity. The main empirical claims are that (1) class sensitivity is negatively correlated with robustness to corruptions (2) class sensitivity is positively correlated with robustness to adversarial perturbations.



##########################################################################

Reasons for score: 
 

Overall I vote for rejection. The authors frame the results in connection to sparsity and dimensionality. But at present the evidence for this connection appears preliminary. For example the differences in the class selectivity curves in Figure 2 appear marginal. On the other hand, Figure 1 does seem convincing for the claim reducing class selectivity improves robustness to corruptions.


##########################################################################

Pros: 
* Clear results in Figure 1a
* Important topic
* Potentially relevant results

##########################################################################

Cons: 
* Marginal results in Figure 2
* Unclear results in Figure 4
* Measurements of dimensionality limited to PCA
* Not easy to read. The paper appears hastily written.

##########################################################################

Questions during rebuttal period: 

Please argue why the differences between class-selective curves in Figure 2 are significant. 

I am quite unclear on how to interpret Figure 4. Please clarify.

The authors use the word "causal" several times in the paper, which appears to me dubious. I can find no justification for a claim of causality here, since the results are correlative. Can the authors clarify this?

The authors only examine one dimensionality estimation method: the number of PCA components required to capture 95% of the data variance. Dimensionality estimation with PCA on data with non-linear (i.e. manifold) structure is problematic. Thus I have doubts on measurements of dimensionality used here. The authors may consider adding the method of Levina and Bickel [0]

[0] Maximum Likelihood Estimation of Intrinsic Dimension - Levina and Bickel (Neurips 2004)
https://papers.nips.cc/paper/2577-maximum-likelihood-estimation-of-intrinsic-dimension

#########################################################################

Additional Feedback: 

It is the prerogative of the authors to choose the words they believe best express their message. However I lament the author's choice to use "worst-case perturbation" and "average-case perturbation" to refer to "adversarial attack"and "corruption". The literature on adversarial attacks is quite large at this point, and "adversarial" is the de facto terminology. The authors claim their terminology is more general, however I cannot see the justification given the widespread existing usage. 

#########################################################################

POST-REBUTTAL RESPONSE:

I read the author's rebuttal but have decided to not increase my score. I still have doubts over the claims in this paper. 
review_title: Results appear preliminary

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary:
This paper studies the relationship between class selectivity and robustness. In particular, it finds that higher class selectivity tends to lead to worse average-case robustness (a.k.a. performance on naturalistically perturbed inputs) but better worst-case robustness (a.k.a. performance on adversarial inputs). 

It further finds that variability of input-unit gradient across samples and units is proportional to a network’s overall class selectivity. Besides, the increase of dimensionality of activation caused by corruption is larger for worst-case perturbation and low-selectivity networks.

################################################

Reasons for score:
Overall, this paper makes a series of interesting observations on the relationship between class selectivity and robustness. I feel that the paper’s contribution is a bit limited to some interesting empirical observations on a particular task (image classification) and type of model (resnet) on two datasets (CIFAR-10 and Tiny ImageNet). Also, no attempt to make much theoretical connections and the practical impact of the paper is not very clear. 

################################################

Pros:

+mostly well-written and easy to follow

+interesting empirical findings and exploration of sensitivity on worst-case robustness and average-case robustness

+experiments support the claims 

Cons:

-evaluation looks a bit limited in terms of tasks/models/datasets.

As even the authors stated in the paper “We hesitate to generalize too broadly about our findings, as they are limited to CNNs trained on image classification tasks. It is possible that the results we report here are specific to our models and/or datasets, and also may not extend to other tasks.” The results are indeed a bit limited in terms of its scope. The evaluation is limited to only ResNets (Res18 and Res20, which are similar to each other in terms of properties) for the image classification task only. I wonder if the findings can hold for deeper models since Fig 3 (b) seems to show that for deeper layers the variability of different SI tends to be increasingly similar.


-pure empirical results and not enough theoretical justification

I understand that this is the first paper trying to connect average-case robustness and worst-case robustness with class selectivity but I still would like to see a bit more discussion on the theoretical side.

-the impact of the observations need more discussions.

I understand that this paper focuses on empirically linking average-case robustness and worst-case robustness. However, I think it needs a bit more discussion on how other researchers/practitioners can benefit from these findings. For example, how should one leverage such observations to detect adversarial / naturally perturbed error-prone input , improve robustness, or find the optimal tradeoff between different measures (e.g. natural accuracy, average-case robustness, worst-case robustness. 

################################################

Typo:
In Figure A6 caption “Figure ??”

################################################

Questions:
At the end of sec 4.2 “indicating that distributed semantic representations are more vulnerable to worst-case perturbation because they are less stable than sparse semantic representations.” What does “distributed semantic representations” mean here? What’s the difference between “distributed semantic representations” and “sparse semantic representations“ in this sentence?

################################################

Suggestions:
The paper can be improved by addressing at least one of the three cons I mentioned.


################################################

Post-Rebuttal:
Thanks the authors for their detailed responses! 
The authors responses addressed most of my concerns. In particular,1. the authors showed that their results generalize to Tiny-ImageNet and ResNet50 with additional experiments. 2. the authors added some discussions on the theoretical side. Besides, I also appreciate the addition of experiments on AugMix and PGD which imply the bidirectional causality of  class selectivity and perturbation robustness. Overall, I think this work will potentially be a good addition to the existing understanding of trade-off between worst-case robustness and average-case robustness for the community. Therefore, I decide to increase my score to 6.

review_title: Interesting empirical findings on robustness tradeoff with limited scope

Meta-review: The reviewers had raised a number of concerns which were mostly addressed during the discussion phase thanks to the additional experiments/explanations that the authors provided. However, some of the reviewers are not yet convinced about the main claims of the paper. While the paper provides a number of interesting/important/novel observations, which are obtained using an extensive set of carefully designed experiments, there are still some ambiguities about the main claims that may need further investigation/ justification. 

One general issue with the current version is lack of clarity. I recommend that the authors revise the writing of the paper and make an effort to better explain/justify the main claims, ideas, and setups in the paper. Perhaps it would help to add a section early in the paper and define/review the basic concepts/definitions.  I also encourage the authors to reason about their main claims, choice of loss function (and why it is the right choice), and their experimental setups.   As an example, the authors should further investigate the impact of their proposed regularization on semantic representations.  The authors interpret the (distribution of) class information as “semantic” content/features. Indeed, when considering data with a variety of average-case perturbations,  one could argue that semantic features like brightness or snow are actually uniform over classes, which might means that class selectivity could not appropriately capture the effect of those features. To evaluate semantic robustness,   it seems necessary to find a way to isolate specific semantic features in the data, i.e. by changing from snow to rain in corresponding images, rather than looking at the performance on the same brightness subset for different levels of their regularization parameter alpha.  Hence there is a need for further investigation on the effectiveness of class selectivity. 

Some of the reviewers have indicated that differences in the class selectivity curves in Figures 1,2 appear marginal (e.g. at most 3% difference). Hence, additional experiments (with other data sets) could be beneficial in this regard. 





Decision: Reject