Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 8: Accept
review_content: This work investigates the collection of methods that have been proposed to find units in neural networks that are selective for certain object classes.  Previous works have used different measures of selectivity (with sometimes contradictory results), and the authors investigate the degree to which these units qualify as “object detectors”.

This research area is important for understanding deep networks because claims have been made as to the relative importance (or lack thereof) of these individual units as identified by different measures vis-a-vis distributed representations -- the identification of such units would be interesting for understanding the predictions of classification networks.

The authors find that (1) different proposed measures of selectivity are not consistent and (2) units identified as selective cannot be considered object detectors due to the high false alarm / low hit rates, analyzing a large number of selectivity measures.  I would have liked to see experiments on more recent architectures (the focus of the paper is on a dated architecture (AlexNet)); there is analysis on units in GoogLeNet and VGG-16 but it would also be interesting to see results for more modern architectures (e.g. DenseNet and ResNet).

Overall, I think that the authors have presented a strong meta-analysis and compelling argument for further study in rigorously identifying the presence (or lack thereof) of selective units in neural networks and the degree to which they may be considered "object detectors."
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: The paper empirically studies the category selectivity of individual cells in hidden units of CNNs. It is a sort of "meta-study" and comparison of different metrics proposed to identify cells with a preference for a specific target category. The claimed finding is that there are no cells that are "sufficiently" selective to be called object detectors.

The paper is seemingly motivated by the authors' perceiving a contradiction: it is assumed that the power of neural networks is (among others) due to the distributed representation; whereas the presence of object detectors would, in the extreme case, mean that the representation is disentangled into a separate unit per category. It may be a matter of terminology, but this is where my disagreement with the authors start. I do not see a simplistic dichotomy, where one could or should determine which of the two interpretations is "right" or "wrong". In my view, which I believe is the mainstream interpretation, a distributed representation does not contradict the presence of specialised units. Some categories probably are easily identified by few distinctive features, so there will be more detector-like units; others are complex and hence more diffusely spread through the network; and of course there is no guarantee that the learned "object detectors" are tuned to exactly the target categories, after all it is the purpose of the network to gradually translate the data distribution to the label distribution - if the categories were directly apparent in the data, nearest-neighbour would be enough. So it is not only possible, but rather likely that the learned "object detectors" are to some degree driven by the statistics of the data, not the labels - e.g., there could be a highly selective "bird" unit which nevertheless has high false positive rate for any of the more specific bird species categories in the imageNet nomenclature. And vice versa, there could be a highly specific "Ferrari" detector that is so specialised that it has low recall for the "sports car" class (this case includes, among others, the case of viewpoint-specific detectors for certain categories). In the words of the paper, the "selective units are sensitive to some feature that is frequently, but not exclusively associated with the class" - I thought this is the standard majority view, not a surprising finding. In this context terminology matters: the study effectively tries to disprove that the network learns "near-perfect single output-category detectors", but who claims that it would do that?

I agree with the authors that there is by now a zoo of selectivity metrics that are not always highly correlated. But is that a problem? We have a zoo of quality metrics for many machine learning problems - that is not necessarily a weakness, but simply reflects the obvious fact that a single number is not enough to characterise performance in a complex cognitive task. It is the job of the researcher/user to chose the metric that s most suitable for their specific question, and to correctly interpret its numerical value.

Regarding the methodology, the paper did a lot of work to systematically crunch the numbers and analyse network units. It is a laudable effort that someone took on that job. A few technical decisions are unclear to me. Why analyse only some of the units? If one collects statistics over >2000 units of a fully connected layer, one might as well do the complete job and use all 4096 units. Similarly, why analyse only the correctly classified images? While it is clear that one must separately look at them, also the activations on incorrectly classified ones could provide valuable insights. E.g., do false positives of class X on average activate a certain "class X detector"? Why chose only the class with highest mean activation for CCMAS? That might be unrepresentative, e.g., a neuron might, for that particular class, always have high activation due to some very common background context, and still be not selective at all.

Regarding the results, I find them much less clear-cut than the paper claims. For example, I find it quite remarkable that some unit has 8% recall at perfect precision. After all, only approximately 0.1% of the images are in the correct category, so a unit that flags 8% of them without making a mistake is a pretty good detector for (part of) the target class, cf Fig. 3. Also regarding Fig. 2 / maximum informedness, the statistics actually do not look bad. Of course false alarm proportion remains high - but the chance level here is 99.9%, so even a 99% false alarm rate means that your unit can, on its own, reject 90% of the true negatives. I find the proposed "minimum condition" for an object detector (>50% recall at >50% precision) unrealistic: the top-1 accuracy of AlexNet is, to my knowledge, <63%. Even the complete network probably never reaches 50% recall for most classes.

Especially the user study - which is again a commendable effort - in my view does not confirm the claims. According to that study, almost 60% 0f all fc8 units are "object detectors", with very high conherence between humans and selectivity metrics.

Overall, while it is an interesting study, it remains unclear to me what I should learn from it. I don't see why different measures provide "misleading conclusions" that need to be rectified. Conclusions are the responsibility of the researcher interpreting the numbers, not of the formula to calculate some statistical performance metric. I am in a difficult situation here: the study is one of those things (like determining human performance on ImageNet, or re-coding some baseline where the original code is not available) where I find it valuable that someone did them in the community, but still I don't think they need a reviewed paper.  A note on the blog, or on arXiv, is enough.

review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: The paper makes an empirical claim that CNNs for object recognition do not contain hidden neuron which is highly selective to each class, mainly based on three aspects: (a) metrics related to the maximum informedness, (b) jitterplots of activation data, and (c) a user study assessing whether generated images maximizing a given unit is perceptible to the user. The paper point out these results are in contrast to the case of RNN, where it has been reported that many localist hidden units emerge. It is also noticed that the existing metrics for selectivity do not adequately discriminate highly selective units in CNN.  

In overall, the manuscript is well-written and easy-to-follow. I particularly appreciated a kindly presented overview on the literature and thoroughly conducted experiments including a complete user study. One of my key concerns, however, is that I am still not fully convinced whether the key finding in this paper - the lack of highly selective units in CNNs - is an indeed important problem for ICLR community: Personally, I feel the "existence" of selective units in RNN could be interesting, but the "non-existence" in the case of CNN is not that surprising for some readers, as it seems much likely (at least to me): The final layer of CNN would be surely selective across classes, but it may be not the case for the hidden layers - Nevertheless, some of the units may act selectively, not across classes but in some other concepts: e.g. stripes, orientations, etc. Therefore, I wonder if the paper could further provide a discussion on the importance of the key finding. 

- In Section 2 - Network and Dataset - "... We selected 233 ..." : Which criteria is actually used to choose the candidate units for the analysis?
- Do the overall results mean that the "maximum informedness"-based metrics are superior to the others for assessing selectivity of a unit? Also, are those metrics original to this paper?
- Currently, I feel the concept of "selectivity" is presented in somewhat subjectively: the definition could vary across the context. It would be nice if this could be more formalized to support the claims in the paper, e.g. the superiority of the proposed metrics.
review_title: Official Blind Review #3

Meta-review: This paper conducted a number of empirical studies to find whether units in object-classification CNN can be used as object detectors. The claimed conclusion is that there are no units that are sufficient powerful to be considered as object detectors. Three reviewers have split reviews. While reviewer #1 is positive about this work, the review is quite brief. In contrast, Reviewer #2 and #3 both rate weak reject, with similar major concerns. That is, the conclusion seems non-conclusive and not surprising as well. What would be the contribution of this type of conclusion to the ICLR community? In particular, Reviewer #2 provided detailed and well elaborated comments. The authors made efforts to response to all reviewers’ comments. However, the major concerns remain, and the rating were not changed. The ACs concur the major concerns and agree that the paper can not be accepted at its current state.
Decision: Reject