Are there any 'object detectors' in the hidden layers of CNNs trained to identify objects or scenes?
Authors: Ella M. Gale
Nicholas Martin
Ryan Blything
Anh Nguyen
Jeffrey S. Bowers
Venue:  (2020)
Keywords: neural networks
localist coding
selectivity
object detectors
CCMAS
CNNs
activation maximisation
information representation
network dissection
interpretabillity
signal detection

Abstract: Various methods of measuring unit selectivity have been developed with the aim of better understanding how neural networks work.  But the different measures provide divergent estimates of selectivity, and this has led to different conclusions regarding the conditions in which selective object representations are learned and the functional relevance of these representations. In an attempt to better characterize object selectivity, we undertake a comparison of various selectivity measures on a large set of units in AlexNet, including localist selectivity, precision, class-conditional mean activity selectivity (CCMAS), network dissection, the human interpretation of activation maximization (AM) images, and standard signal-detection measures.  We find that the different measures provide different estimates of object selectivity, with precision and CCMAS measures providing misleadingly high estimates. Indeed, the most selective units had a poor hit-rate or a high false-alarm rate (or both) in object classification, making them poor object detectors.  We fail to find any units that are even remotely as selective as the 'grandmother cell' units reported in recurrent neural networks. In order to generalize these results, we compared selectivity measures on a few units in VGG-16 and GoogLeNet trained on the ImageNet or Places-365 datasets that have been described as 'object detectors'. Again, we find poor hit-rates and high false-alarm rates for object classification. 

Under review as a conference paper at ICLR 2020
ARE THERE ANY 'OBJECT DETECTORS' IN THE HIDDEN LAYERS OF CNNS TRAINED TO IDENTIFY OBJECTS OR SCENES?
Anonymous authors Paper under double-blind review
ABSTRACT
Various methods of measuring unit selectivity have been developed with the aim better understanding how neural networks work. But the different measures provide divergent estimates of selectivity, and this has led to different conclusions regarding the conditions in which selective object representations are learned and the functional relevance of these representations. In attempt to better characterize object selectivity, we undertake a comparison of various selectivity measures large set of units in AlexNet, including localist selectivity (Bowers et al., 2014), precision (Zhou et al., 2015), class-conditional mean activity selectivity (CCMAS) (Morcos et al.) 2018), network dissection (Zhou et al. 2018 the human interpretation of activation maximization (AM) images, and standard signal- detection measures. We find that the different measures provide different estimates of object selectivity, with precision and CCMAS measures providing misleadingly high estimates. Indeed, the most selective units had poor hit-rate or a high false- alarm rate (or both) in object classification, making them poor object detectors. We fail to find any units that are even remotely as selective as the 'grandmother cell' units reported in recurrent neural networks. In order generalize these results, we compared selectivity measures on a few units in VGG-16 and GoogLeNet trained on the ImageNet or Places-365 datasets that have been described as 'object detectors'. Again, we find poor hit-rates and high false-alarm rates for object classification.
INTRODUCTION
There have been recent attempts understand how neural networks (NNs) work by analyzing hidden units one-at-a-time using various measures such localist selectivity (Bowers et al. 2014). class-conditional mean activity selectivity (CCMAS) (Morcos et al. 2018), precision (Zhou et al. 2015), network dissection (Zhou et al. 2018a). and activation maximization (AM) (Erhan al. 2009). These measures are all taken to provide evidence that some units respond highly selectively categories of objects under some conditions. Not only these findings surprising given the widespread assumption that NNs only learn highly distributed and entangled representations, they raise a host of questions, including the functional importance of these selective representations (Zhou et al. 2018b), the conditions in which they are learned (e.g., Morcos (e.g., et al., 2018), and the relation between these representations and the selective neurons observed in cortex (Bowers) 2009).
To answer these question, it is necessary to have a better understanding of what these metrics actually measure, and how they relate one another. Accordingly, we directly compare these measures of selectivity on the same set of units as well as adopt standard to provide better measures of single-unit selectivity to object category. In addition, to provide more intuitive assessment of selectivity, we report jitterplots for few of the most selective units that visually display how the unit responds to the different image categories. We focus on AlexNet (Krizhevsky et al. 2012) trained on ImageNet (Deng et al.) 2009) because many authors have studied the selectivity of single hidden units in this model using a range of quantitative (Zhou et al. 2a 2015) and qualitative (Nguyen al. 2017; Yosinski et al. 2015 Simonyan et al. 2013) methods. But we also compare different selectivity measures on specific units VGG-16 (Simonyan and Zisserman 2014) and GoogLeNet (Szegedy et al. 2015) trained on the the ImageNet and Places-365
measures in an attempt

---

Under review as a conference paper at ICLR 2020
datasets that were characterized by Zhou et al. (2018a) "object detectors" based on their Network Dissection method (Zhou et al. 2018a). Our main findings are:
1. The precision and CCMAS measures are misleading with near-maximum selectivity scores associated with units that strongly respond to many different image categories. By contrast, the signal-detection measures more closely capture the level of selectivity displayed in the jitterplots (Sec. 3.1
2. Units with interpretable AM images do not correspond to highly selective representations (Sec. (32).
3. The Network Dissection method also provides misleading measure for "object detectors"
(Sec. (.3)
In one line of research, Bowers et al. (0144 2016) assessed the selectivity of single hidden units in recurrent neural networks (RNNs) designed to model human short-term memory. They reported many 'localist' or 'grandmother cell' units that were 100% selective for specific letters or words, where all members of the selective category were more active than and disjoint from all non-members, as can be shown in jitterplots (Berkeley et al. 1995) (see Fig. ffor a unit selective to the letter 'j'). The authors argued that the network learned these representations in order to co-activate multiple letters or words at the same time in short-term memory without producing ambiguous blends of overlapping distributed patterns (the so-called 'superposition catastrophe"). Consistent with this hypothesis, localist units did not emerge when the model was trained on letters or words one-at-a-time (Bowers et al. 2014) (see Fig. [ for an example of a non-selective unit).
In parallel, researchers have reported selective units in the hidden layers of various CNNs trained to classify images into one multiple categories (Zhou et al. 2015] Morcos et al. 2018 Zeiler and Fergus 2014 EEhha et al., 2009), for review see see|Bowers (2177). For example, Zhou et al. (2015) assessed the selectivity of units in the pool5 layer of two CNNs trained to classify images into 1000 objects and 205 scene categories, respectively. They reported many highly selective units that they characterized as 'object detectors' in both networks. Similarly, |Mocoos et al. (2018) reported that CNNs trained on CIFAR-10 and ImageNet learned many highly selective hidden units, with CCMAS scores approaching the maximum of 1.0. These later findings appear to be inconsistent with Bowers al. (2016) who failed to observe selective representations in fully connected NNs trained on stimuli one-at-a-time (see Fig. [T), but the measures of selectivity that have been applied across studies are different, and accordingly. it is difficult to directly compare results.
A better understanding of the relation between selectivity measures is vital given that different measures are frequently used to address similar issues. For example, both the human interpretability of generated images (013) and localist selectivity (Bowers et al.] 2014) have been used to make claims about "grandmother cells', but is not clear whether they provide similar insights into unit selectivity. Similarly, based on their precision metric. metric,Zhou et al. (2015) claim that the object detectors learned in CNNs play an important role in identifying specifie objects, whereas|Morcos et al.) (2018) challenge this conclusion based on their finding that units with high CCMAS measures not especially important in the performance of their CNNs and concluded: "...it implics that methods for understanding neural networks based on analyzing highly selective single units, or finding optimal inputs for single units, such as activation maximization (Erhan et al.] (2009) may be misleading". This makes a direct comparison between selectivity measures all the more important.
In order to directly compare and have a better understanding of the different selectivity measures we assessed (1) localist, (2) precision, and (3) CCMAS selectivity of the conv5, fc6, and fc7 of AlexNet trained on ImageNet, and in addition, we employed a range of signal detection methods on these units, namely, (4) recall with 100% and 95% precision, (5) maximum informedness, (6) specificity at maximum informedness.. and (7) recall (also called sensitivity) at maximum informedness, and false alarm rates at maximum informedness (described in Sec. Sec. We also assessed the selectivity of few units in VGG-16 16 and GoogLeNet models trained on the ImageNet and Places-365 dataset that were highly selective according to the Network Dissection method (Zhou et al. 2018a). We show that the precision and CCMAS measures often provide misleadingly high estimates of object selectivity compared to other measures, and we do not find any units that can be reasonably described as 'object detectors' given that the most selective units show low hit-rate a high false-alarm rate (or both) when classifying images. At best, the most selective units in CNNs are sensitive to some unknown feature that is weakly associated with the class in question.
2

---

Under review as a conference paper at ICLR 2020
Figure 1: Examples of selectivity measures used. Top left: jitterplot of unit 113 in an RNN (under the superposition constraint) selective to the letter 'j" (Bowers et al. 2016). Top middle: jitterplot of a non-selective unit 160 found in an RNN trained on words one-at-a at--mmm- from (Bowers et al. 2016).
Top right: Activation maximization image of unit conv5g AlexNet that resembles a lighthouse (Nguyen et al., 2016). Bottom: highest-activation images for a 'lamp' detector with 84% precision in the layer conv5 of AlexNet; from Zhou et al. 2015).
In addition to these quantitative measures and jitterplots we assessed selectivity with a common qualitative measure, namely, human interpretation of images generated by state-of-the- art activation maximization (AM) method (Nguyen et al. 2017). AM images are generated to strongly activate individual units, and some of them interpretable by humans (e.g., a generated image that looks like a lighthouse, see Fig. D For the first time, we systematically evaluated the interpretability of the AM images and compare these ratings with the selectivity measures for corresponding units. We show that the few hidden units with interpretable AM images are not highly selective.
2
METHODS
Network and Dataset All 11.3M photos from the ImageNet ILSVRC 2012 dataset (Deng et al. 2009) were cropped to 277 x 277 pixels and classified by the pre-trained AlexNet CNN (Krizhessky et al. 2012) shipped with Caffe (Jia et al., 2014), resulting Once classified, the images are not re-cropped nor subject to any changes. We analyzed the fully connected (fc) layers: fc6 and fc7 (4096 units), and the top convolutional layer conv5 which has 256 filters. We only recorded the activations of correctly classified images. The activation files are stored in .h5 format and will be available at [btpp:// //nonynmiimi....... zedF orReview We randomly selected 233 conv5, 2738 fc6, 2239 fc7 units for analysis.
721,536 correctly classified images.
Localist selectivity Following Bowers et al. (2014), we define a unit to be localist for class A if the set of activations for class A was higher and disjoint with those of -A. Localist selectivity is easily depicted with jitterplots (Berkeley et al. 1995) in which a scatter plot for each unit is generated (see Figs. ② and 3) Each point in plot corresponds to unit's activation in response a single image, and only correctly classified images are plotted. The level of activation is coded along the r-axis, and an arbitrary value is assigned to each point on the y-axis.
Precision Precision refers to the proportion of items above some threshold from a given class. The precision method of finding object detectors involves identifying a small subset of images that most strongly activate unit and then identifying the critical part of these images that are responsible for driving the unit. Zhou al. (2013) took the 60 images that activated unit the most strongly and asked independent raters to interpret the critical image patches (e.g., if 50 of the 60 images were labeled as 'lamp', the unit would have precision index of 50/60 or 83%; see Fig. [T]. Object detectors were defined as units with precision score > 75%: they reported multiple such detectors. Here, approximate this approach by considering the 60 images that most strongly activate a given unit and assess the highest percentage of images from given output class.
CCMAS Morcos et al. (2018) introduced a selectivity index called the Class-conditiona. Mean Activation Selectivity (CCMAS). The CCMAS for class A compares the mean activation of all images in class A, MAs with the mean activation of all images not class A, p.A. and is given by: (nA a) (μA l ). Here, we assessed class selectivity for the highest mean activation class.
3

---

Under review as a conference paper at ICLR 2020
Activation Maximization We harnessed activation maximization method called Plug & Play Generative Networks (Nguyen et al. 2017) in which an image generator network was used to generate images (AM images) that highly activate a unit in a target network. We used the public code released by Nguyen et al. (2017) and their default hyperparameters] [T] We generated 100 separate images that maximally activated each unit in the conv5, fc6, and fc8 layers of AlexNet and asked participants to judge whether they could identify any repeating objects, animals, or places in images behavioral experiment (Sec. 3.2). Readers can test themselves at: https: part www.cccpaantggiiioe www.o//ggnnnnnnaaooo/op.. dynamiicci....mm 639077B222 3C3 9 ------5-------F--44--9----9-
sc/
Recall with perfect and 95% precision Recall with perfect and 95% precision are related to localist selectivity except that they provide a continuous rather than discrete measure. For recall with perfect precision we identified the image that activated a given unit the most and counted the number of images from the same class that were more active than all images from all other classes. We then divided this result by the total number of correctly identified images from this class. recall with a perfect precision score of is equivalent to a localist representation. Recall with a 95% precision allows 5% false alarms.
Maximum informedness Maximum informedness identifies the class and threshold where the highest proportion of images above the threshold and the lowest proportion of images below the threshold are from that class (Powers 2011). The informedness is computed for each class at each threshold, with the highest value selected. Informedness summarises the diagnostic performance of unit for a given class at certain threshold based the recall [True Positives / (True Positives False Negatives)] and specificity [True Negatives / (True Negatives + False Positives) in the formula (Powers 2011).
jinformedness recall + specificity
Sensitivity or Recall at Maximum Informedness For the threshold and class selected by Maximum Informedness, recall (or hit-rate) is the proportion of items from the given class that are above the threshold. Also known as true postive rate.
Specificity Maximum Informedness For the threshold and class selected by Maximum Informed- ness, the proportion of items that are not from the given class that are below the threshold. Also known as true negative rate.
False Alarm Rate at Maximum Informedness For the threshold and class selected by Maximum Informedness, the proportion of items that are not from the given class that are above the threshold.
Network Dissection To assess the selectivity of a unit in the Network Dissection technique, Zhou jet al. (2018a) compute the Intersection over Union (IoU) of an annotated input image I.c. for the set of all "concepts' c and spatial activation map. Mj. of where a unit k is. A unit h is taken as a detector for concept if its IoU,,e exceeds a pre-defined threshold T. See Zhou et al. (0018a) for more details.
3 RESULTS
3.1 COMPARISON OF SELECTIVITY MEASURES IN ALEXNET
The results from the various of selectivity measures applied to the conv5, fc6. and fc7 layers of AlexNet are displayed in Fig. -pi. We did not plot the localist selectivity as there were localist gganndmtthrr units'. The first point to note is that multiple units in the fc6 and fc7 layers had near 100% precision scores and multiple units had CCMAS scores approaching 1. For example, in layer fc7, we found 14 units with a precision > 0.9, and 1487 units with a CCMAS > 0.9. The second point that other measures provided much reduced estimates of selectivity. For example, the unit with the highest recall with a perfect precision score was only .08 (unit 255 responding to images of Monarch butterflies), and the unit with the top maximum informedness score (unit 3290 also responding to images of Monarch butterflies with score of 0.91) had false alarm rate above its optimal threshold > 99% (indeed the minimum false alarm rate was 0.96).
To illustrate the contrasting measures of selectivity consider unit fc6 depicted in Fig. 3 that has a precision score of 98% and a CCMAS score of .92. By Zhou et al. / criterion, this is a Monarch Butterfly' detector (its precision score > 75%). By contrast, the scatter plot and signal-detection
https://github...
4

---

Under review as a conference paper at ICLR 2020
Precision
b. No. of classes in top 100
CCMAS
No. of Classes top 100
0.0
0.8
0.6
CCMAS inighest
0.4
20
0.2
lce
Recall (precision 1)
c. Recall (precision 0.95)
Max. informedness
0.15
pretision 1.0
Rocallas at prectiion 35
0.0
Max Informetnnss
0.05
0.1
0.05
0.02
0.2
0.11
comv5
g. Specificity (at informedness)
h. Recall (at max. informedness)
False alarm proportion
(at max. informedness)
Spociiiitty for Max Informddnoss
Recal for Max informedrese
0.9
0.6
0.6
0.8
0.4
0.7
0.6
0.5
0onv5
conV5
conv5
E:6
127
Figure 2: Different selectivity measures across the conv5, fc6, and fc7 layers of AlexNet. Red-line: median of data, top and bottom of box edges is the 25th and 75th percentile, whiskers extend to extreme edges of distribution not considered outliers and red crosses are outliers. Green points and dashed lines the means of the distributions with standard errors. The high levels of selectivity observed with the precision and CCMAS measures are in stark contrast with the low levels of selectivity observed with the recall with perfect precision and high false-alarm rates at maximum informedness.
scores show this is a mischaracterisstion of this unit given that the false alarm rate at maximum informedness was greater than 99% and the modal response Monarch butterflies was
What level of selectivity is required before a unit can be considered an 'object detector' for a given category? In the end, this is terminological point. On an extreme view, one might limit the term to the 'grandmother units' that categorize objects with perfect recall and specificity, or alternatively, it might seem reasonable to describe unit as a detector for specific object category if there is some threshold of activation that supports more hits than misses (the unit is strongly activated by the majority of images from a given category), and at the same time, supports more hits than false alarms (the unit is strongly activated by items from the given category more often than by items from other categories). Or perhaps lower standard could be defended, but in our view, the term "object detector" suggests a higher level of selectivity than 8% recall at perfect precision. That said, our results show that some units respond strongly to some (unknown) features that are weakly correlated with an object category. For instance, unit fc6 is responding to features that occur more frequently in Monarch Butterflies than other categories. This can also be seen a recent ablation study in which removing the most selective units tended to impair the CNN's performance in identifying the corresponding object categories more than other categories (Zhou et al.] (0188). But again, the pattern of performance is not consistent with the units being labeled 'object detectors'.
5

---

Under review as a conference paper at ICLR 2020
50
40
Artivation
CO
20
40
Artivation
60
80
100
Max Adtvasonn
Figure 3: Data for unit fc61199. Left: activation jitterplot, black diamonds: Monarch butterfly images; grey circles: all other classes; white dashed line: threshold for the butterfly class maximum informedness; blue solid line: threshold for top 60 activations. Middle: histogram of activations of Monarch butterflies; red dashed line: threshold for the butterfly class maximum informedness; black solid line: threshold for top 60 activations. Inset: zoomed-in histogram of all activations across all ImageNet classes of unit fc61199 (N.B. this plot shows only the highest 121.
121,586 ,5866 activations; there are 596,734 activations at 0). There are Monarch butterfly images covering the whole range of values, with 72 images (5.8% of the total) having an activation of O. Right: example ImageNet images with activations of (top), the mean, 39. 39.2++0.6, (middle), and the maximum, 95, (bottom) of the range. Although the high precision score suggests that this unit is a butterfly detector this is misleading given there are butterfly images over the entire activation range (including 0).
3.2 HUMAN INTERPRETATION OF ACTIVATION MAXIMIZATION IMAGES FOR ALEXNET UNITS
Activation Maximization is one of the most commonly used interpretability methods for explain- ing what a single unit has learned in many artificial CNNs and even biological neural networks (see (seeenguyen et al. (2019) for survey). Our behavioral experiment provides the first quantitative assessment of AM images and compares AM interpretability to other selectivity measures.
Table 1: Human judgements of whether AM images look like familiar objects in layers conv5, fc6, and fc8 in AlexNet.
layer
'yes' responses (a)
% units 2 80% 'yes' response (b)
% overlap between humans and:
humans
most active
CCMAS
(c)
object (d)
class (c)
conv5
21.7% (11)1%)
4.3% (= 1,3%)
89.5% (1.5.7%)
34.1% (114.44%
0%
fc6
21.0% (#044%)
3.1% (= 0.4%)
80.4% (+4.11%)
23.3% (+5 (.)
18.9% (+5599%
fc8
71.2% (#0.6%)
59.3% 1..6)
96.5% (+0.4%)
95.4% (+0.6%)
94.6% (+00.%)
We generated 100 AM images images for every unit in the layers conv5, fc6, and fc8 in AlexNet, as in Nguyen et al. (0177), and displayed them as 10 x 10-image panels. A total of 3,299 image panels were used in the experiment (995 fc8, 256 conv5, and 2048 randomly selected fc6 image panels) and were divided into 64 counterbalanced lists for testing. To assess the interpretability for these units as object detectors, 333 paid volunteers were asked to look at image panels and asked if the images had an object/ / animal or place in common. If the answer was 'yes'. they were asked to write down generic name for that object (e.g. "fish" rather than "goldfish"). Analyses of common responses was done for any units where over 80% of humans agreed there was object present.
The results are summarized in Table I Not surprisingly, the AM images for output fc8 units are the most human- remaan-ecoonnzzable when they were given a consistent interpretation, they almost always (95,4% Table Id] match the corresponding ImageNet category. By contrast, less than 5% of units in conv5 or fc6 were associated with consistently interpretable images (Table [p), and the interpretations only weakly matched the category associated with the highest-activation images CCMAS selectivity (Table [Td Apart from showing that there are few interpretable units in the hidden layers of AlexNet, our findings show that the interpretability of images does not imply a high level of selectivity given the sienal-detection
objects across the AlexNet layers (71.2% Table Ta). In addition,

---

Under review as a conference paper at ICLR 2020
(a) conv5103
(b) fc6319
(c) fc8909
conv5es
fc6103
(1) fc88es
Figure 4: Example AM images that were either judged by all participants to contain objects (a-c) to be uninterpretable as objects (d-f). The human label for unit conv5183 (a) was 'dogs'; the most active image of 'flat-coated retriever': CCMAS class was 'monitor'. For fc6319 (b), subjects reported "green peppers' or 'apples' (all classified as the same broad class our analysis); both the most active item and CCMAS class 'Granny Smith apples', For fc8969 (c), humans suggested "beverage" or 'drink'; both the most active item and CCMAS class were 'eggnog'.
results (Fig, 2d-h). See Fig. for an example of the types of images that participants rated as objects or non-objects.
3.3 COMPARING SELECTIVITY MEASURES IN OTHER CNNS
100
200
300 400
25
50
75
100
100
200
300
400
a. GoogLeNet on ImageNet inception4e49a precision: 0.0 CCMAS:0.52
b. GoogLeNet Places-365
c. VGG-16 on Places-365
inception4eg24 precision: 0.27
conv5 __
precision: 0.53
CCMAS: 0.55
CCMAS: 0.82
14 72.50 μ=- = 22.81
#A 40.99 JL-,A = 11.78
= 157.6 μl = 15.2
Figure 5: The units with with the highest Network Dissection scores for the category 'bus". The scatter plots, precision, and CCMAS scores all indicate a low selectivity for this category. blue squares: 'school bus'; red pentagons: 'trolleybus'; green stars: 'minibus'; grey circles: other classes.
Thus far we have assessed the selectivity of hidden units in AlexNet and shown that no units can reasonably be characterized object detectors despite the high precision and CCMAS scores of some units. This raises the question as to whether more recent CNNs learn object detector units. In order to address this display jitterplots for three units that have the highest IoU scores according to the Network Dissection for the category BUS in (a) GoogLeNet trained on ImageNet. (b) GoogLeNet trained on Places-365, and (c) VGG-16 trained on Places-365. respectively (Zhou et al. 2018a). Models trained on the Places-365 dataset learn to categorize images into scenes (e.g., bedrooms, kitchens, etc.) rather than into object categories, and nevertheless, Zhou et al. (20T8 T8)) reported more object detectors in the former models. We illustrate the selectivity of the BUS category because is an output category in ImageNet so we can easily plot the jitterplots for these units.
As was the case with AlexNet, the jitterplots show that the most selective units show some degree of selectivity, with the BUS images more active on average compared to non-Buses and the percentage of nonzero activations for BUS higher than the non-BUS categories (see tables A3 A5] in the appendix for summary of more units). But the units are no more selective than the units we observed in AltxNet. Indeed, the precision measure of selectivity for the first units is 0.0, with none of the
7

---

Under review as a conference paper at ICLR 2020
units having a precision of .75 that was the criterion of object detectors by Zhou al. (2013). and CCMAS scores for first two units were roughly similar to the mean CCMAS score for AlexNet units in conv 5 (and much lower than the mean in fc6 and fc7). The most selective VGG-16 16 unit trained on Places-365 has lower precision and CCMAS scores than the Monarch Butterfly unit depicted in Figure 3. So again, different measures of selectivity provide support different conclusions, and even the most selective units are far from the selective units observed in recurrent networks as reported in in the appendix for more details about these three units.
Figure la. See tables A3 A5
4 DISCUSSIONS AND CONCLUSIONS
Our central finding is that different measures of single-unit selectivity for objects support very different conclusions when applied to the same units in AlexNet. In contrast with the precision (Zhou et al. (2135) and CCMAS (Morcos et al.) 2018) measures that suggest some highly selective units for objects in layers conv5, fc6, and fc7, the recall with perfect precision and false alarm rates at maximum informedness show low levels of selectivity. Indeed, the most selective units have a poor hit-rate or a high false-alama rate (or both) for identifying an object class. The same outcome was observed with units in VGG-16 and GoogLeNet trained on either ImageNet or the (35 dataset.
Not only do the different measures provide very different assessments of selectivity, the precision, CCMAS, and Network Dissection measures provide highly misleading estimates of selectivity that have led to mistaken conclusions. For example, unit fc61199 in AlexNet trained on ImageNet is considered an Monarch Butterfly detector according to Zhou et al. (2015) with a precision score of 98% (and CCMAS score of .93.. But the jitterplot in Fig. 3 and signal detection scores (e.g.. high false alarm rate at maximum informedness) show this is a mischaracterisation of this unit. In the same way, the Network Dissection method identified many object detectors in 16 and GoogLeNet CNNs, but the jitterplots in Fig. 5 (and precision scores) show that this conclusion is unjustified. For additional problems with the CCMAS score see Figure 5 in Appendix C. Similarly, the images generated by Activation Maximization also provided a misleading estimate of selectivity given that interpretable images were associated with very low selectivity scores. This has led to confusions that have delayed theoretical progress. For example, describing single units in CNNs as "object detectors" in response high precision measures (Zhou et al.) suggests similar types of representations are learned in CNNs and RNNs. Indeed, we are not aware of anyone in the machine learning community who has even considered the hypothesis that selectivity is reduced in CNNs compared RNNs. Our findings highlight the contrasting results.
What should be made of the finding that localist representations are sometimes learned in RNNs (units with perfect specificity and recall), but not in AlexNet and related CNNs? The fai failure to observe localist units in the hidden layers of these CNNs is consistent with Bowers et al. 2014)'s s claim that these units emerge in order to support the co-activation of multiple items at the same time in short-term memory. That is, localist representations may be the solution to the superposition catastrophe, and these CNNs only have to identify one image at time. The pressure to learn highly selective representations in response to the superposition constraint may help explain the reports of highly selective neurons in cortex given that the cortex needs to co-activate multiple items the same time in order to support short-term memory (Bowers et al. 2016).
Note, the RNNs that learned localist units were very small in scale compared to CNNs we have studied here, and accordingly, it is possible that the contrasting results reflect the size of the networks rather than the superposition catastrophe per se. Relevant to this issue number of authors have reported the existence of selective units in larger RNNs with short term memory (LSTM) units (Karpathy et al., 2016; Radford al. 2017) Lakretz et al.) 2019 Na al. 2019). Indeed, Lakretz let al. (2019) use the term 'grandmother cell' to describe the units they observed. It will be interesting to apply our measures of selectivity to these larger RNNs and see whether these units are indeed "grandmother units'.
should also be noted that there are recent reports of impressively selective representations in Generative Adversarial Networks (Bau et al. 2019) and Variational Autoencoders (Burgess et al. 2018) where the superposition catastrophe is not an issue. Again, it will be interesting to assess the selectivity of these units according signal detection measures in order to whether there additional computational pressures to learn highly selective or even grandmother cells. We will be exploring these issues in future work.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
D. Bau,J. J.-Y. Zhu, H. Strobelt, B. Z.ou, B. Tenenbaum, W. T. Freeman, and A. Torralba. Visualizing and understanding generative adversarial networks. In International Conference on Learning Representations, 2019. URL https:://ppeneeiieer net Eorueoormmioo -id-eiiddyyy Hyg_2255555
I. S. Berkeley, M. R. Dawson, D. A. Medler, D. P. Schopflocher, and L. Hornsby. Density plots of hidden value unit activations reveal interpretable bands. Connection Science, 7(2):167-187, 1995.
S. Bowers. On the biological plausibility of grandmother cells: implications for neural network theories in psychology and neuroscience. Psychological review, 1 16(1):2220 2009.
J. S. Bowers. Grandmother cells and localist representations: a review of current thinking. Language, Cognition, and Neuroscience, pages 257-273, 2017.
J. S. Bowers, I. Vankov, M. F. Damian, and C. J. Davis. Neural networks learn highly selective representations in order to overcome the superposition catastrophe. Psychological review, 121(2): 248-261, 2014.
J. S. Bowers, I. Vankov, M. Damian, and C. J. Davis. Why do some neurons in cortex respond to information in selective manner? insights from artificial neural networks. Cognition, 148:47-63, 2016.
C. Burgess, Higgins A. Pal, L. Matthey, N. Watters, G. Desjardins, and A. Lerchner. Under- standing disentangling in 3- VAE. arXiv preprint arXiv:1804.03999 2018.
J. Deng. W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In Computer Vision and Pattern Recognition, 2009. CVPR 2009, IEEE Conference on, pages 248-255 IEEE, 2009.
D. Erhan, Y. Bengio, A. Courville, and P. Vincent. Visualizing higher-layer features of a deep network. University of Montreal, 13;1 2009.
Y. Jia, E. Shelhamer, Donahue, S. Karayev, Long, Girshick, S. Guadarrama, and T. Darrell. Caffe: Convolutional architecture for fast feature embedding. arXiv preprint arXiv: arXiv:1408. 44845009,, 2014.
A. Karpathy, J. Johnson, and Fei-Fei. Visualizing and understanding recurrent networks. Workshop Truck at International Conference on Learning Representations, 2016.
A. Krizhevsky, I. Sutskever, and G. F. Hinton. Imagenet classification with deep convolutional neural networks. In Advances neural information processing systems, pages 1097-1105, 2012.
Y. Lakretz, G. Kruszewski, T. Desbordes, Hupkes, S. Dehaene, and M. Baroni. The emergence of number and syntax units in Istm language models. arXiv preprint arXiv: 1903.00933775 2019.
Q. V. Le. Building high-level features using large scale unsupervised learning. In 2013 IEEE international conference on acoustics, speech and signal processing, pages 8595-8598. IEEE, 2013.
A. S. Morcos, D. G. Barrett, N. C. Rabinowitz, and M. Botvinick. On the importance of single directions for generalization. In International Conference on Learning Representations, 2018. URL htthhttss///ppereeeeee net. www.:oiidmmuuug11111
S. Na, J. Choe, D.-H. Lee, and G. Kim. Discovery of natural language concepts in individual units of cnns. International Conference on Learning Representations, 2019. URL https
//openreview. www.teefuuuunet.
A. Nguyen, A. Dosovitskiy, J. Yosinski. T. Brox, and J. Clune. Synthesizing the preferred inputs for neurons in neural networks via deep generator networks. In Advances in Neural Information Processing Systems, pages 3387-3395. 2016.
A. Nguyen, J. Clune, Y. Bengio, A. Dosovitskiy, and J. Yosinski. Plug & play generative networks: Conditional iterative generation of images in latent space. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4467-4477, 2017.

---

Under review as a conference paper at ICLR 2020
A. Nguyen, J. Yosinski, and J. Clune. Understanding neural networks via feature visualization: A survey. arXiv preprint arXiv:1904.08939, 2019.
D. M. Powers. Evaluation: from precision, recall and f-measure to roc, informedness, markedness and correlation. 2011.
A. Radford, Jozefowiez, and I. Sutskever, Learning to generate reviews and discovering sentiment. arXiv preprint arXiv:/704.0144: 2017.
K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint UrXiv:1409. 2014.
K. Simonyan, A. Vedaldi, and A. Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint urXiv: 2013.
C. Szegedy, W. Liu, Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1-9, 2015.
Yosinski, J. Clune, A. Nguyen, T. Fuchs, and H. Lipson. Understanding neural networks through deep visualization. arXiv preprint arXiv:/506. 06579, 2015.
M. D. Zeiler and R. Fergus. Visualizing and understanding convolutional networks. In European conference on computer vision, pages 818-833. Springer, 2014.
B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba. Object detectors emerge in deep scene CNNs. In International Conference on Learning Representations, 2015.
B. Zhou, D. Bau, A. Oliva, and A. Torralba. Interpreting deep visual representations via network dissection, IEEE transactions pattern analysis and machine intelligence, 2018a.
B. Zhou, Y. Sun, D. Bau, and A. Torralba. Revisiting the importance of individual units in CNNs via ablation. arXiv preprint arXiv: 2018b.
10

---

Under review as a conference paper at ICLR 2020
APPENDIX
A METHODOLOGICAL DETAILS FOR THE BEHAVIORAL EXPERIMENT
One hundred generated images were made for every unit in layers conv5, fc6 and fc8 in AlexNet, as in Nguyen et al. (2017), and displayed as 10x10 10 image panels (figures A4 and Figures AaarressAAaana and A3). total of 3,299 image panels were used in the experiment (995 fc8, 256 conv5, and 2048 randomly selected fc6 image panels) and were divided into 64 counterbalanced lists of 51 or 52 (4 conv5, 15 or 16 fc8 and 32 fc6). 51 of the lists were assigned to 5 participants and 13 lists were assigned to participants.
To test the interpretability of these units, paid volunteers were asked to look at image panels and asked if the images had object / animal place in common. If the answer 'yes', they were asked to name that object simply (i.e. fish rather than goldfish). Analyses of common responses was 80% of humans agreed there was object present, by reading
carried for any units where the human responses and comparing them to both each other and to the output classes. Agreement was taken if the object was the same rough class. For example, 'beer', 'glass', and 'drink' were all considered to be in agreement in the general object of 'drink', and in agreement with both the classes of "wine glass' and 'beer' these classes were also general drink classes (this is an actual example, most responses were more obvious and required far less interpretation than that). Participants were given six practice trials, each with panels of 20 images before starting the main experiment. Practice trials included images that varied in their interpretability.
Q1. Can you identify multiple examples of an everyday object, place or animal?
Click "Yes' or "No'. In this example, the answer is After After pressing "Yes', two follow-up questions will appear, shown below.
Yes
No
Q2. You answered 'Yes'. Indicate what percentage of people you think would also answer 'Yes":
Click on one of the boxes (in this case, 80-100% is appropriate answer)
20-40%
90%%
Q3. Please write which everyday object, place or animal you were referring to: fish, goldfish
Type your answer in the text box. You may write a maximum things.
Figure A1: Example screen from the identification task shown participants part of the instructions. The images included on this practice trial are ImageNet2012 images, not AM images.
Some examples of the 10x10 grids of activation maximisation images that were presented to partici- pants are shown in Figures A2 A3 and A4 Figure A2 shows an example from conv5 that human participants agreed had no obvious object common (although there are repeated shape motifs, the participants were specifically asked for objects, and not abstract concepts like shape or color. Figure A3 is also from the conv5 and was judged by participants
images containing 'dogs'.
11

---

Under review as a conference paper at ICLR 2020
Figure paper.
A4 is the AM images for the supposed 'butterfly detector' unit example discussed in the
c t
Figure A2: Example activation maximisation images for unit conv5.65. These images were judged by humans to not contain any interpretable objects in common (although the reader may agree that there are some shape and colour similarities in the images).
12

---

Under review as a conference paper at ICLR 2020
Figure A3: Example activation maximisation images for unit conv5.183, These images were judged by humans to contain some interpretable images, in this case, of the type 'dogs'.
13

---

Under review as a conference paper at ICLR 2020
Figure A4: Example activation maximisation images for unit f199. Whilst there are some butterfly wing shapes in these images, there are not obvious butterflies. N.B. the second highest activating class for this unit is ladybirds, and there are some orange round shapes that could conceivably be ladybug-alikes.
14

---

Under review as a conference paper at ICLR 2020
B FURTHER DATA ON THE SELECTIVITY MEASURES ACROSS ALEXNET
Table Al gives the highest values of CCMAS and precision for each layer in AlexNet. It worth noting that the highest CCMAS score of all hidden units was .94 (fc7.31), which first glance suggests that this unit is close to 'perfect' selectivity. However, this unit only has low a precision score of 11%. In other words, although the mean activation for the given class is very high relative to the mean of all other activations (high CCMAS), the proportion of items from that class in the 100 most active items is low (low precision). See appendix Sec. C discussion of how this occurs and Fig. a5 for an illustrative example.
LAYER.UNIT CCMAS
Precision
Top CCMAS units
output.322
0,991
100%
fc7.31
0,94
11%
fc6.582
0.93
1%
conv5.78
0.75
5%
Top precision units
output.0
0,99
100%
fc7 .255
0.90
97%
fc6.1199
0.92
95%
conv5.0
0.55
77%
Table A1: The units with the highest CCMAS and precision scores in AlexNet. Unit fc6.1199 was displayed in Fig.
Table A2 shows positive correlations between four of the selectivity measures used. There are moderate positive correlations between precision and CCMAS; and precision and Recall at 95% precision. The other correlations between selectivity measures have weak positive correlations. All four selectivity measures negatively correlated with the number of classes present in the 100 most active items, that is, the more selective the unit, the fewer classes will be represented in the most active 100 items.
CCMAS 0.38
recalllo 95
Max. Inf.
No. classes in top100
precision CCMAS
0.30
0.15
-0.68
0.09
0.14
-0.47
recalllo.gg 9 Max. Inf.
0.10
-0.19
-0.22
Table A2: The correlations between the different measures. (All < .001)
C FURTHER ISSUES WITH THE CCMAS MEASURE
The CCMAS measure is based on comparing the mean activation of a category with the mean activation for all other items, and this is problematic for few reasons. First, in many units large proportion of images do not activate unit at all. For instance, our butterfly "detector' unit fc6.1196 has high proportion of images with an activation of 0.0 (see figure (3). Indeed, the inset on the middle figure shows that the distribution can be better described by exponential-derived fits rather than Gaussian. This means that the CCMAS selectivity is heavily influenced by the the proportion of images that have an activation value of zero (or close to zero). This can lead to very different estimates of selectivity for CCMAS and precision or localist selectivity, which are driven by the most highly activated items. In A5 we generate example data to highlight ways in which CCMAS score may be non-intuitive. In subplot (a) we demonstrate that unit can have a CCMAS score of of 1.0 despite only single item activating the unit. The point that we wish to emphasise is that high CCMAS score does not necessarily imply selectivity for a given class, but might in fact relate to selectivity for small subset of items from given class, and this is especially true when unit's
15

---

Under review as a conference paper at ICLR 2020
activation is sparse (many items do not activate the unit). However, the reverse also be true. In subplot (c) we demonstrate that unit can have a very low CCMAS score of .06 despite all of the most active items being from the same class.
In addition, if the CCMAS provided a good measure unit's class selectivity, then one should expect that high measure of selectivity for class would imply that the unit is not highly selective for other classes. However, the CCMAS score for the most selective category and the second most selective category (CCMAS2) were similar across the conv5, fc6 and fc7. layers, with the mean CCMAS scores .491, .844, and .848, and the CCMAS2 scores ,464, .821, .831. For example, unit fc7.0 has a CCMAS of .813 for the class 'maypole', and CCMAS2 score of .808 for "chainsaw' (with neither of these categories corresponding 'orangutan" that had the highest precision of score of 14%).
CCMAS prec. 11%, TCS = 1%
CCMAS prec. 100%, TCS 100%
CCMAS 0.06, 100%, TCS 100%
0.2
0 0.8
Activation
0.8
0.2
Activation
Activation
(a) One active item from one class.
(c) One class more active than the others.
(b) Archetypal 'GMC' unit. CCMAS = precision 100%
CCMAS=
CCMAS 0.06
precision 11%
precision 100%
Figure A5: Example of where the CCMAS does not match intuitive understandings of selectivity. Generated example data: (a) If a unit is off to all but single image from large class of objects, the CCMAS for that class is (maximum possible selectivity). (b) An archetypal 'grandmother' cell (GMC), where the unit is strongly activated all members of class and off everything else. The CCMAS is the same for (b) as for (a) although the precision is vastly different. (c): If a unit has high activations for all classes, but one class (black squares) 0.1 more than all others (coloured circles), the CCMAS is very low (0.06) despite being % 100 precision. The generated examples are for 10 classes of 100 items
D TESTING UNITS IN OTHER MODELS
To investigate units claimed by Zhou et al. (018a) to be object detectors, we focus on units from single layer that are reported be 'bus detectors' that is, units with IoU 2 .04. We used the first 100 images per class from the ImageNet 2012 dataset as our test data. There are three classes of bus in this dataset:
4 school bus", 'n04487081 trolleybus, trolley coach, trackless trolley', "n03769881 minibus', and this corresponded to 300 items out of 100000 images. Data for all bus unit detectors for VGG trained on places 365 are shown in table for GoogLeNe trained on places 365 in table table[A4] and for GoogLeNet trained on ImageNet are shown in table AS Note that for all units there are very few busses with activation at zero and that the mean activation for busses is higher than the mean activation for non- non-busses. However, all precision scores all below .6, meaning that of the 100 items that most strongly activated the unit, at least 40 of them were not busses. Together these results suggests that whilst these units demonstrate some sensitivity to busses, they show poor specificity for busses (e.g., high false-alarm rate).
16

---

Under review as a conference paper at ICLR 2020
unit
loU
no.0,.20 Ir E
no.a,,0
μA
precision
CCMAS
I. E -A
conv5 3191
.15
99.0%
63.9%
131.9
16.1
.45
.78
conv5_33
.15
99.0%
49.1%
157.6
15.2
.53
.82
conv5_33ss 5333
.08
99.0%
71.4%
101.7
17.5
.24
.71
conv5_31e
.07
97.3%
61.7%
75.5
12.5
.19
.72
conv5_3 __11
.06
97.4%
41.0%
62.8
9.1
.12
.75
conv5 3443
.04
95.3%
38.2%
59.3
8.1
.12
.76
conv5 313t
.04
93.7%
22.3%
54.0
5.86
.08
.80
Table A3: Selectivity measures for VGG-16, trained on Places-365, top convolutional layer units identified by /hhou et al. (2018a) object detectors. Standard errors not shown for space, but were below +5. The loU from Zhou et al. (2018a)'s network dissection method. no.a,>0 and n0.0,>0 A refer to the proportion of activations that were greater than zero for busses and non-busses respectively. μa and a are the class means for busses and non busses respectively.
unit
loU
correct
n
no.0,>0
PA
P-A
precision
CCMAS
x E A
AA
incep4e224
.17
100.0
91.4
41.0
11.8
.27
.55
incep44745
.13
T
98.3
74.8
34.8
11.4
.06
.51
incep4e791
.11
98.3
71.4
32.7
5.3
.41
.72
incep4e194
.11
100.0
85.3
26.6
8.8
.02
.51
incep4eg2
.11
100.0
97.3
26.7
10.9
.14
.42
incep4er30
.11
100.0
78.8
38.7
9.9
.05
.59
incep4ep03
.10
96.0
38.0
33.4
3.7
.15
.80
incep4eşa
.10
100.0
91.6
38.3
9.5
.35
incep4e77
.08
97.3
54.6
21.7
5.2
.00
.61
incep4ena
.08
100.0
88.0
24.9
.02
incep4ey08
.06
F
100.0
85.1
29.7
9.1
.02
.53
incep4eg01
.06
100.0
64.5
35.2
.14
incep4e100
.06
F
99.7
92.2
21.5
7.7
.09
.47
incep4es
.05
99.7
83.5
18.5
7.3
.01
.43
incep4e121
.05
F
100.0
90.4
17.9
8.9
.01
.34
incep4e622
.05
96.0
65.0
27.5
6.4
.20
.62
incep4es7
.04
99.3
86.4
21.1
9.3
.04
.39
Table A4: Selectivity measures for GoogLeNet, trained Places-365, layer inception4e units identified by Zhou al. (20T8a) as object detectors. Standard errors not shown for space, but were below +2. The IoU is from Zhou et al. 2018a)' network dissection method. A units is marked as correct if there was single bus in the example pictures on the website (http: //t////aaaee di ssect] and false if not. Where units False, this might suggest that the units were responding to 'bus like' features in none bus objects. no.a,>0 and no.a,>0 E A refer to the proportion of activations that were greater than zero for busses and non-busses respectively. μ. and μ-A are the class means for busses and non busses respectively.
csail mit edu/ http//issecttt wwwss/et/seee /eeeeettttooooloeeeooeooeoooooot googl waareeepppceeeeeeeeeeeee
17

---

Under review as a conference paper at ICLR 2020
unit
loU
correct
n0.
n
μA
precision
CCMAS
x E -
incep4ex14
111
F
99.0
82.4
72.5
22.8
.00
.52
incep4ea28
10
T
100.0
72.6
109.4
17.6
.45
.72
incep4es69
.10
T
99.7
85.9
74.9
20.0
.05
.58
incep4eни4
.10
T
100.0
71.6
67.0
18.5
.00
.57
incep4e4ss
.09
T
99.7
89.6
69.1
14.3
.66
incep4es79
.09
T
100.0
97.0
91.5
26.0
.23
.56
incep4e331
.08
T
98.0
75.5
51.0
11.8
.12
.62
incep4esa2
.08
T
100.0
83.4
125.7
21.95
.58
.70
incep4ex98
.07
97.7
77.2
73.5
15.0
.52
.66
incep4e534
.07
F
99.3
81.2
62.7
19.
.02
.53
incep4ea03
.07
T
98.7
91.2
75.4
22.3
.15
.54
incep4e573
.07
T
99.7
88.4
88.6
23.0
.33
.59
incep4exse
.06
T
98.7
78.1
34.7
14.6
.00
.41
incep4e207
.06
T
100.0
93.5
76.1
21.3
.07
.56
incep4e4m1
.06
F
99.0
74.5
41.1
13.7
.01
.50
incep4es15
.06
T
98.0
83.9
59.9
18.1
.20
.54
incep4es27
.06
F
100.0
91.5
58.0
21.7
.00
.46
incep4esm
.05
F 100.0
89.4
53.5
21.7
.00
.42
incep4es0s
.05
F
100.0
89.4
53.5
21.7
.00
.42
incep4es41
.05
F
99.67
88.7
44.9
13.7
.00
.53
incep4e367
.05
T
97.3
80.3
37.7
15.4
.02
.42
incep4esss
.05
T
100.0
82.45
107.2
21.0
.67
incep4e512
.05
T
100.0
91.5
52.9
22.4
.05
.41
incep4e297
.04
T
99.7
90.2
48.2
17.9
.00
.46
incep4e4an
.04
T
100.0
92.9
69.4
21.4
.02
.53
Table A5: Selectivity measures for GoogLeNet, trained on ImageNet, layer inception4e units units identified by Zhou al. (2018a) as object detectors. Standard errors not shown for space, but were below +2. A units is marked as correct if there was single bus in the 4 example pictures on the web- site (ht //netdi ssect. csail. mit edu/diss dissecc///// www.gooooo.ooooooooooooooooe denet__maaenee__. and false if not. This might suggest that the units were responding to 'bus like' features in none bus objects.
18


Reviews:
[{"review_id": "HJxqg9VQFB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper makes an empirical claim that CNNs for object recognition do not contain hidden neuron which is highly selective to each class, mainly based on three aspects: (a) metrics related to the maximum informedness, (b) jitterplots of activation data, and (c) a user study assessing whether generated images maximizing a given unit is perceptible to the user. The paper point out these results are in contrast to the case of RNN, where it has been reported that many localist hidden units emerge. It is also noticed that the existing metrics for selectivity do not adequately discriminate highly selective units in CNN.  \n\nIn overall, the manuscript is well-written and easy-to-follow. I particularly appreciated a kindly presented overview on the literature and thoroughly conducted experiments including a complete user study. One of my key concerns, however, is that I am still not fully convinced whether the key finding in this paper - the lack of highly selective units in CNNs - is an indeed important problem for ICLR community: Personally, I feel the \"existence\" of selective units in RNN could be interesting, but the \"non-existence\" in the case of CNN is not that surprising for some readers, as it seems much likely (at least to me): The final layer of CNN would be surely selective across classes, but it may be not the case for the hidden layers - Nevertheless, some of the units may act selectively, not across classes but in some other concepts: e.g. stripes, orientations, etc. Therefore, I wonder if the paper could further provide a discussion on the importance of the key finding. \n\n- In Section 2 - Network and Dataset - \"... We selected 233 ...\" : Which criteria is actually used to choose the candidate units for the analysis?\n- Do the overall results mean that the \"maximum informedness\"-based metrics are superior to the others for assessing selectivity of a unit? Also, are those metrics original to this paper?\n- Currently, I feel the concept of \"selectivity\" is presented in somewhat subjectively: the definition could vary across the context. It would be nice if this could be more formalized to support the claims in the paper, e.g. the superiority of the proposed metrics.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "B1x6uRTsYB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work investigates the collection of methods that have been proposed to find units in neural networks that are selective for certain object classes.  Previous works have used different measures of selectivity (with sometimes contradictory results), and the authors investigate the degree to which these units qualify as “object detectors”.\n\nThis research area is important for understanding deep networks because claims have been made as to the relative importance (or lack thereof) of these individual units as identified by different measures vis-a-vis distributed representations -- the identification of such units would be interesting for understanding the predictions of classification networks.\n\nThe authors find that (1) different proposed measures of selectivity are not consistent and (2) units identified as selective cannot be considered object detectors due to the high false alarm / low hit rates, analyzing a large number of selectivity measures.  I would have liked to see experiments on more recent architectures (the focus of the paper is on a dated architecture (AlexNet)); there is analysis on units in GoogLeNet and VGG-16 but it would also be interesting to see results for more modern architectures (e.g. DenseNet and ResNet).\n\nOverall, I think that the authors have presented a strong meta-analysis and compelling argument for further study in rigorously identifying the presence (or lack thereof) of selective units in neural networks and the degree to which they may be considered \"object detectors.\"", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "B1e6QO3SKH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper empirically studies the category selectivity of individual cells in hidden units of CNNs. It is a sort of \"meta-study\" and comparison of different metrics proposed to identify cells with a preference for a specific target category. The claimed finding is that there are no cells that are \"sufficiently\" selective to be called object detectors.\n\nThe paper is seemingly motivated by the authors' perceiving a contradiction: it is assumed that the power of neural networks is (among others) due to the distributed representation; whereas the presence of object detectors would, in the extreme case, mean that the representation is disentangled into a separate unit per category. It may be a matter of terminology, but this is where my disagreement with the authors start. I do not see a simplistic dichotomy, where one could or should determine which of the two interpretations is \"right\" or \"wrong\". In my view, which I believe is the mainstream interpretation, a distributed representation does not contradict the presence of specialised units. Some categories probably are easily identified by few distinctive features, so there will be more detector-like units; others are complex and hence more diffusely spread through the network; and of course there is no guarantee that the learned \"object detectors\" are tuned to exactly the target categories, after all it is the purpose of the network to gradually translate the data distribution to the label distribution - if the categories were directly apparent in the data, nearest-neighbour would be enough. So it is not only possible, but rather likely that the learned \"object detectors\" are to some degree driven by the statistics of the data, not the labels - e.g., there could be a highly selective \"bird\" unit which nevertheless has high false positive rate for any of the more specific bird species categories in the imageNet nomenclature. And vice versa, there could be a highly specific \"Ferrari\" detector that is so specialised that it has low recall for the \"sports car\" class (this case includes, among others, the case of viewpoint-specific detectors for certain categories). In the words of the paper, the \"selective units are sensitive to some feature that is frequently, but not exclusively associated with the class\" - I thought this is the standard majority view, not a surprising finding. In this context terminology matters: the study effectively tries to disprove that the network learns \"near-perfect single output-category detectors\", but who claims that it would do that?\n\nI agree with the authors that there is by now a zoo of selectivity metrics that are not always highly correlated. But is that a problem? We have a zoo of quality metrics for many machine learning problems - that is not necessarily a weakness, but simply reflects the obvious fact that a single number is not enough to characterise performance in a complex cognitive task. It is the job of the researcher/user to chose the metric that s most suitable for their specific question, and to correctly interpret its numerical value.\n\nRegarding the methodology, the paper did a lot of work to systematically crunch the numbers and analyse network units. It is a laudable effort that someone took on that job. A few technical decisions are unclear to me. Why analyse only some of the units? If one collects statistics over >2000 units of a fully connected layer, one might as well do the complete job and use all 4096 units. Similarly, why analyse only the correctly classified images? While it is clear that one must separately look at them, also the activations on incorrectly classified ones could provide valuable insights. E.g., do false positives of class X on average activate a certain \"class X detector\"? Why chose only the class with highest mean activation for CCMAS? That might be unrepresentative, e.g., a neuron might, for that particular class, always have high activation due to some very common background context, and still be not selective at all.\n\nRegarding the results, I find them much less clear-cut than the paper claims. For example, I find it quite remarkable that some unit has 8% recall at perfect precision. After all, only approximately 0.1% of the images are in the correct category, so a unit that flags 8% of them without making a mistake is a pretty good detector for (part of) the target class, cf Fig. 3. Also regarding Fig. 2 / maximum informedness, the statistics actually do not look bad. Of course false alarm proportion remains high - but the chance level here is 99.9%, so even a 99% false alarm rate means that your unit can, on its own, reject 90% of the true negatives. I find the proposed \"minimum condition\" for an object detector (>50% recall at >50% precision) unrealistic: the top-1 accuracy of AlexNet is, to my knowledge, <63%. Even the complete network probably never reaches 50% recall for most classes.\n\nEspecially the user study - which is again a commendable effort - in my view does not confirm the claims. According to that study, almost 60% 0f all fc8 units are \"object detectors\", with very high conherence between humans and selectivity metrics.\n\nOverall, while it is an interesting study, it remains unclear to me what I should learn from it. I don't see why different measures provide \"misleading conclusions\" that need to be rectified. Conclusions are the responsibility of the researcher interpreting the numbers, not of the formula to calculate some statistical performance metric. I am in a difficult situation here: the study is one of those things (like determining human performance on ImageNet, or re-coding some baseline where the original code is not available) where I find it valuable that someone did them in the community, but still I don't think they need a reviewed paper.  A note on the blog, or on arXiv, is enough.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper conducted a number of empirical studies to find whether units in object-classification CNN can be used as object detectors. The claimed conclusion is that there are no units that are sufficient powerful to be considered as object detectors. Three reviewers have split reviews. While reviewer #1 is positive about this work, the review is quite brief. In contrast, Reviewer #2 and #3 both rate weak reject, with similar major concerns. That is, the conclusion seems non-conclusive and not surprising as well. What would be the contribution of this type of conclusion to the ICLR community? In particular, Reviewer #2 provided detailed and well elaborated comments. The authors made efforts to response to all reviewers’ comments. However, the major concerns remain, and the rating were not changed. The ACs concur the major concerns and agree that the paper can not be accepted at its current state.
Decision: Reject