Are there any 'object detectors' in the hidden layers of CNNs trained to identify objects or scenes?

1 INTRODUCTION . There have been recent attempts to understand how neural networks ( NNs ) work by analyzing hidden units one-at-a-time using various measures such as localist selectivity ( Bowers et al. , 2014 ) , class-conditional mean activity selectivity ( CCMAS ) ( Morcos et al. , 2018 ) , precision ( Zhou et al. , 2015 ) , network dissection ( Zhou et al. , 2018a ) , and activation maximization ( AM ) ( Erhan et al. , 2009 ) . These measures are all taken to provide evidence that some units respond highly selectively to categories of objects under some conditions . Not only are these findings surprising given the widespread assumption that NNs only learn highly distributed and entangled representations , they raise a host of questions , including the functional importance of these selective representations ( Zhou et al. , 2018b ) , the conditions in which they are learned ( e.g. , Morcos et al. , 2018 ) , and the relation between these representations and the selective neurons observed in cortex ( Bowers , 2009 ) . To answer these question , it is necessary to have a better understanding of what these metrics actually measure , and how they relate to one another . Accordingly , we directly compare these measures of selectivity on the same set of units as well as adopt standard signal-detection measures in an attempt to provide better measures of single-unit selectivity to object category . In addition , to provide a more intuitive assessment of selectivity , we report jitterplots for a few of the most selective units that visually display how the unit responds to the different image categories . We focus on AlexNet ( Krizhevsky et al. , 2012 ) trained on ImageNet ( Deng et al. , 2009 ) because many authors have studied the selectivity of single hidden units in this model using a range of quantitative ( Zhou et al. , 2018a ; 2015 ) and qualitative ( Nguyen et al. , 2017 ; Yosinski et al. , 2015 ; Simonyan et al. , 2013 ) methods . But we also compare different selectivity measures on specific units in VGG-16 ( Simonyan and Zisserman , 2014 ) and GoogLeNet ( Szegedy et al. , 2015 ) trained on the the ImageNet and Places-365 datasets that were characterized by Zhou et al . ( 2018a ) as “ object detectors ” based on their Network Dissection method ( Zhou et al. , 2018a ) . Our main findings are : 1 . The precision and CCMAS measures are misleading with near-maximum selectivity scores associated with units that strongly respond to many different image categories . By contrast , the signal-detection measures more closely capture the level of selectivity displayed in the jitterplots ( Sec . 3.1 ) . 2 . Units with interpretable AM images do not correspond to highly selective representations ( Sec . 3.2 ) . 3 . The Network Dissection method also provides a misleading measure for “ object detectors ” ( Sec . 3.3 ) . In one line of research , Bowers et al . ( 2014 ; 2016 ) assessed the selectivity of single hidden units in recurrent neural networks ( RNNs ) designed to model human short-term memory . They reported many ‘ localist ’ or ‘ grandmother cell ’ units that were 100 % selective for specific letters or words , where all members of the selective category were more active than and disjoint from all non-members , as can be shown in jitterplots ( Berkeley et al. , 1995 ) ( see Fig . 1 for a unit selective to the letter ‘ j ’ ) . The authors argued that the network learned these representations in order to co-activate multiple letters or words at the same time in short-term memory without producing ambiguous blends of overlapping distributed patterns ( the so-called ‘ superposition catastrophe ’ ) . Consistent with this hypothesis , localist units did not emerge when the model was trained on letters or words one-at-a-time ( Bowers et al. , 2014 ) ( see Fig . 1 for an example of a non-selective unit ) . In parallel , researchers have reported selective units in the hidden layers of various CNNs trained to classify images into one of multiple categories ( Zhou et al. , 2015 ; Morcos et al. , 2018 ; Zeiler and Fergus , 2014 ; Erhan et al. , 2009 ) , for a review see Bowers ( 2017 ) . For example , Zhou et al . ( 2015 ) assessed the selectivity of units in the pool5 layer of two CNNs trained to classify images into 1000 objects and 205 scene categories , respectively . They reported many highly selective units that they characterized as ‘ object detectors ’ in both networks . Similarly , Morcos et al . ( 2018 ) reported that CNNs trained on CIFAR-10 and ImageNet learned many highly selective hidden units , with CCMAS scores approaching the maximum of 1.0 . These later findings appear to be inconsistent with Bowers et al . ( 2016 ) who failed to observe selective representations in fully connected NNs trained on stimuli one-at-a-time ( see Fig . 1 ) , but the measures of selectivity that have been applied across studies are different , and accordingly , it is difficult to directly compare results . A better understanding of the relation between selectivity measures is vital given that different measures are frequently used to address similar issues . For example , both the human interpretability of generated images ( Le , 2013 ) and localist selectivity ( Bowers et al. , 2014 ) have been used to make claims about ‘ grandmother cells ’ , but it is not clear whether they provide similar insights into unit selectivity . Similarly , based on their precision metric , Zhou et al . ( 2015 ) claim that the object detectors learned in CNNs play an important role in identifying specific objects , whereas Morcos et al . ( 2018 ) challenge this conclusion based on their finding that units with high CCMAS measures were not especially important in the performance of their CNNs and concluded : “ ... it implies that methods for understanding neural networks based on analyzing highly selective single units , or finding optimal inputs for single units , such as activation maximization ( Erhan et al. , 2009 ) may be misleading '' . This makes a direct comparison between selectivity measures all the more important . In order to directly compare and have a better understanding of the different selectivity measures we assessed ( 1 ) localist , ( 2 ) precision , and ( 3 ) CCMAS selectivity of the conv5 , fc6 , and fc7 of AlexNet trained on ImageNet , and in addition , we employed a range of signal detection methods on these units , namely , ( 4 ) recall with 100 % and 95 % precision , ( 5 ) maximum informedness , ( 6 ) specificity at maximum informedness , and ( 7 ) recall ( also called sensitivity ) at maximum informedness , and false alarm rates at maximum informedness ( described in Sec . 2 ) . We also assessed the selectivity of a few units in VGG-16 and GoogLeNet models trained on the ImageNet and Places-365 dataset that were highly selective according to the Network Dissection method ( Zhou et al. , 2018a ) . We show that the precision and CCMAS measures often provide misleadingly high estimates of object selectivity compared to other measures , and we do not find any units that can be reasonably described as ‘ object detectors ’ given that the most selective units show a low hit-rate or a high false-alarm rate ( or both ) when classifying images . At best , the most selective units in CNNs are sensitive to some unknown feature that is weakly associated with the class in question . In addition to these quantitative measures and jitterplots we assessed selectivity with a common qualitative measure , namely , human interpretation of images generated by a state-of-the-art activation maximization ( AM ) method ( Nguyen et al. , 2017 ) . AM images are generated to strongly activate individual units , and some of them are interpretable by humans ( e.g. , a generated image that looks like a lighthouse , see Fig . 1 ) . For the first time , we systematically evaluated the interpretability of the AM images and compare these ratings with the selectivity measures for corresponding units . We show that the few hidden units with interpretable AM images are not highly selective . 2 METHODS . Network and Dataset All ∼1.3M photos from the ImageNet ILSVRC 2012 dataset ( Deng et al. , 2009 ) were cropped to 277× 277 pixels and classified by the pre-trained AlexNet CNN ( Krizhevsky et al. , 2012 ) shipped with Caffe ( Jia et al. , 2014 ) , resulting in 721,536 correctly classified images . Once classified , the images are not re-cropped nor subject to any changes . We analyzed the fully connected ( fc ) layers : fc6 and fc7 ( 4096 units ) , and the top convolutional layer conv5 which has 256 filters . We only recorded the activations of correctly classified images . The activation files are stored in .h5 format and will be available at http : //anonymizedForReview . We randomly selected 233 conv5 , 2738 fc6 , 2239 fc7 units for analysis . Localist selectivity Following Bowers et al . ( 2014 ) , we define a unit to be localist for class A if the set of activations for class A was higher and disjoint with those of ¬A . Localist selectivity is easily depicted with jitterplots ( Berkeley et al. , 1995 ) in which a scatter plot for each unit is generated ( see Figs . 1 and 3 ) . Each point in a plot corresponds to a unit ’ s activation in response to a single image , and only correctly classified images are plotted . The level of activation is coded along the x-axis , and an arbitrary value is assigned to each point on the y-axis . Precision Precision refers to the proportion of items above some threshold from a given class . The precision method of finding object detectors involves identifying a small subset of images that most strongly activate a unit and then identifying the critical part of these images that are responsible for driving the unit . Zhou et al . ( 2015 ) took the 60 images that activated a unit the most strongly and asked independent raters to interpret the critical image patches ( e.g. , if 50 of the 60 images were labeled as ‘ lamp ’ , the unit would have a precision index of 50/60 or 83 % ; see Fig . 1 ) . Object detectors were defined as units with a precision score > 75 % : they reported multiple such detectors . Here , we approximate this approach by considering the 60 images that most strongly activate a given unit and assess the highest percentage of images from a given output class . CCMAS Morcos et al . ( 2018 ) introduced a selectivity index called the Class-conditional Mean Activation Selectivity ( CCMAS ) . The CCMAS for class A compares the mean activation of all images in class A , µA , with the mean activation of all images not in class A , µ¬A , and is given by : ( µA − µ¬A ) / ( µA + µ¬A ) . Here , we assessed class selectivity for the highest mean activation class . Activation Maximization We harnessed an activation maximization method called Plug & Play Generative Networks ( Nguyen et al. , 2017 ) in which an image generator network was used to generate images ( AM images ) that highly activate a unit in a target network . We used the public code released by Nguyen et al . ( 2017 ) and their default hyperparameters.1 We generated 100 separate images that maximally activated each unit in the conv5 , fc6 , and fc8 layers of AlexNet and asked participants to judge whether they could identify any repeating objects , animals , or places in images in a behavioral experiment ( Sec . 3.2 ) . Readers can test themselves at : https : //research.sc/ participant/login/dynamic/63907FB2-3CB9-45A9-B4AC-EFFD4C4A95D5 Recall with perfect and 95 % precision Recall with perfect and 95 % precision are related to localist selectivity except that they provide a continuous rather than discrete measure . For recall with perfect precision we identified the image that activated a given unit the most and counted the number of images from the same class that were more active than all images from all other classes . We then divided this result by the total number of correctly identified images from this class . A recall with a perfect precision score of 1 is equivalent to a localist representation . Recall with a 95 % precision allows 5 % false alarms . Maximum informedness Maximum informedness identifies the class and threshold where the highest proportion of images above the threshold and the lowest proportion of images below the threshold are from that class ( Powers , 2011 ) . The informedness is computed for each class at each threshold , with the highest value selected . Informedness summarises the diagnostic performance of unit for a given class at a certain threshold based on the recall [ True Positives / ( True Positives + False Negatives ) ] and specificity [ True Negatives / ( True Negatives + False Positives ) ] in the formula [ informedness = recall + specificity− 1 ] ( Powers , 2011 ) . Sensitivity or Recall at Maximum Informedness For the threshold and class selected by Maximum Informedness , recall ( or hit-rate ) is the proportion of items from the given class that are above the threshold . Also known as true postive rate . Specificity at Maximum Informedness For the threshold and class selected by Maximum Informedness , the proportion of items that are not from the given class that are below the threshold . Also known as true negative rate . False Alarm Rate at Maximum Informedness For the threshold and class selected by Maximum Informedness , the proportion of items that are not from the given class that are above the threshold . Network Dissection To assess the selectivity of a unit in the Network Dissection technique , Zhou et al . ( 2018a ) compute the Intersection over Union ( IoU ) of an annotated input image Lc , for the set of all ‘ concepts ’ c and a spatial activation map , Mk , of where a unit k is . A unit k is taken as a detector for concept c if its IoUk , c exceeds a pre-defined threshold T . See Zhou et al . ( 2018a ) for more details .

Summary: The paper makes an empirical claim that CNNs for object recognition do not contain hidden neuron which is highly selective to each class, mainly based on three aspects: (a) metrics related to the maximum informedness, (b) jitterplots of activation data, and (c) a user study assessing whether generated images maximizing a given unit is perceptible to the user. The paper point out these results are in contrast to the case of RNN, where it has been reported that many localist hidden units emerge. It is also noticed that the existing metrics for selectivity do not adequately discriminate highly selective units in CNN.