Venue: ICLR 2017 conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper attempts to understand and visualize what deep nets are representing as one ascends from low levels to high levels of the network.  As has been shown previously, lower levels are more local image feature based, whereas higher levels correspond to abstract properties such as object identity.  In semantic space, we find higher level nodes to be more semantically selective, whereas low level nodes are more diffuse.

This seems like a good attempt to tease apart deep net representations.  Perhaps the most important finding is that color figures prominently into all levels of the network, and that performance on gray scale images is significantly diminished.  The new NF measure proposed here is sensible, but still based on the images shown to the network.  What one really wants to know is what function these nodes are computing - i.e., out of the space of *all* possible images, which most activate a unit?  Of course this is a difficult problem, but it would be nice to see us getting closer to understanding the answer.  The color analysis here I think brings us a bit closer.  The semantic analysis is nice but I'm not sure what new insight we gain from this.  

review_title: Review of "indexing neuron selectivity"

final_score: rating: 3: Clear rejection
review_content: The authors analyze trained neural networks by quantifying the selectivity of individual neurons in the network for a variety of specific features, including color and category.   

Pros:
   * The paper is clearly written and has good figures. 
   * I think they executed their specific stated goal reasonably well technically.   E.g. the various indexes they use seem well-chosen for their purposes. 

Cons:
   * I must admit that I am biased against the whole enterprise of this paper.   I do not think it is well-motivated or provides any useful insight whatever.   What I view their having done is produced, and then summarized anecdotally, a catalog of piecemeal facts about a neural network without any larger reason to think these particular facts are important.  In a way, I feel like this paper suffers from the same problem that plagues a typical line of research in neurophysiology, in which a catalog of selectivity distributions of various neurons for various properties is produced -- full stop.  As if that were in and of itself important or useful information.   I do not feel that either the original neural version of that project, or this current model-based virtual electrophysiology, is that useful.   Why should we care about the distribution of color selectivities?   Why does knowing distribution as such constitute "understanding"?    To my mind it doesn't, at least not directly.   

Here's what they could have done to make a more useful investigation:
  
     (a) From a neuroscience point of view, they could have compared the properties that they measure in models to the same properties as measured in neurons the real brain.   If they could show that some models are better matches on these properties to the actual neural data than others, that would be a really interesting result.   That is is to say, the two isolated catalogs of selectivities (from model neurons and real neurons)  alone seem pretty pointless.  But if the correspondence between the two catalogs was made -- both in terms of where the model neurons and the real neurons were similar, and (especially importantly) where they were different --- that would be the beginning of nontrivial understanding.   Such results would also complement a growing body of literature that attempts to link CNNs to visual brain areas.  Finding good neural data is challenging, but whatever the result, the comparison would be interesting. 

and/or 

    (b) From an artificial intelligence point of view, they could have shown that their metrics are *prescriptive* constraints.   That is, suppose they had shown that the specific color and class selectivity indices that they compute, when imposed as a loss-function criterion on an untrained neural network, cause the network to develop useful filters and achieve significantly above-chance performance on the original task the networks were trained on.     This would be a really great result, because it would not only give us a priori reason to care about the specific property metrics they chose, but it would also help contribute to efforts to find unsupervised (or semi-supervised) learning procedures, since the metrics they compute can be estimated from comparatively small numbers of stimuli and/or high-level semantic labels.    To put this in perspective, imagine that they had actually tested the above hypothesis and found it to be false:  that is, that their metrics, when used as loss function constraints, do not improve performance noticeably above chance performance.  What would we then make of this whole investigation?  It would then be reasonable to think that the measured properties were essentially epiphenomenal and didn't contribute at all to the power of neural networks in solving perceptual tasks.  (The same could be said about neurophysiology experiments doing the same thing.)  
     [--> NB: I've actually tried things just like this myself over the years, and have found exactly this disappointing result.  Specifically,  I've found a number of high-level generic statistical property of DNNs that seem like they might potentially "interesting", e.g. because they apparently correlate with complexity or appear to illustrate difference between low, intermediate and high layers of DNNs.  Every single one of these, when imposed as optimization constraints, has basically lead nowhere on the challenging tasks (like ImageNet) that cause the DNNs to be interesting in the first place.  Basically, there is to my mind no evidence at this point that highly-summarized generic statistical distributions of selectivities, like those illustrated here, place any interesting constraints on filter weights at all.   Of course, I haven't tried the specific properties the authors highlight in these papers, so maybe there's something important there.]

I know that both of these asks are pretty hard, but I just don't know what else to say -- this work otherwise seems like a step backwards for what the community ought to be spending its time on. 
 
review_title: review of "UNDERSTANDING TRAINED CNNS BY INDEXING NEURON SELECTIVITY"

final_score: rating: 7: Good paper, accept
review_content: This paper makes three main methodological contributions:
 - definition of Neural Feature (NF) as the pixel average of the top N images that highly activation a neuron
 - ranking of neurons based on color selectivity
 - ranking of neurons based on class selectivity

The main weaknesses of the paper are that none of the methodological contributions are very significant, and no singularly significant result arises from the application of the methods.

However, the main strengths of the paper are its assortment of moderately-sized interesting conclusions about the basic behavior of neural nets. For example, a few are:
 - “Indexing on class selectivity neurons we found highly class selective neurons like digital-clock at conv2, cardoon at conv3 and ladybug at conv5, much before the fully connected layers.” As far as I know, this had not been previously reported.
 - Color selective neurons are found even in higher layers. (25% color selectivity in conv5)
 - “our main color axis emerge (black-white, blue-yellow, orange-cyan and cyan- magenta). Curiously, these two observations correlate with evidences in the human visual system (Shapley & Hawken (2011)).” Great observation!

Overall, I’d recommend the paper be accepted, because although it’s difficult to predict at this time, there’s a fair chance that one of the “smaller conclusions” would turn out to be important in hindsight a few years hence.


Other small comments:
 - The cite for “Learning to generate chairs…” is wrong (first two authors combined resulting in a confusing cite)

 - What exactly is the Color Selectivity Index computing? The Opponent Color Space isn’t well defined and it wasn’t previously familiar to me. Intuitively it seems to be selecting for units that respond to a constant color, but the highest color selectivity NF in Fig 5 i for a unit with two colors, not one. Finally, the very last unit (lowest color selectivity) is almost the same edge pattern, but with white -> black instead of blue -> orange. Why are these considered to be so drastically different? This should probably be more clearly described.

 - For the sake of argument, imagine a mushroom sensitive neuron in conv5 that fires highly for mushrooms of *any* color but not for anything else. If the dataset contains only red-capped mushrooms, would the color selectivity index for this neuron be high or low? If it is high, it’s somewhat misleading because the unit itself actually isn’t color selective; the dataset just happens only to have red mushrooms in it. (It’s a subtle point but worth considering and probably discussing in the paper)

review_title: Solid work leading to a few interesting conclusions

final_score: rating: null
review_content: Could you clarify how your proposed "Neural Feature" is different from the Zeiler & Fergus method of finding images that maximally activate a neuron?

review_title: Clarification of neural feature

final_score: rating: null
review_content: In Figure 1, what is the x-axis label? What is the y-axis label (i.e. what do the percents represent)? When computing the NF, Section 3 specifies “They [the activations] need to be accordingly ranked with the rest of the activations of the layer.” Does this mean that activations are compared from one neuron against other neurons in that same layer?
review_title: Details of NF

Meta-review: While this is interesting work, one major concern comes from Reviewer 2 regarding the attempt of characterizing the tuning properties, which has proven useless both in neuroscience and in machine learning. Currently, no attempt so far has lived up to the promises this line of research is aiming for. In summary, this work is explorative and incremental but worthwhile. We encourage the authors to further refine their research effort and resubmit.
Decision: Reject