Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper describes a novel procedure (MILAN) to interpret deep learning models for computer vision by generating natural language description that specifies the activation selectivity of a given neuron in the model. For this aim, they first define an exemplar set of input image regions for each neuron by thresholding its activation value. Then they search a natural language description by optimizing the point-wise mutual information between descriptions and the exemplar set. The probability distributions for calculating the mutual information are approximated by training the SAT model and a two-layer LSTM language model on a newly collected dataset (MILANNOTATIONS), which includes annotations of 20k units labeled by human participants. The authors first test the generalizability of MILAN descriptions across different model architectures, datasets, and tasks, showing its privilege of generating higher agreement with human annotations compared to baseline methods. They then demonstrate three interesting applications of MILAN procedure and show how these natural language descriptions help us to understand and control the learned models.
main_review: - Strength
  - This paper is well-written and easy to follow. The authors provide sufficient technical details for readers to understand and reproduce their work. Data, code, and the trained model will be open source.
  - This paper picks up an intriguing topic that aims to interpret deep learning models by investigating hidden units and summarizing their exemplar activation by natural language descriptions. The proposed method is concise, straightforward, and well-motivated.
  - The natural language descriptions can capture categorical, relational, and logical structure across different levels in the learned features. It’s nice to see low-level features like edges (“the top boundaries of horizontal objects”), middle-level features like shapes (“Poles and legs”), and relatively high-level features like objects (“dog faces”) could all be generated quite well by this same model.
  - The results suggest generalizability across different model architectures, datasets, and tasks. This makes MILAN readily useful for many other potential applications, including the three interesting experiments shown in section 5-7.
- Comments
  - The model is trained on a newly collected dataset MILANNOTATIONS. Each unit was annotated by three human participants. But the inter-annotator agreement among human annotations seems not quite high (Table 4, Figure 10), compared to the BERTScore between model-generated descriptions and human annotations (Table 2, Table 3). How did the authors handle this inter-annotator inconsistency during their model training? Is there any additional quality control/validation performed for this MILANNOTATIONS dataset?
  - Following the first point, I wonder if the authors would consider scaling up their methods by leveraging the existing large, multimodal datasets like GQA/Visual Genome or visual-language model trained on large paired image-text datasets like CLIP/ALIGN?
  - Fig. 3 is not referenced in the main text. And I think these failure modes are interesting, and taking a closer look at them might be inspiring for improving this model in future studies. Do authors have further comments or thoughts about this result?
  - The results in Fig. 4 are quite interesting. The bar chart suggests low-level visual features are more described by adjectives, middle-level units need more prepositions and verbs to describe relational features, and high-level units need more complex composition of words thus resulting in longer length and deeper parser trees. This probably aligns well with our intuition and expectation. For units that may contribute to those "non-robust" model behavior, are they described by more nouns with higher max word diff? Will the proposed MILAN model be able to detect those "non-robust" units and edit the network to improve its performance?
- Minor:
  - What do different dots refer to in Fig. 5?

summary_of_the_review: Overall I think this work is well-motivated, technically sound, and showing promising results that support potential applications for interpreting and improving deep learning models for computer vision. Some minor changes could be made to improve the clarity. More details about how authors control / validate the quality of the MILANNOTATIONS dataset could be included.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors introduced MILAN, for mutual-information-guided linguistic annotation of neurons) that automatically labels neurons with open-ended, compositional, natural language descriptions. 
This is done by searching for descriptions that maximize point-wise mutual information with the image regions in which the neurons are active.

It uses (Bau, et. al. 2017) model for the selection of 15 images with regions, and Xu et. al. 2015 (Show-Attend-Tell) with a modification in pmi (probability of mutual information) for describing these regions. The Show-Attend-Tell model is trained on MILANANOTATIONS dataset, a large contribution of the technique. It is a dataset of images and fine-grained region descriptions. The dataset is comprised of 20K neurons (sets of regions) with descriptions.

TESTING:
The testing section is a large part of the contribution as well.
* Section 4: MILAN obtained higher agreement with human annotations on held-out networks than baseline. It also shows that the model works across architecture, dataset, and task
* Section 5: neurons captioned with many adjectives or prepositions are relatively important to model behavior
* Section 6: Models trained on blurred faces acquire neurons selective for blurred faces
* Section 7: Networks devotes substantial capacity to identifying text labels in images.

main_review: The paper is about visualization and explainability. It is good read and inspirational to me since I have been an advocate of the sub-field. To be able go deeper on making use of the intermediate stages of a network, for visibility as well as new AI product features. The summary above shows the short of the discoveries. I have to experiment with the technique myself to go further. However, it looks promising.

I also appreciate the amount of effort put on testing the system, on many architectures, datasets, and tasks. If there is more, I would like to have better characterization of the limits of the approach. 

Grammars, typos, etc.:
pg.1, par.1: convlution —> convolution

summary_of_the_review: * A higher level of explainability of the regions responsible for network final result
* A thorough analysis of the actionable insights that can be used for the model
* Limits of the approach needs to be discussed further
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This work proposes a framework for interpreting model behaviour by generating language descriptions of neurons in the model. The method is trained to maximize mutual information between the description and input examples that activate the neuron. The experiments show results for image classification, generation, and unsupervised representation learning. In particular, this method yields model description that achieve higher BERTScore than max-likelihood training when transferring to neurons in new architectures. Moreover the authors show how this technique can be used to compute correlations between descriptions and neuron importance (e.g. wrt classification accuracy), interpreting model behaviour (e.g. anonymized models still select for unblurred faces), and that one can remove spurious feature corrleations by removing models with specific descriptions.
main_review: I want to preface my review by stating that I am a domain expert in NLP and not CV. While I attempted a preliminary search for related work, I am not sure if there is prior work in specifically generating language descriptions of neurons (I understand that there is prior work in explaining neuron behaviour by examining inputs that activate it). Hence I may significantly modify my review based on the response of my peer reviewers.

This is a strong piece of work that clearly states its hypotheses and carefully designs experiments to test said hypotheses. The technical novel of the particular method is low, however I do not think that is the point of this work. What this paper does show is a novel way to interpret model behaviour, and allows for very useful downstream applications (e.g. fast filtering of neurons activated by a particular feature through text selection). I advocate for acceptance of this work, and propose some potential improvement. I do have some concerns about the scoping of this work. The title and description suggests a more general method, however experiments are purely based on images. I think the paper title should be more finely scoped (NL descriptions of Deep Image Features).

Major:
- Do these transfer results generalize to other tasks? or are they specific to image models? Or are they specific to the fact that all of these image models are trained/evaluated on (two) datasets that are similar to each other in terms of image distribution? I think perhaps the paper makes this technique seem more portable than it actually is. For example I would have to collect data for my task distribution in order for this technique to work. If you can show this working with pretrained zero-shot image captioning models than this result would be a lot more convincing.
- How do image captioning models do on the task of generating descriptions conditioned on inputs? The implicit hypothesis here is that they do not work well because they operate on higher levels of abstraction, but I would like to see empirical results of this by using caption models as baselines (e.g. zero-shot, fine-tuned).
- What do generated descriptions look like? How long are the descriptions? What is the vocabulary size? How diverse are the descriptions?
- What does the released dataset look like? Do you control for bias in the description? Diversity? Coverage?
- This objective not only needs to maximize over all input space but also all description space. The former is talked about but the latter is skimmed over (by stating that beam search is performed). It's unclear whether beam search would yield useful description if the description is more complex (e.g. in a language task like question-answering). 
- It would be nice to show spurious feature filtering results on something that is not a contrived task (e.g. classification with spurious text labels on top left corner of image) and instead on something impactful (e.g. removing racial/gender bias by filtering out neurons with corresponding descriptions).


Minor:
- typo; convlutional network

Questions:
- Why is there so much variance across architecture pairs? For example large gains on AlexNet -> Places and ResNet -> ImageNet but small gains on ResNet -> Places and AlexNet -> ImageNet.
summary_of_the_review: Strong paper demonstrating how to generation descriptions of image features and how to use method to analyze model behaviour and filter out neurons by description.
review_title: null

Meta-review: This paper presents a method to interpret neurons in the vision neural models by generating natural language description that specifies the activation selectivity of a given neuron. The proposed method first identifies an exemplar set of input image regions that corresponds to a neuron, then searches a natural language description by optimizing the point-wise mutual information between descriptions and the exemplar set.

Strength:
- Reasonable method design and clear writing
- Important problem and broad applications
- Extensive experiments for evaluation of the proposed method

Weakness:
- Need more discussion on the limitations of the proposed method
- Elaboration on the human inter-annotation agreement
- Analysis on method transferability across tasks.
Decision: Accept (Oral)