Natural Language Descriptions of Deep Visual Features
Authors: Evan Hernandez
Sarah Schwettmann
David Bau
Teona Bagashvili
Antonio Torralba
Jacob Andreas
Venue: ICLR 2022 Oral (2022)

Abstract: Some neurons in deep networks specialize in recognizing highly specific perceptual, structural, or semantic features of inputs. In computer vision, techniques exist for identifying neurons that respond to individual concept categories like colors, textures, and object classes. But these techniques are limited in scope, labeling only a small subset of neurons and behaviors in any network. Is a richer characterization of neuron-level computation possible? We introduce a procedure (called MILAN, for mutual information-guided linguistic annotation of neurons) that automatically labels neurons with open-ended, compositional, natural language descriptions. Given a neuron, MILAN generates a description by searching for a natural language string that maximizes pointwise mutual information with the image regions in which the neuron is active. MILAN produces fine-grained descriptions that capture categorical, relational, and logical structure in learned features. These descriptions obtain high agreement with human-generated feature descriptions across a diverse set of model architectures and tasks, and can aid in understanding and controlling learned models. We highlight three applications of natural language neuron descriptions. First, we use MILAN for analysis, characterizing the distribution and importance of neurons selective for attribute, category, and relational information in vision models. Second, we use MILAN for auditing, surfacing neurons sensitive to human faces in datasets designed to obscure them. Finally, we use MILAN for editing, improving robustness in an image classifier by deleting neurons sensitive to text features spuriously correlated with class labels.

Published as a conference paper at ICLR 2022
NATURAL LANGUAGE DESCRIPTIONS OF DEEP VISUAL FEATURES
Evan Hernandez! Sarah Schwettmann'
David Bau
Teona Bagashvili*
Antonio Torralba'
Jacob Andreas'
'MIT CSAIL "Nottheastern University 'Allegheny College
{dez, schwett, teona, torralba, jda Hemit edu
d. bauumortheesterr..co
ABSTRACT
Some neurons deep networks specialize in recognizing highly specific percep- tual, structural, semantic features of inputs. In computer vision, techniques exist for identifying neurons that respond to individual concept categories like colors, textures, and object classes. But these techniques are limited in scope, la- beling only small subset of neurons and behaviors any network. characterization of neuron-level computation possible? We introduce procedure (called MILAN, for iww.aaliiiioooooonngnn -uided linguistic annotation of neurons) that automatically labels neurons with open-ended, compositional, natural lan- guage descriptions. Given a neuron, MILANN generates a description by searching for natural language string that maximizes pointwise mutual information with the image regions in which the neuron is active. MILAN produces fine-grained descriptions that capture categorical, relational, and logical structure in learned features. These descriptions obtain high agreement with geneantee feature descriptions across diverse set of model architectures and tasks, and can aid in understanding and controlling learned models. We highlight three applications of natural language neuron descriptions. First, we use MILAN for analysis, character- izing the distribution and importance of neurons selective for attribute, category. and relational information vision models. Second, we use MILAN for auditing. surfacing neurons sensitive to human faces in datasets designed to obscure them. Finally, we use MILAN for editing, improving robustness in an image classifier by deleting neurons sensitive text features spuriously correlated with class labels
richer
INTRODUCTION
surprising amount be learned about the behavior of deep network by understanding the indi- vidual neurons that make it up. Previous studies aimed at visualizing or automatically categorizing neurons have identified range of interpretable functions across models and application domains: low-level convolutional units in image classifiers implement color detectors and Gabor filters (Erhan) al.) 2009) while some later units activate for specific parts and object categories (Zeiler & Fer- gus 2014 Bau et al.) (017). Single neurons have also been found to encode sentiment in language data (Radford et al.) 2017) and biological function in computational chemistry (Preuer et al. 2019) Given new model trained to perform a new task, can we automatically catalog these behaviors?
Techniques for characterizing the behavior of individual neurons are still quite limited, Approaches based visualization (Zeiler & Fergus, 2014] Girshick et al.] 2014; Karpathy et al.) 2015] Ma- hendran & Vedaldi 20155 Olah et al. 2017) leave much of the work interpretation up to human users, and cannot be used for large-scale analysis. Existing automated labeling techniques (Bau et al. 2017 7 2019 Mu & Andreas 2020) require researchers to pre-define a fixed space of candidate neuron labels; they label only subset of neurons in given network and cannot be used to surface novel or unexpected behaviors.
This paper develops alternative paradigm for labeling neurons with expressive, compositional, and open-ended annotations in the form of natural language descriptions. We focus on the visual
Codee data, and interactive demonstration may be found http:httmmaaan....... http:///wiaaaaaaa....oo

---

Published as a conference paper at ICLR 2022
groape of cars Pl
p( oe of
and people
Engutt images
(c) Patch deseription model trained
max, pmiloh
plaid and checkered pattorns
MELAN (this work)
wwww
NetDissect (Bou et 2017)
similar patlerns
plabell| examplars)
grass around animals
MELAN (this work)
(0) Nearons in deep network
(d) Generated nearon libels
(b) Exumplar-bsse neuron representations
Figure I: (a) We aim to generate natural language descriptions of individual neurons in deep networks. (b)
We first represent each neuron via exemplar set of input regions that activate it. (c) In parallel, we collect dataset of fine-grained human descriptions of image regions, and use these train model of p'description exemplars) and p(description). (d) Using these models, we search for description that has high pointwise mutual information with the exemplars, ultimately generating highly specific neuron annotations.
domain: building on past work on theeooettionthhooettiic approaches to model interpretability, we formulate neuron labeling as a problem of finding informative descriptions of neuron's 's pattern of activation on input images. We describe a procedure (called MILAN, for mutual- inturaliiffrrmation guided linguistic annotation of neurons) that labels individual neurons with fine-grained natural language descriptions by searching for descriptions that maximize pointwise mutual information with the image regions in which neurons are active. To do so, we first collect new dataset of fine-grained image annotations (MILANNOTATTIONS, Figure [e), then use these to construct learned approximations the distributions over image regions (Figure [T)) and descriptions. In some cases, MILAN surfaces neuron descriptions that more specific than the underlying training data (Figure Itt..
MILAN is largely model-agnostic and can surface descriptions for different classes of neurons, rang- ing from convolutional units in CNNs to fully connected units in vision transformers, even when the target network is trained on data that differs systematically from MILANNOTATIONS' images. These descriptions can in turn serve a diverse set of practical goals in model interpretability and dataset design. Our experiments highlight three: using MILAN-generated descriptions to (1) analyze the role and importance of different neuron classes in convolutional image classifiers, (2) audit mod- els for demographically sensitive feature by comparing their features when trained on anonymized (blurred) and non-anonymized datasets, and (3) identify and mitigate the effects of spurious cor- relations with text features, improving classifier performance on adversarially distributed test sets. Taken together, these results show that fine- fine-grained, automatic annotation of deep network models both possible and practical: rich descriptions produced by automated annotation procedures can surface meaningful and actionable information about model behavior.
RELATED WORK
Interpreting deep networks MILAN builds on a long line of recent approaches aimed at explain- ing the behavior of deep networks by characterizing the function of individual neurons, either by visualizing the inputs they select for (Zeiler & Fergus, 2014; Girshick et al. 2014 Karpathy et al.
2013 Mahendran & Vedaldi 2013 Olah et al. 2017 7) or by automatically categorizing them accord- ing to the concepts they recognize (Bau et al. 2017 2018 Mu & Andreas) 2020; Morcos et al.) 2018 Dalvi et al., 2019). Past approaches to automatic neuron labeling require fixed, pre-defined label sets; in computer vision, this has limited exploration to pre-selected object classes, parts, ma- terials, and simple logical combinations of these concepts. While manual inspection of neurons has revealed that wider range of features play an important role in visual recognition (c.g. orientation, illumination, and spatial relations: Cammarata et al. (02T) MILAN is the first automated approach that can identify such features at scale. Discrete categorization is also possible for directions in rep- resentation space (Kim et al., 2018 Andreas et al. 2017, Schwettmann et al. 202I) and for clusters of images induced by visual representations (Laina al. 2020); in the latter, off-the-shelf im- age captioning model is used to obtain language descriptions of the unifying visual concept for the cluster, although the descriptions miss low-level visual commonalities. As MILAN requires only primitive procedure for generating model inputs maximally associated with the feature or direction of interest, future work might extend it these settings as well.
2

---

Published as a conference paper at ICLR 2022
Natural language explanations of decisions
Previous work aimed explaining computer vision
classifiers using natural language has focused on generating explanations for individual classification
decisions (e.g.. (ee,rickdss et al. 2016 Park et al.) 2018 Hendricks et al. 2018 Zellers al.) 2019).
Outside of computer vision, several recent papers have proposed procedures for generating natural language explanations of decisions in text classification models (Zaidan & Eisner 2008 Camburu itt al. 2018 Rajani et al. Naran et al. 2020) and of representations in more general sequence modeling problems (Andreas & Klein 2017). These approaches require task-specific datasets and often specialized training procedures, and do not assist with interpretability at the model level. To the best of our knowledge, MILAN is the first approach for generating compositional natural language descriptions for interpretability at the level of individual features rather than input-conditional deci- sions or representations. More fundamentally, MILAN can do so independently of the model being described, making it (as shown in Section modular, portable, and to a limited extent task-agnostic.
3 APPROACH
Neurons and exemplars Consider the neuron depicted in Figure Db. located in a convlutional network trained to classify scenes (Zhou et al. 2017). When the images in Figure aree provided as input to the network, the neuron activates in patches of grass near animals, but not in grass without animals nearby. How might we automate the process of automatically generating such description?
While the image regions depicted in Fig. Dp do not completely characterize the neuron's 's function in the broader network, past work has found that actionable information can be gleaned from such regions alone. [Bau et al. (2020) 2019) use them to identify neurons that can trigger class predictions or generative synthesis of specific objects: [Andreas & Klein (2017) them predict sequence outputs on novel inputs: Olah et al. (2018) and & Andreas (2020) use them to identify adversarial vulnerabilities. Thus, building on this past work, approach to neuron labeling also begins by representing each neuron via the set of input regions on which its activity exceeds a fixed threshold. Definition 1. Let f X + Y be neural network, and let fi(x)) denote the activation value of the ith neuron in f given an input x Then, an exemplar representation of the neuron f. is given by:
E. = (x E X f((z)= > 7 for some threshold parameter qi (discussed in more detail below).
Exemplars and descriptions Given this explicit representation of f.'s behavior, it remains to construct description d, of the neuron. Past work (Bau et al.. 2017: Andreas et al., 2017) begins with a fixed inventory of candidate descriptions (e.g. object categories), defines an exemplar set E for each such category (e.g. via the output of a semantic segmentation procedure) then labels neurons by optimizing d; == arg min, 8(E,, E2)) for some measure of set distance Jaccard 1912).
In this work, we instead adopt probabilistic approach neuron labeling. In computer vision appli- cations, each E, is a set of image patches. Humans are adept at describing such patches (Rashtchian ct al. 2010) and one straightforward possibility might be directly optimize d: := arg max, p(d E.). In practice, however, the distribution of human descriptions given images may not be well- aligned with the needs of model users. Fig. 2 includes examples of numan-generatee for exemplar sets. Many of them (e.g. text for AlexNet conv are accurate, but generic; in real- ity, the neuron responds specifically to text on screens. The generated description of a neuron should capture the specificity of its function-especiann
descriptions
relative to other neurons in the same model.
We thus adopt an information- theoreeic criterion for selecting descriptions: our final neuron deserip- tion procedure optimizes pointwise mutual information between descriptions and exemplar sets; Definition 2. The maxx-mttullinnormmet
description of the neuron Sa given by: MILAN(f.) := -arg max pmi(d; E.)= = arg max log p(d E.) p(d).
To turn Eq. 2 into practical procedure for annotating neurons, three additional steps are required: constructing a tractable approximation to the exemplar set E, (Section 3.I.. using naman--enneated image descriptions to model p(d E) and p(d) (Section 3.2 and Section 3.3). and finding a high- quality descríption d in the infinite space of natural language strings (Section 3.4).
II this paper, we will be primarily concered with neurons in convolutional layers; for each neuron, we will thus take the input space X to he the space of all image patches equal in size to the neuron's 's receptive field.
3

---

Published as conference paper at ICLR 2022
Generalization across architecture
Generalization across dataset
Generalization across task
AlexNel ResNet
ImageNet Places
CNN + GAN
ResNet 1aye2-445
AlieNNet comv4-25
BizGAN layert-26
Human: the off the line
Humun: colorffl balls and parts from pictures
Human: bouses built mountain cliff
MILAN: The top boundary horizontal objects
MILAN: colorful toys
MILAN: Rocks and stone walls
1ay3nt-1338
AlixNNet cunvi-163
BigGAN layer1-528
Human:
Jong, thin objects
Human: buildings and stairs
Human: keyhoards
MILAN: Long slender objects
MILAN: Objects ridges
MILAN: keyboards
BesNellAAlexhle
Places ImageNet
GAN CNN
AlexNit conv5-150
AlexNet convé-178
conva-222
Human: Striped animals and objects Repeating parallel
Human: Bird logs, bird beaks, and sticks Poles legs
Human: Icxt
Computer screen, screen
AlexNet conv5-202
AlexNer conv5-226
Human: Dog fnces and the top of cupeakes Dog
Humam: animal backs and driver's scats
Human: vertical bars organ
transportation
CNN ViT
layer5-57
DINO Iaver:233
Human: falling water MILAN: Splashes
Human: curved lines
Human: patches of für MILAN: Animal fur
Round edges objects
Figure 2: Examples of MILAN descriptions the generalization tasks described in Section Even highly specific labels (like the top boundaries of horizontal objects) can predicted neurons new networks. Failure modes include semantic errors, c.g. MILAN misses the cupcakes the dog faces and cupcakes neuron.
3.1 APPROXIMATING THE EXEMPLAR SET
As written, the exemplar set in Equation (T) captures neuron's 's behavior on all image patches. This set large (limited only by the precision used represent individual pixel values), so we follow past work (Bau et al. 2017) by restricting each E; to the set of images that cause the greatest activation in the neuron ft. For convolutional neurons in image processing tasks, sets E, ultimately comprise k images with activation masks indicating the regions of those images in which fi fired (Fig, I see Bau et al. 2017 for details). Throughout this paper, we use exemplar sets with & 15 images and choose 77 equal the 0.99 percentile of activations for the neuron f..
3.2 MODELING p(d) E) AND p(d)
The term pmi(d: d; E,) Equation [ can be expressed in terms of two distributions: the probability p(d E.) that human would describe an image region with d, and the probability p(d) that human would use the description for any neuron. p(d F.)) is, roughly speaking, a distribution over image captions (Donahue et al. (015).. Here, however, the input to the model is not a single image but a set of image regions (the masks Fig. Π we seek natural language descriptions of the common features of those regions. We approximate p(d E.) with learned model
the Show-Attend Tell image description model of Xu et al. (015) trained on the MILANNOTATIONS dataset described below, and with several modifications tailored to
case. We approximate
p(d) with a two-layer LSTM language model (Hochreiter & Schmidhuber, 1997) trained on the text of MILANNOTATIONS. Details about both models are provided in Appendix
4

---

Published as a conference paper at ICLR 2022
3.3 COLLECTING HUMAN ANNOTATIONS
As p(d | E,) and p(d) are both estimated using learned models, they require training data. In particular, modeling p(d Ei) requires dataset of captions that describe regions from multiple different images, such as the ones shown in Fig. I These descriptions must describe not only objects and actions, but all other details that individual neurons select for. Existing image captioning datasets, like MSCOCO (Lin et 20144) and Conceptual Captions (Sharma et al. 2018). only focus on scene-level details about a single image and do not provide suitable annotations for this task. We therefore collect novel dataset of captions for image regions to train the models underlying MILAN.
First, we must obtain a set of image regions to annotate. To ensure that these regions have a similar distribu- tion to the target neurons themselves, we derive them directly from the ex- emplar sets of neurons in set of seed models. We obtain the exemplar sets for a subset of the units each seed model in Table I using the method from Section B.I We then present each set human annotator and ask them to describe what common to the image regions.
Network
Arch.
Task
Annotated
Units
ImageNet
1152
AlexNet
CNN
Class.
Placex365
ceav. 1-5
1376
ResNet152
CNN
Class.
ImageNet
conv.
3904
Places365
res. 1-4
3904
ImageNet
3744
BigGAN
rel. 0-5
Places365
4992
MLP 1-12
DINO
VIT
BYOL
ImageNat
1200
100)
Table 1: Summary of MILANNOTATIONS, which labels 20k units across models with different network architectures, datsasets, and tasks. Each unit is annotated by three human participants.
total, we construct exemplar sets using neurons from seven vision models, totaling 20k neurons. These models include two archi- tectures for supervised image classification, AlexNet (Krizhessky et al. 2012) and ResNet152 (He) et al. 2015); one architecture for image generation, BigGAN (Brock et al. 2018); and one for supervised representation learning trained with a "Bootsrap Your Own Latent" (BYOL) objective (Chen & He| 2020 Grill et al. 2020), DINO (Caron al.) 2021). These models cover datasets, specifically ImageNet (Deng et al., 2009) and Places365 (Zhou et al., 2017). as well as two com- pletely different families of models, CNNs and Vision Transformers (ViT) (Dosovitskiy et al. [02I], Each exemplar set is shown to three distinct human participants, resulting 60k total annotations. Ex- amples are provided in Appendix A (Fig. T0). We recruit participants from Amazon Mechanical Turk. This data collection effort was approved by MIT's Committee on the Use of Humans as Ex- perimental Subjects. To control for quality, workers were required to have HIT acceptance rate of at least 95%, have at least 100 approved HIT's, and pass short qualification test. Full details about our data collection process and the collected data can be found in Appendix A
Table IaeeeSsuna summarizes the dataset, which we call MILANNOTATIONS.
3.4 SEARCHING IN THE SPACE OF DESCRIPTIONS
Directly decoding descriptions from pmi (d; E,) tends to generate disfluent descriptions. This is because the p(d) term inherently discourages common function words like the from appearing in descriptions. Past work language generation (Wang et al.] (020)) has found that this can be remedied by first introducing hyperparameter À to modulate the importance of p(d) when computing PMI, giving a new weighted PMI objective:
wpmi(d) = p(d] F.) - > log p(d).
Next, search is restricted to a set of captions that are high probability under p(d E,), which are reranked according to Eq. (3). Specifically, we run beam search on p(d E,). and use the full beam after the final search step as a set of candidate descriptions. For all experiments, we set X = .2 and beam size to 50.
4 DOES MILAN GENERALIZE?
Because it is trained on set of human-annotated exemplar sets obtained from a set of seed net- works, MILAN is useful as an automated procedure only generalizes and correctly describes neurons in trained models with new architectures, new datasets, and new training objectives. Thus, before describing applications of MILAN to specific interpretability problems, we perform cross-
5

---

Published as a conference paper at ICLR 2022
validation experiments within the MILANNOTA- TIONS data to validate that MILAN can reliably label new neurons. We additionally verify that MILAN provides benefits over other neuron annotation techniques by comparing its descriptions baselines: NetDissect (Bau et al.] (201)). which assigns a single concept label to each neuron by comparing the neuron's exemplars to semantic segmentations of the same images; Compositional Explanations (Mu & Andreas 2020), which follows a similar procedure to generate logical concept la- bels; and ordinary image captioning (selecting de- scriptions using p(d) E) instead of pmi(d; E)).
Model
CF
ND
pid
R)
pmild, R)
Imagghhhr
.01
.24
.34
.38
MlexNes-Placee
002
.11
33
.37
ResNet-Imag ImageNett
25
ResNet-Plaar Places
.03 .22
.30
.31
three
Table 2: BERTScores for neuron labeling meth- ods relative to human annotations. MILAAN oh- tains higher agreement than Compositional Expla- nations (CE) NetDissect (ND).
Generalization
Train + Test
BERTScore (f)
within network
AlexNet ImageNet AlexNet-Places
.39
ResNet152-Imagehh
.35
ResNet152- Plscces
.88
BigGAN ImageNet
.49
Method In each experiment, we train MILAN on a subset of ANNNTTNTTS and evaluate its per-
BigGAN Paccess
.22
Train
Test
formance on held-out subset. To compare MILAN to the baselines, we train on all data except single held-out network; we obtain the baseline labels by running the publicly available code with the default settings on the held-out network. To test general- ization within network, we train on 90% of neu- rons from each network and test on the remaining 10%. To test generalization across architectures, train on all AlexNet (ResNet) neurons and test on all ResNet (AlexNet) neurons; we also train all CNN neurons and test on ViT neurons. To test gen- eralization across datasets, we train on all neurons from models trained on ImageNet (Places) and test on neurons from models for the other datasets. To test generalization across tasks, we train on all clas- sifier neurons (GAN neurons) and test on all GAN neurons (classifier neurons). We measure perfor- mance via BERTScore (Zhang et al. (0200) relative to the human annotations. Hyperparameters for each of these experiments are Appendix C
noross arch.
AlexNet
ResNet152
.28
ResNeti52
AlexNet
.35
CNNs
VII
.34
across datasets
ImageNet
Places
30
Places
ImageNet
.33
across tasks
Classifiers
BigGAN
34
BigGAN
Classitiers
Table 3: BERTScores on held out neurons rela- tive to the human annotations. Each train/test split evaluates different kind of generalization, ul- timately evaluating how well MILAN generalizes to networks with architectures, datasets, and tasks unseen in the training annotations.
Results Table 2 shows results for MILAN and all three baselines applied to four different net- works. MILAN obtains higher agreement with human annotations on held-out networks than baselines. It is able to surface highly specific behaviors in its descriptions, like the splashes of water neuron shown in Figure 2 (splashes has no clear equivalent in the concept sets used by NetDissect (ND) or Compositional Explanations (CE)). MIL.AN also outperforms the ablated p(d F)) decoder, justifying the choice of pmi as an objective for obtaining specific and high-quality descriptions
Table 3 shows that MILAN exhibits different degrees of generalization across models, with gener- alization to new GAN neurons the same network easiest and GAN-to-classifier generalization hardest. MILAN
generalize to novel architectures. It correctly labels ViT neurons (in fully connected layers) as often as it correctly labels other convolutional units (c.g.. in AlexNet). We observe that transferability across tasks is asymmetric: agreement scores are higher when trans- ferring from classifier neurons to GAN neurons than the reverse. Finally, Figure 3 presents some of MILAN'S failure cases: when faced with new visual concepts, MILAN sometimes mislabels the concept (c.g.. by calling brass instruments noodle dishes), prefers a vague description (c.g., similar color patterns), or ignores the highlighted regions and describes the context instead.
We emphasize that this section is primarily intended as sanity check of the learned models underly- ing MILAN, and not as direct evidence of its usefulness or reliability as a tool for interpretability. We
'It may seem surprising that ND outperforms CE, even though ND can only output one-word labels. One reason is that ND obtains image segmentations from multiple segmentation models, which support a large vo- cabulary of concepts. By contrast, CF uses fixed dataset of segmentations and has smaller hase vocabulary. CE also tends generate complex formulas (with up to two logical connectives), which lowers its precision.

---

Published as a conference paper at ICLR 2022
follow Vaughan& & Wallach (2020) in arguing that the final test of any such tool must be its ability to produce actionable in- sights for human users, as in the three applications described below.
MILAN failures
AlexNet conv5-239
Human: yellow and green animals, food, instruments, objects
5 ANALYZING FEATURE IMPORTANCE
MILAN: Noodle dishes
The previous section shows that MILAN can generalize to new architectures, datasets, and tasks. The remainder of this paper focuses applications that use generated labels to understand how neurons influence model behavior. As a first example: descriptions in Figure reveal that neurons have different de- grees of specificity. Some neurons detect objects with spatial constraints (the area on top of the line), while others fire for low-level but highly specific perceptual qualities (long, thin objects). Still others detect perceptually similar but fundamen- tally different objects (dog faces and cupcakes). How impor- tant are these different classes of neurons model behavior?
BigGAN layer 1-86 1-86
Human: sea life
MILAN: Similar color patterns
DINO layer8-22
Human: the crowd
MILAN: Athletes
Method We use MILAN trained on all convolutional units in MILANNOTATIONS to annotate every neuron in ResNet 8- ImageNet. We then score each neuron according one of seven criteria that capture different syntactic or structural properties of the caption. Four syntactic criteria each count the number of times that a specific part of speech appears in caption: nouns, verbs, prepositions, and adjectives. Three structural criteria measure properties of the entire caption: its length, the depth of its parse tree (a rough measure of its compositional com- plexity, obtained from the spaCy parser of Honnibal et al. 2020), and its maximum word difference (a measure of the semantic coherence of the description, measured as the maximum Euclidean dis- tance between any two caption words, again obtained via spaCy). Finally, neurons are incrementally ablated in order of their score. The network is tested on the ImageNet validation set and its accuracy recorded. This procedure is then repeated, deleting 2% of neurons at each step. We also include five trials in which neurons are ordered randomly. Further details and examples of ablated neurons are provided in Appendix D
Figure 3: Examples of MILAN failures. Failure modes include incorrect gener- alization (top). vague descriptions for concepts not seen in the training set (middle), and mistaking the context for the highlighted regions (bottom).
Results Figure 4 plots accuracy on the ImageNet validation set as a function of the number of ab- lated neurons. Linguistic features of neuron descriptions highlight several important differences be- tween neurons. First, neurons captioned with many adjectives or prepositions (that is, neurons that capture attributes and relational features) relatively important to model behavior. Ab- lating these neurons causes a rapid decline in performance compared to ablating random neurons or nouns. Second, neurons that detect dissimilar concepts appear to be less important. When the
0.8
1.0
paree cap
0.8
propoRAce
rectureeryy
0.4
1xp 105 A1:
0.4
0.2
16
32
1.6
ablated
unita abluted
000v1
layect
1aye22
layert
Figure 4: ResNet18 accuracy on the ImageNet validation set as units ablated (left, middle), and distribution of neurons matching syntactic and structural criteria each layer (right). each configuration, neurons are scored according to property of their generated description (e.g.. number of nouns/words in description, etc.), sorted based on their score, and ablated in that order. Neurons described with adjectives appear crucial for good performance, while neurons described with very different words (measured by word embedding difference; max word diff.) appear less important for good performance. Adjective-selective neurons are most prevalent early layers, while neurons with large semantic differences are more prevalent late ones.
7

---

Published as a conference paper at ICLR 2022
caption contains highly dissimilar words (max word diff.), ablation hurts performance substantially less than ablating random neurons. Such neurons sometimes detect non-semantic compositions of concepts like the dog faces and cupcakes neuron shown in Fig. 2 Mu & Andreas (2020) find that these units contribute to non-robust model behavior. We reproduce their robustness experiments us- and reach similar conclusions. Finally, highhghhs
ing these neurons in (Fiiuree that neurons satisfying each criterion are not evenly distributed across layers- for example, middle layers contain the largest fraction of relation-selective neurons measured via prepositions.
6 AUDITING ANONYMIZED MODELS
One recent line of work in computer vision aims to construct privacy-aware datasets, e.g. by de- tecting and blurring all faces avoid leakage of information about specific individuals into trained models (Yang et al., 202T). But to what extent does this form of anonymization actually reduce models' reliance on images of humans? We wish to understand if mod- els trained on blurred data still construct features that can human faces,
or even specific categories of faces. A core function of tools for inter- pretable machine learning is to enable auditing of trained models for such behavior; here, we apply MILAN to investigate the effect of blurring- based dataset privacy.
Uuits
Method We use MILAN to caption a subset convolutional units in 12 different models pretrained for image classification on the blurred ImageNet images (blurred models). These models are distributed by the original authors of the blurred ImageNet dataset (Yang et al., (02TT. We caption the same units in models pretrained on regular ImageNet (un- blurred models) obtained from torchvision (Paszke al. 2019). We then manually inspect all neurons in the blurred and unblurred models for which MILAN descriptions contain the words face, head, nose, eyes, and mouth (using exemplar sets containing only unblurred images).
unlshumedd
blsared
Figure 5: Change in # of face neurons found by MI- LAN (cach pair of points is model architecture). Blurring reduces, but does not eliminate, units selec- tive for unblurred faces.
Across models trained on ordinary ImageNet, MILANN identified 213 neurons selective for human faces. Across models trained on blurred ImageNet, MILAN iden- tified 142 neurons selective for human faces. MILAN can distinguish between models trained on blurred and unblurred data (Fig. 5) However, it also reveals that models trained on blurred data acquire neurons selec- tive for unblurred faces. Indeed, it is possible to use MILAN'S labels to extract these face-selective neurons di- rectly. Doing so reveals that several of them not sim- ply face detectors, but appear to selectively identify fc- male faces (Fig. (6)) and Asian faces (Fig. Go). Blurring does not prevent models from extracting highly specific features for these attributes. Our results in this section highlight the use of MILAN for both quantitative and qual- itative, human- in-the loop auditing of model behavior.
Results
(b)
Faces of people
(c)
Huran Faces
Figure 6: (a) The blurred ImageNet dataset. (b-c) Exemplar sets and labels for two neu- rons in a blurred model that activate on un- blurred faces and appear to preferentially (but not exclusively) respond faces in spe- cific demographic categories.
7 EDITING SPURIOUS FEATURES
Spurious correlations between features and labels are a persistent problem in machine learning applications. especially in the presence of mismatches between training and testing data (Storkey) 2009). In object recognition, one frequent example is correlation between backgrounds and objects (e.g. cows are more likely to appear with green grass in the background, while fish are more likely to appear with blue background; Xiao et al. 2020). In more recent example, models trained on joint text and image data are subject to "text-based adversarial attacks", in which e.g. an apple with the word iPod written on it is classified as an iPod (Goh et al. 2021). Our final experiment shows that MILAN can be used to reduce models" sensitivity to these spurious features.
8

---

Published as a conference paper at ICLR 2022
Data We create controlled dataset imitating/Goh et al. (2021 's spurious text features. The dataset consists of 10 ImageNet classes. In the training split, there are 1000
images per class; 500 are annotated with (correct) text labels in the top-left corner. The test set contains 100 images per class (from the ImageNet validation set); in all these images, a random (usually incorrect) text label is included. We train and evaluate fresh ResNet18 8 model on this dataset, holding out 10% of the training data as a validation dataset for early stopping. Training details can be found in Appendix
(a) training dataset
(b) adversarial test dataset
1oyer3-134, "words and 1
truok
Method We use MILAN to obtain descriptions of ev- ery residual neuron in the model as well as the first convolutional layer. We identify all neurons whose de- scription contains text, word, or letter. To identify spu- rious neurons, we first assign each text neuron an in- dependent importance score by removing it from the network and measuring the resulting drop in valida- tion accuracy (with non-adversarial images). We then sort neurons by importance score (with the least impor- tant first), and successively ablate them from the model.
(c) text neuron
Figure 7: Network editing. (a) We train image classifier on synthetic dataset in which half the images include the class label written in text in the corner
(b) We eval- uate the classifier on an adversarial test set, in which every image has random textual label. (c) Nearly third of neurons the trained model model detect text, hurting its performance on the test set.
Results
The result of this procedure on adversarial test
0.72
accuracy is shown in Fig. 8 Training on the spurious data
0.70
substantially reduces ResNet18's performance on the ad- versarial test set: the model achieves 58.8% accuracy, opposed to 69.9% when tested on non-spurious data. MI- LAN identifies 300 text-related convolutional units (out of 1024 examined) in the model, confirming that the model has indeed devoted substantial capacity to identifying text labels in the image. Figure shows an example neurons specifically selective for airline and truck text. By delet- ing only 13 such neurons, test accuracy is improved by 4.9% (a 12% reduction in overall error rate) Fl This in- crease cannot be explained by the sorting procedure de- scribed above: if instead we sort all neurons according to validation accuracy (orange line), accuracy improves by less than 1%. Thus, while this experiment does not com- pletely eliminate the model's reliance on text features, it shows that MILAN's predictions enable direct editing of networks to partially mitigate sensitivity to spuri- ous feature correlations.
0.68
0.66
0.02
0.60
0.58
0.56
sort text
0.54
sort all
0.52
Initial performance
0.50
taxt distractors
0.48
Figure 8: ResNet18 accuracy on the adver- sarial test set as neurons are incrementally ablated. Neurons are sorted by the model's validation accuracy when that single neuron is ablated, then ablated in that order. When ablating neurons that select for the spurious text, the accuracy improves by 4.9 points. When zeroing arbitrary neurons, accuracy still improves, but by much less.
8 CONCLUSIONS
We have presented MILAN, approach for automatically labeling neurons with natural language descriptions of their behavior. MILAN selects these descriptions by maximizing pointwise mutual information with image regions in which each neuron is active. These mutual information esti- mates are in turn produced by a pair of learned models trained on ANNOTATIO,,, a dataset of fine-grained image annotations released with this paper. Descriptions generated by MILANN surface diverse aspects of model behavior, and can serve as a foundation for numerous analysis, auditing, and editing techniques workflows for users of deep network models.
'Stopping criteria are discussed more in Appendix E. if no adversarial data is used to determine the number of neurons prune, an improvement of 3.1 % is still achievable.

---

Published as a conference paper at ICLR 2022
IMPACT STATEMENT
In contrast to most past work on neuron labeling, MILAN generates neuron labels using another black-box learned model trained on human annotations of visual concepts. With this increase in expressive power come a number of potential limitations: exemplar-based explanations have known shortcomings (Bolukbasi et al. 202T), human annotations of exemplar sets may be noisy, and the captioning model may itself behave in unexpected ways far outside the training domain. The MI- LANNOTATIONS dataset was collected with annotator tests to address potential data quality issues, and our evaluation in Section 4 characterizes prediction quality on new networks; we nevertheless emphasize that these descriptions are partial and potentially noisy characterizations of neuron func- tion via their behavior a fixed-sized set of representative inputs. MILAN complements, rather than replaces, both formal verification (Dathathri et al., 2020) and careful review predictions and datasets by expert humans (Gebru et al. 2018: Mitchell et al. 2019).
ACKNOWLEDGMENTS
We thank Ekin Akyürek and Tianxing He for helpful feedback early drafts of the paper. We also thank IBM for the donation of the Satori supercomputer that enabled training BigGAN on MIT Places. This work initiative at MIT, a Sony Faculty Innovation Award, DARPA SAIL-ON BRR0011-2-C-000 and a hardware gift from NVIDIA under the NVAIL grant program.
partially supported by the MIT-IBM Watson AI lab, the ThatLLemm
REFERENCES
Jacob Andreas and Dan Klein. Analogs of linguistic structure in deep representations. In Proceed- ings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp- 2893- 2897, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10.18653/v 1/D77-131
URL https: /wwwp. awwwwa..ww.ccc..cc.. wtw.:e/b.rr///mtooooo/////111111
Jacob Andreas, Anca D Dragan, and Dan Klein. Translating neuralese. In ACL (1), 2017.
Dzmitry Bahdanau, Kyung Hyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In ICLR, January 2015.
Anthony Bau, Yonatan Belinkov, Hassan Sajjad, Nadir Durrani, Fahim Dalvi, and James Glass. Identifying and controlling important neurons in neural machine translation. In International Conference on Learning Representations, 2018.
David Bau, Bolci Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection: Quantifying interpretability of deep visual representations. In Computer Vision and Pattern Recognition (CVPR), 2017.
David Bau, Jun-Yan Yan Zhu, Hendrik Strobelt, Bolei Zhou, Joshua B Tenenbaum, William Freeman, and Antonio Torralba. Gan dissection: Visualizing and understanding generative adversarial net- works. In International Conference on Learning Representations (ICLR), 2019.
David Bau, Lun-Yan Zhu, Hendrik Strobelt, Agata Lapedriza, Bolei Zhou, and Antonio Torralba. Understanding the role individual units deep neural network. Proceedings of the National Academy of Sciences (PNAS), 2020.
Tolga Bolukbasi, Adam Pearce, Ann Yuan, Andy Coenen, Emily Reif, Fermanda Viégas, and Martin Wattenberg. An interpretability illusion for bert. arXiv preprint 2eXiv:2204.00 0433, 2021.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. In International Conference Learning Representations, 2018.
Oana-Maria Camburu, Tim Rocktäschel, Thomas Lukasiewicz, and Phil Blunsom. e-snli: Natural language inference with natural language explanations. arXiv preprint ///iv: 193, 2018.
Nick Cammarata, Gabriel Goh, Shan Carter, Chelsea Voss, Ludwig Schubert, and Chris Olah. Curve circuits. Distill, 601 6(1):e000240006 2021.
10

---

Published as a conference paper at ICLR 2022
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties self-supervised vision transformers. Proceedings of the International Conference Computer Viston (ICCV), 2021.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning, 2020.
Fahim Dalvi, Nadir Durrani, Hassan Sajjad, Yonatan Belinkov, Anthony Bau, and James Glass. What is one grain of sand in the desert? analyzing individual neurons in deep nlp models. In Proceedings of AAAI, 2019.
Sumanth Dathathri, Krishnamurthy Dvijotham, Alexey Kurakin, Aditi Raghunathan, Jonathan Ue- sato, Rudy Bunel, Shreya Shankar, Jacob Steinhardt, Ian Goodfellow, Percy Liang, et al. Enabling certification of verification-agnosti networks via memory-efficient semidefinite programming. In Neural Information Processing Systems (NeurIPS), 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Computer Vision and Pattern Recognition (CVPR), 2009.
Jeffrey Donahue, Lisa Anne Hendricks, Sergio Guadarrama, Marcus Rohrbach, Subhashini Venu- gopalan, Kate Saenko, and Trevor Darrell. Long-term recurrent convolutional networks for visual recognition and description. In Proceedings of the IEEE conference on computer vision and pat- tern recognition, pp. 2625-2634, 2015.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An im- age is worth 16x16 words: Transformers for image recognition at scale. International Confer- on Learning Representations (ICLR), 2021.
Dumitru Erhan, Yoshua Bengio, Aaron Courville, and Pascal Vincent. Visualizing higher-layer features of a deep network. 2009.
Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, and Kate Crawford. Datasheets for datasets. arXiv preprint arXiv: 1803.000300000 000000000, 2018.
Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for ac- curate object detection and semantic segmentation. In computer vision and pattern recognition (CVPR), pp. 580-587, 2014.
Gabriel Goh, Nick Cammarata, Chelsea Voss, Shan Carter, Michael Petrov, Ludwig Schubert, Alec Radford, and Chris Olah. Multimodal neurons artificial neural networks. Distill, 2021.
Jcan-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Ghesh- laghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent: A approach to self-supervised learning, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition, 2015.
Lisa Anne Hendricks, Zeynep Akata, Marcus Rohrbach, Jeff Donahue, Bernt Schiele, and Trevor Darrell. Generating visual explanations. In European conference on computer vision, pp. 3-19. Springer, 2016.
Lisa Anne Hendricks, Ronghang Hu, Trevor Darrell, and Zeynep Akata, Grounding visual expla- nations. In Proceedings of the European Conference Computer Vision (ECCV), pp. 264-279, 2018.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. In Neural computation, 1997.
Matthew Honnibal, Ines Montani, Sofie Van Landeghem, and Adriane Boyd. spaCy: Industrial- strength Natural Language Processing in Python, 2020. URL 52811 5281/ Izenooo. 1212303
11

---

Published as a conference paper at ICLR 2022
Paul Jaccard. The distribution of the flora in the alpine zone. New Phytologist, 11(2):37-50, 1912.
Andrej Karpathy, Justin Johnson, and Li Fei-Fei. Visualizing and understanding recurrent networks. arXiv preprini arXiv: 1506.000767 2015.
Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International conference on machine learning (ICMI), 2018.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep con- volutional neural networks. In Advances in Neural Information Processing Systems (NeurIPS), 2012.
Iro Laina, Ruth C. Fong, and Andrea Vedaldi. Quantifying learnability and describability of visual concepts emerging in representation learning. Advances in Neural Information Processing Sys- tems, 1020-December, 2020. ISSN 1049-5258. Funding Information: We would like to thank Yuki Asano and Christian Rupprecht for helpful discussions and for their feedback on this work. We are also grateful for the EPSRC programme grant Seebibyte EP/M013774/1 (LL..,, ERC start- ing grant IDIU 638009 (I.L), and Open Philanthropy Project (R.F.). Publisher Copyright: @ 2020 Neural information processing systems foundation. All rights reserved.; 34th Conference on Neu- ral Information Processing Systems, NeurIPS 2020 Conference date: 06- 06-122-2020 Through 12- 12-2020.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars (eds.), Computer Vision ECCV 2014, pp. 740-755, Cham, 2014. Springer International Publishing. ISBN 978-3-319-10602-1.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019.
Aravindh Mahendran and Andrea Vedaldi. Understanding deep image representations by inverting them. In computer vision and pattern recognition (CVPR), 2015.
Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. Model cards for model reporting. In Proceedings of the conference on fairness. accountability, and transparency, pp. 220-229, 2019.
Ari Morcos, David GT Barrett, Neil C Rabinowitz, and Matthew Botvinick. On the importance of single directions for generalization. In International Conference on Learning Representations (ICLR), 2018
Jesse Mu and Jacob Andreas. Compositional explanations of neurons. In Advances in Neural Information Processing Systems, 2020.
Sharan Narang, Colin Raffel, Katherine Lee, Adam Roberts, Noah Fiedel, and Karishma Malkan. WT5?! Training text-to-text models to explain their predictions. arXiv preprint N0t::20044145466, 2020.
Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Feature visualization. In Distill, 2017.
Chris Olah, Arvind Satyanarayan, Ian Johnson, Shan Carter, Ludwig Schubert, Katherine Ye, and Alexander Mordvintsev. The building blocks of interpretability. In Distill, 2018
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311-318, 2002.
Dong Huk Park, Lisa Anne Hendricks, Zeynep Akata, Anna Rohrbach, Bernt Schiele, Trevor Dar- rell, and Mareus Rohrbach. Multimodal explanations: Justifying decisions and pointing to the evidence. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8779-8788, 2018
12

---

Published as a conference paper at ICLR 2022
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf. Ed- ward Yang. Zachary Vitooo Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché- Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 8024-8035, Curran Associates, Inc., 2019. URL http::/ nwwrips www.iie....c.pppapaaeeeeeeeeeeee pc/paper/ 9015-pytorch- an imperative
pdf
Kristina Preuer, Günter Klambauer, Friedrich Rippmann. Sepp Hochreiter, and Thomas Unterthiner. Interpretable deep learning in drug discovery. In Explainable Al: Interpreting, Explaining and Visualizing Deep Learning, pp. 331-345. Springer, 2019.
Alec Radford, Rafal Jozefowicz, and llya Sutskever. Learning to generate reviews and discovering sentiment. arXiv preprint arXiv: /ttiv:/70..000 770.00444. 2017.
Nazneen Fatema Rajani, Bryan McCann, Caiming Xiong, and Richard Socher. Explain yourself! leveraging language models for commonsense reasoning. arXiv preprint arXiv:/906,00000 /906,02361, 2019.
Cyrus Rashtchian, Peter Young, Micah Hodosh, and Julia Hockenmaier. Collecting image anno- tations using amazon's 's mechanical turk. In Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon's 's Mechanical Turk, pp. 200,
Sarah Schwettmann, Evan Hernandez, David Bau, Samuel Klein, Jacob Andreas, and Antonio Tor- ralba. Toward a visual concept vocabulary for gan latent space. International Conference on Computer Vision, 2021.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypemymed, image alt-text dataset for automatic image captioning. In Proceedings of ACI,, 2018 88
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations (ICLR), 2015.
Amos Storkey. When training and test sets are different: characterizing learning transfer. Dataset shift in machine learning, 30:3-28, 2009.
Jennifer Wortman Vaughan and Hanna Wallach. A human-centered agenda for intelligible machine learning. Machines Trust: Getting Along with Artificial Intelligence, 2020.
Zeyu Wang, Berthy Feng, Karthik Narasimhan, and Olga Russakovsky. Towards unique and infor-
mative captioning of images. In European Conference on Computer Vision (ECCV), 2020.
Kai Xiao, Logan Engstrom, Andrew Ilyas, and Alcksander Madry. Noise or signal: The role of image backgrounds in object recognition. arXiv preprint vrXiv:2006.0999) 2020.
Kelvin Xu, Jimmy Lei Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville. Ruslan Salakhutdinov, Richard S. Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption genera- tion with visual attention. Proceedings of the 32nd International Conference on International Conference on Machine Learning Volume 37, ICML, 15, pp. 2048-2242 2048--007..0 JMLR.org, 2015.
Kaiyu Yang, Jacqueline Yau, Li Fei-Fei, Jia Deng, and Olga Russakovsky. A study of face obfusca- tion in imagenet. arXiv preprint arXiv:2103.06191, 2021.
Omar Zaidan and Jason Eisner. Modeling annotators: A generative approach to learning from annotator rationales. In Proceedings of the 2008 conference on Empirical methods in natural language processing, pp. 31-40, 2008.
Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. ECCV, 2014.
Rowan Zellers, Yonatan Bisk. Ali Farhadi, and Yejin Choi. From recognition to cognition: Visual commonsense reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pp. 6720-6731, 2019.
13

---

Published as a conference paper at ICLR 2022
Bo torighighin righte reggis Toyp Borond layang
Detailed image summarizor test
withir prrnpe you surnery zur sppies
Madel of peocs
peoole eeing
pecdec mater
(a) qualification test
annotation form
Figure 9: Screenshots of the Amazon Mechanical Turk forms used to collect the CaNCAn dataset. (a) The qualification test. Workers asked pick the best description for two hand-chosen neurons from model not included our corpus. (b) annotation form. Workers are shown the top-15 higheet--attivatin images for a neuron and asked to describe what is common to them in one sentence.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. Bertscore: Evalu- ating text generation with bert. In International Conference Learning Representutions, 2020. URL htttps://oppereee/
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE transactions on pattern analysis and machine intelligence, 2017.
Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision, 127(3)3302-321, 2019.
A MILANNOTATIONS
We recruited annotators from Amazon Mechanical Turk to describe one neuron at a time given its top-activating images. screenshot of the template shown Figure Tb. Participants were given the instructions:
one sentence, summarize everything shown inside the high- lighted regions in the images. They might all show the same thing, or they might show several different things.
Instructions:
In your answer, DO NOT mention that you are describing highlighted regions images.
Workers were given up to an hour to complete each annotation, but early trials revealed they required about 30 seconds per HIT. We paid workers $0.08 per annotation, which at $9.60 per hour exceeds the United States federal minimum wage.
14

---

Published as a conference paper at ICLR 2022
Human annotation examples
Reshe--laggNa 1a35s--1335
ManNatelmoocNii 3-660
ReNNcePasee lived- 70
chuins, poles and animads long, than objects bvown smakes
objects that pairs the sdge of two rotund thangs wext to eoch other two of object
plaist shirts and frees stropes and plaid shirts pluid shirts, bushes
other
St convi-2112
AloxNct-lmngeedd 0oev3-290
Alediet Places 20002-160
the logs of animaals bieds and antmals
whishees antimats and flowers with thin loues
rod objects
brighht and yellow ofjects brightt glowing arens that include yellow
apper portion of animal legs
lines eadiating oud from common anclens
BigGAN ImageNet layeri-170
HigGAN-Placs layert-702
faoes of people
corvlages and
top of builkling or bus bus casligs
bottom lamman fooes chins
andiga curr
variations of bus and train oistes
DINO-Imoeeel layertas
DING- ImageNet bayerk-50
DINO-ImapaNa laver-122
alf volicle hecollighhs mucks headllgghs
rusty metaat metal Books rusty objeccs
peatoic torl feathers
Figure 10: Example human annotations for exemplars in MILANNOTATIONS. which contains annota- tions neurons in seven networks. Each set of images annotated by three distinct human participants.
To control for quality, we required workers to pass short qualifi- cation test in which they had to choose the most descriptive cap- tion for two manually chosen neurons from VGG-16 (Simonyan Zisserman 2015) trained on ImageNet (not included as part of MI- shown Figure Tp.
Model
AlexNet
ImageNet
.25
Places365
.27
ResNet152
ImageNet
LANNOTATIONS). screenshot this
Placex365
.17
Table 4 shows the inter-annotane agreement of neuron annotations for each model, and Table 5S shows some corpus statistics broken down by model and layer. Layers closest to the image (early lay- ers in CNNs and later layers GANs) are generally described with more adjectives than other layers, while annotations for layers far- ther from the image include more nouns, perhaps highlighting the low-level perceptual role of the former and the scene- and object- centric behavior of the latter. Layers farther from the image tend have longer descriptions (e.g. in BigGAN-ImageNet, AlexNet- ImageNet), but this trend is not consistent across all models (e.g. in models trained on Places365, the middle layers have the longest average caption length).
BigGAN
InageNet Places365
.26
.24
DINO
ImageNet
.23
Table 4: Average inter-annotator agreement among human annota- tions, measured in BorrS Some models have clearer neu- ron exemplars than others.
B MILAN IMPLEMENTATION DETAILS
B.1 IMPLEMENTING p(d) F.)
We build the Show, Attend, and Tell (SAT) model for describing images (Xu et al. 2015). SAT designed for describing the high-level content a single images, so we must make several modi- fications support use case, where our goal is describe sely of regions in images.
15

---

Published as a conference paper at ICLR 2022
Model
Layer
# Units
# Words
Len.
% Noun
% Adj
% Prep
AlexNet-ImageNet
conv1
64
185
4.8
37.5
24.3
12.2
conv2
192
384
5.5
37.8
19.4
13.2
conv3
384
661
5.3
41.0
16.4
13.0
conv4
256
608
5.5
43.1
11.9
12.5
conv5
256
693
5.5
46.0
10.2
10.4
AlexNet-Places365
conv1
96
153
4.3
38.4
26.8
12.7
conv2
256
297
4.8
37.8
26.0
12.7
conv3
384
412
4.7
40.2
24.8
10.5
conv4
384
483
4.4
43.7
19.9
10.3
conv5
256
486
4.1
45.8
17.6
10.6
ResNer152-ImageNe.
conv1
64
285
4.7
43.8
11.8
10.3
layer1
256
653
5.5
43.1
10.5
12.5
layer2
512
936
5.1
44.0
12.7
12.6
layer3
1024
1222
4.2
49.6
10.9
11.3
layer4
2048
1728
4.6
47.8
8.6
7.8
ResNet152-Places365
convl
64
283
5.2
47.3
11.1
14.6
layerl
256
633
5.3
46.3
9.4
13.3
layer2
512
986
5.8
46.0
8.3
13.8
layer3
1024
1389
4.8
48.2
6.7
12.7
layer4
2048
1970
5.3
46.3
5.5
11.9
BigGAN--mageNet
layer()
1536
1147
3.9
52.4
7.8
8.2
layerl
768
853
3.5
53.0
9.4
8.9
layer2
768
618
3.2
52.6
12.3
9.5
layer3
384
495
3.7
49.9
14.3
10.9
layer4
192
269
3.3
47.9
18.0
13,4
layer5
69
2.6
53.6
22.8
14.6
BigGAN-Places365
layerO
2048
1062
4.2
53.3
5,4
8.3
layerl
1024
708
3.9
55.0
6.1
11.5
layer2
1024
410
4.6
52.7
8.1
16.3
layer3
512
273
5.2
50.4
7.6
15.0
layer4
256
192
4.6
47.5
9.3
14.9
layer5
128
123
4.2
46.7
13.5
13.0
Diag-lma
layer0
100
320
4.4
45.7
12.7
4.8
layerl
100
321
4.2
49.8
9.1
6.8
layer2
100
285
3.9
53.3
6.2
7.5
layer3
100
312
3,9
54.4
6.2
7.1
layer4
100
304
3.9
53.5
4.4
7.0
layer5
100
287
3.5
55.1
5.5
5.2
layeró
100
377
3.9
51.3
8.2
5,4
layer7
100
374
3.8
52.0
6.4
6.2
layer8
100
330
3.4
53.0
7.0
8.8
layer9
100
350
3.1
56.1
6.3
9,6
layer10
100
369
3,9
50.3
9.3
8.2
layerll
100
294
3.3
52.4
7.5
9.4
Total
20272
4597
4.5
48.7
9.4
10.9
Table 5: Corpus statistics for MILANNOTATIONS descriptions broken down by model and layer. The # Words column reports the number of unique words used across all layer annotations. the Len. column reports the average number of words in each caption for that layer, and the % columns report the percentage of all words across all captions for that layer that are a specific part speech.
16

---

Published as a conference paper at ICLR 2022
attention
katures
LSTM
mask
building and dome.
image
Neuron captioning model. Given the set of top-activating images for a neuron and masks for the regions of greatest activation, we extract features maps from cach convolutional layer of a pretrained image classifier. We then downsample the masks and them to pool the features before concatenating them into single feature vector per image. These feature vectors are used as input the decoder attention mechanism.
Figure 11:
In the original SAT architecture, a single input image is first converted to visual features by passing it through an encoder network y, typically an image classifier pretrained on large dataset. The output of the last convolutional layer extracted as a matrix visual features:
[11; u2;
These visual features are passed to a decoder LSTM whose hidden state is initialized as function of the mean of the visual features = 1/k vg.. At each time step, the decoder attends over the fea- tures using an additive attention mechanism (Bahdanau et al. 2015). then consumes the attenuated visual features and previous token as input to predict the next token.
The SAT architecture makes few assumptions about the structure of the visual features. We will take advantage of this generality and modify how e is constructed to support our task, leaving the decoder architecture intact.
Now, instead single image x, the model inputs are the k top-activating images Vÿ for a neuron well mask m, for each image that highlights the regions of greatest activation. Our task is to describe what the neuron detecting, based strictly on the highlighted regions of the Ty. In support of this, the visual features must (1) include information about all k images, (2) encode multiple resolutions of the images to capture both low-level perceptual and high-level scene details about the image, and (3) pay most (but not exclusive) attention to the regions of greatest activation in the image.
Describing sets of images The l features SAT correspond different spatial localities of single image. In our architecture, each feature by corresponds to one input image Zy-
Encoding multiple resolutions Instead of encoding the image with just the convolutional layer of g. we use pooled convolutional features from every layer. Formally, let gr())) denote the output of layer in the pretrained image encoder with L layers, and pool denote pooling function that uses the mask to pool the features (described further below). The feature vector for the jth image xy is:
V = (ool(m).1xx;)))) ...:: pool(m,,g1 ge (z()))
Highlighting regions of greatest activation Each of the top-activating images z, that hand to our model comes with a mask my highlighting the image regions of greatest activation. We incorporate these masks into the pooling function pool from above. Specifically, ple the mask my to the same spatial shape as ge (x,,) using bilinear interpolation, which we denote upsample(m, We then apply the mask to each channel at layer L, written (x,) via element- wise multiplication (0) with upsample(m,). Finally, we sum spatially along each channel, resulting length vector. Formally:
first downsam-
(o)2, = I vec(upsample(m:) @ gr,c(z;))
Each v is thus length Ee Cs vector, where C7 is the number of channels layer of g.
17

---

Published as a conference paper at ICLR 2022
Gen.
Train Test
Units
# Words
Len.
% Noun
% Adj
% Prep
within netwak
AlexNet- AlexNet-ImageNer
115
100
3.5
45.7
16.4
11.9
AlexNet- Places
137
46
2.5
49.3
28.7
9.6
ResNet- ImageNet
390
ResNet-Places
390
376
4.3
46.5
8.7
10.9
BigGAN--magoNet
374
112
2.2
59.8
17.5
10.4
BigGAN Places
499
245
3.8
54.2
9.0
Train
Test
across arch.
AlexNet
RosNet
7808
326
3.0
46.1
21.0
8.9
ResNet
AlexNet
2528
275
2.7
48.0
27.1
6,4
CNNs
VIT
1200
200
2.6
55.0
18.2
13.0
across
ImageNet
10272
Places
ImageNet
8800
309
3.1
47.8
26,9
7,8
task
Classifiers BigGAN
BigGAN
8736
202
2.1
53.0
25.3
6.1
Classifiers
10336
336
3.2
54.3
14.2
16.8
Total
51585
1002
2.7
51.9
19.8
11.1
Table 6: Statistics for MILAN generated descriptions on the held-out neurons from the generalization experi- ments of Section 4 Columns are the same in Table
Throughout our experiments, g is ResNet101 pretrained for image classification on ImageNet, provided by PyTorch Paszke et al. (2019). We extract visual features from the first convolutional layer and all four residual layers. We do not fine tune any parameters in the encoder. The decoder is a single LSTM cell with an input embedding size of 128 and a hidden size of 512. The attention mechanism linearly maps the current hidden state and all visual feature vectors size 5 512 vectors before computing attention weights. We always decode for maximum of 15 steps. The rest of the decoder exactly the same as Xu al. (2015).
The model is trained to minimize cross entropy on the training set using the AdamW optimizer Loshchilov & Hutter (2019) with a learning rate of 1e-3 and minibatches size 64. We include the double stochasticity regularization term used by Xu et al. (2015) with À = 1. We also apply dropout (p = .5) to the hidden state before predicting the next word. Across configurations, 10% of the training data is held out and used as validation set, and training stops when the model's BLEU score (Papineni et al. 2002) does not improve on this set for epochs, up to a maximum of 100 epochs.
B.2 IMPLEMENTING p(d)
We implement p(d) using a two-layer LSTM language model (Hochreiter & Schmidhuber 1997) We use an imput embedding size of 128 with a hidden state size and cell size of 512. We apply dropout to non-recurrent connections (p = .5) during training and hold out 10% of the training dataset as a validation set and following the same early stopping procedure as in Appendix B.1 except we stop on validation loss instead of BLEU.
C GENERALIZATION EXPERIMENT DETAILS
In each experiment, MILAN is trained with the hyperparameters described in Appendix B and Sec- tion 3.4 with the sole exception being the within-network splits-for these, we increase the early stopping criterion to require 10 epochs of improvement to account for the training instability caused by the small training set size.
To obtain NetDissect labels, obtain image exemplars with the same settings do for MILAN, and we obtain segmentations using the full segmentation vocabulary minus the textures.
To obtain Compositional Explanations labels, search for up length 3 formulas (comprised of not, and, and or operators) with a beam size of 5 and no length penalty. Image region exemplars and corresponding segmentations come from the ADE20k dataset (Zhou et al.] 2019).
Finally, Table 6 shows statistics for MILAN descriptions generated on the held out sets from each generalization experiment. Compared human annotators (Table 5), MILANN descriptions are
18

---

Published as a conference paper at ICLR 2022
MILAN examples
Reshit-ImaggiNa layer 3-482
AlexNat-Plaes con-S-144
BiggAN-inageea layo22100
Human: curvatures of different objects
Human: horizontal lines
Human: moses
MILAN: The edges of objects
MIL.AN: Diagonal lines
MILAN: Dog noses
ResNet-ImageNit 1ayer3-299
AlexNa-Phae cunv3-196
BigGAN Places layerd-1441 1415
Human: areus that look like keys on a keybourd MILAN: Number pads
Iuman:: surfaces that show a pallern composed lines
Human; surfaces
MILAN: The ground
MILAN: Grates
Reshler ResNee-Imagee Vet convi-27
AlexNar-Paees conv4-207
BigGAN-lmneeN Iayor-
Human: blue arens pictures
Human: heads and balloons
Human: Fruit, insects, food MILAN: Food
MILAN: Blue lines
MILAN: Faces of things
AlexNet
AlexNer-ImageNN: conv5-119
DINO MoNgo-i layer9.52
Human: colored objects
Human: green animals and plants
Human: hands of person MILAN: Human hands
MILAN: Orange and red objects
MILAN: Grass
Alexhat- convt- 188
AloxNet- AlssNet-EmageNet conv2-126
DINO MANO-ImageNet layar-18
Human: windows and screens
Human: red-orunge text and objects
Human: sign monument, keyhoand
MILAN: Signs and screens
MILAN: colored object text
MILAN: Sower cover, text on a sign, text
AlexNes Places conv2-71
AlexNet-InageNet cone4-5
DINO-ImupeNa layer1-25
Human: picture of white fence heams
Human: circles and rounded edges of many
Human: vertical white and blue lines MILAN: Brighter areas objects
MILAN: Fencing
things people MILAN: Wheels of an object
Figure 12: Randomly chosen examples of MILAN-generated descriptions from the generalization experiments of Section
average shorter (2.7 vs. 4.5 tokens), use fewer unique words (Ik vs. 4.6k), and contain adjectives twice often (9.4% vs. 19.8%). Figure T2 contains additional examples, chosen random.
D ANALYSIS EXPERIMENT DETAILS
We obtain the ResNet18 model pretrained ImageNet from torchvision (Paszke et al. 2019). We obtain neuron descriptions for the same layers that we annotate in ResNet152 (Section (.3 using the MILAN hyperparameters described in Section 3n2 and Section 3.4 We obtain part of speech tags, parse trees, and word vectors for each description from spaCy (Honnibal et al. 2020).
Figure 13 shows examples of neurons that scored high under each criterion (and consequently were among the first ablated in Fig. (5). Note that these examples include some failure cases of MILAN: for example, in the # verbs example, MILANN incorrectly categorizes all brass instruments flutes; and
19

---

Published as a conference paper at ICLR 2022
RosNetl8, 1syor4-450
nouns
RosNe18, beyet66
Test on sign, text web page, text menu adjectives
man holding flute, flune, man playing a fute
ResNet18, layer3
parse depth caption length
Puplie rod and blue Blucessent blue and parple objects
The center part of flower, or radial spikes that extend from center
R8 byye22-2
peopositions
ResNet18, byer4-427
max word difference
Syace right o objjet, space above
Animals, vihicles, and visoo
Figure 13: Examples of ablated neurons each condition Section chosen from among 10 ablated.
Unit: ResNet188 BessNet18-ImngeNNet ImageNet 1ayer4-427 "aninals, vehicles, vases"
containership
amphibian
amphibian
.33
.37
pretzel
hermi crab
hermit crab
.37
jeep
snowplow
jeep
Original Image & Ground Truth Label
Adversarial Image & Model Prediction
Residual Layer
Output Layer
Distractor Image
a
(b)
(c)
(d)
Figure 14: Cut-and-paste paste adversarial attacks highlighting non-robust behavior by neuron scored high on the max-word-diff criterion of Section (a) MILAN finds this neuron automatically because the generated description mentions two or more dissimilar concepts: animals and vehicles. The neuron is directly connected the final fully-connected output layer, and strongly influences amphibian, hermit crab, and jeep predictions according the connection weights. (b) To construct adversarial inputs, we pick three images from the Ima- geNet validation set that not include concepts detected by neuron. (c) We then select different
images to act as distractors that do include the concepts detected by the neuron. (d) By cutting and pasting the central object from the distractor to the original image, the model is fooled into predicting class label that is completely unrelated to the pasted object: e.g., it predicts amphibian when the military vehicle is pasted.
20

---

Published as a conference paper at ICLR 2022
in the adjectives example, the description is disfluent. Nevertheless, these examples confirm intuitions about the kinds of neurons selected for by each scoring criterion, as described in Section
We hypothesized in Section 5 that neurons scoring high the max-word-diff criterion correspond to non-robust behavior by the model. Figure 14 provides some evidence for this hypothesis: we construct cut-and-paste adversarial inputs in the style of Mu & Andreas (2020). Specifically, look at the example max-word-diff neuron shown in Figure 13 crudely copy and paste one of the objects mentioned in its description (e.g., a vehicle-related object like half track), and show that this can cause the model to predict one of the other concepts in the description (e.g., an animal-related class like amphibian).
E EDITING EXPERIMENT DETAILS
Hyperparameters We train a randomly initialized ResNet18 on the spurious training dataset for a maximum of 100 epochs with learning rate of lc-4 and a minibatch size of 128. We annotate the same convolutional and residual units we did for ResNet152 Section 3.3 validation loss does not improve for 4 epochs.
We stop training when
How many neurons should we remove? In practice, we cannot incrementally test our model on an adversarial set. So how do decide the number of neurons to zero? One option is look solely at validation accuracy. Figure 15 recreates Figure g with accuracy on the held out validation set (which is distributed like the training dataset) instead of accuracy
the adversarial test set. The accuracy starts peaks and starts decreasing earlier than in Fig. [] but we were to choose the number be the largest before validation accuracy permanently decreases, would choose neurons, which would still result in a 3.1 % increase in adversarial accuracy.
velaation accurety
0.80
0.78
0.74
sort text units
0.72
units
10
15 20 25
units ablated out of 1024
Figure 15: Same Fig. but shows curacy on the validation datasct, which distributed identically to the training dataset. Dotted line denotes initial accuracy.
21


Reviews:
[{"review_id": "hZxI3t92Fp3", "reviewer": "Reviewer_eaMc", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is about visualization and explainability. It is good read and inspirational to me since I have been an advocate of the sub-field. To be able go deeper on making use of the intermediate stages of a network, for visibility as well as new AI product features. The summary above shows the short of the discoveries. I have to experiment with the technique myself to go further. However, it looks promising.\n\nI also appreciate the amount of effort put on testing the system, on many architectures, datasets, and tasks. If there is more, I would like to have better characterization of the limits of the approach. \n\nGrammars, typos, etc.:\npg.1, par.1: convlution —> convolution\n", "summary_of_the_paper": "The authors introduced MILAN, for mutual-information-guided linguistic annotation of neurons) that automatically labels neurons with open-ended, compositional, natural language descriptions. \nThis is done by searching for descriptions that maximize point-wise mutual information with the image regions in which the neurons are active.\n\nIt uses (Bau, et. al. 2017) model for the selection of 15 images with regions, and Xu et. al. 2015 (Show-Attend-Tell) with a modification in pmi (probability of mutual information) for describing these regions. The Show-Attend-Tell model is trained on MILANANOTATIONS dataset, a large contribution of the technique. It is a dataset of images and fine-grained region descriptions. The dataset is comprised of 20K neurons (sets of regions) with descriptions.\n\nTESTING:\nThe testing section is a large part of the contribution as well.\n* Section 4: MILAN obtained higher agreement with human annotations on held-out networks than baseline. It also shows that the model works across architecture, dataset, and task\n* Section 5: neurons captioned with many adjectives or prepositions are relatively important to model behavior\n* Section 6: Models trained on blurred faces acquire neurons selective for blurred faces\n* Section 7: Networks devotes substantial capacity to identifying text labels in images.\n", "summary_of_the_review": "* A higher level of explainability of the regions responsible for network final result\n* A thorough analysis of the actionable insights that can be used for the model\n* Limits of the approach needs to be discussed further\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "W9B1yB4vEQ", "reviewer": "Reviewer_CDo5", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- Strength\n  - This paper is well-written and easy to follow. The authors provide sufficient technical details for readers to understand and reproduce their work. Data, code, and the trained model will be open source.\n  - This paper picks up an intriguing topic that aims to interpret deep learning models by investigating hidden units and summarizing their exemplar activation by natural language descriptions. The proposed method is concise, straightforward, and well-motivated.\n  - The natural language descriptions can capture categorical, relational, and logical structure across different levels in the learned features. It’s nice to see low-level features like edges (“the top boundaries of horizontal objects”), middle-level features like shapes (“Poles and legs”), and relatively high-level features like objects (“dog faces”) could all be generated quite well by this same model.\n  - The results suggest generalizability across different model architectures, datasets, and tasks. This makes MILAN readily useful for many other potential applications, including the three interesting experiments shown in section 5-7.\n- Comments\n  - The model is trained on a newly collected dataset MILANNOTATIONS. Each unit was annotated by three human participants. But the inter-annotator agreement among human annotations seems not quite high (Table 4, Figure 10), compared to the BERTScore between model-generated descriptions and human annotations (Table 2, Table 3). How did the authors handle this inter-annotator inconsistency during their model training? Is there any additional quality control/validation performed for this MILANNOTATIONS dataset?\n  - Following the first point, I wonder if the authors would consider scaling up their methods by leveraging the existing large, multimodal datasets like GQA/Visual Genome or visual-language model trained on large paired image-text datasets like CLIP/ALIGN?\n  - Fig. 3 is not referenced in the main text. And I think these failure modes are interesting, and taking a closer look at them might be inspiring for improving this model in future studies. Do authors have further comments or thoughts about this result?\n  - The results in Fig. 4 are quite interesting. The bar chart suggests low-level visual features are more described by adjectives, middle-level units need more prepositions and verbs to describe relational features, and high-level units need more complex composition of words thus resulting in longer length and deeper parser trees. This probably aligns well with our intuition and expectation. For units that may contribute to those \"non-robust\" model behavior, are they described by more nouns with higher max word diff? Will the proposed MILAN model be able to detect those \"non-robust\" units and edit the network to improve its performance?\n- Minor:\n  - What do different dots refer to in Fig. 5?\n", "summary_of_the_paper": "This paper describes a novel procedure (MILAN) to interpret deep learning models for computer vision by generating natural language description that specifies the activation selectivity of a given neuron in the model. For this aim, they first define an exemplar set of input image regions for each neuron by thresholding its activation value. Then they search a natural language description by optimizing the point-wise mutual information between descriptions and the exemplar set. The probability distributions for calculating the mutual information are approximated by training the SAT model and a two-layer LSTM language model on a newly collected dataset (MILANNOTATIONS), which includes annotations of 20k units labeled by human participants. The authors first test the generalizability of MILAN descriptions across different model architectures, datasets, and tasks, showing its privilege of generating higher agreement with human annotations compared to baseline methods. They then demonstrate three interesting applications of MILAN procedure and show how these natural language descriptions help us to understand and control the learned models.", "summary_of_the_review": "Overall I think this work is well-motivated, technically sound, and showing promising results that support potential applications for interpreting and improving deep learning models for computer vision. Some minor changes could be made to improve the clarity. More details about how authors control / validate the quality of the MILANNOTATIONS dataset could be included.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "4xkbEpbPDJr", "reviewer": "Reviewer_YviU", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I want to preface my review by stating that I am a domain expert in NLP and not CV. While I attempted a preliminary search for related work, I am not sure if there is prior work in specifically generating language descriptions of neurons (I understand that there is prior work in explaining neuron behaviour by examining inputs that activate it). Hence I may significantly modify my review based on the response of my peer reviewers.\n\nThis is a strong piece of work that clearly states its hypotheses and carefully designs experiments to test said hypotheses. The technical novel of the particular method is low, however I do not think that is the point of this work. What this paper does show is a novel way to interpret model behaviour, and allows for very useful downstream applications (e.g. fast filtering of neurons activated by a particular feature through text selection). I advocate for acceptance of this work, and propose some potential improvement. I do have some concerns about the scoping of this work. The title and description suggests a more general method, however experiments are purely based on images. I think the paper title should be more finely scoped (NL descriptions of Deep Image Features).\n\nMajor:\n- Do these transfer results generalize to other tasks? or are they specific to image models? Or are they specific to the fact that all of these image models are trained/evaluated on (two) datasets that are similar to each other in terms of image distribution? I think perhaps the paper makes this technique seem more portable than it actually is. For example I would have to collect data for my task distribution in order for this technique to work. If you can show this working with pretrained zero-shot image captioning models than this result would be a lot more convincing.\n- How do image captioning models do on the task of generating descriptions conditioned on inputs? The implicit hypothesis here is that they do not work well because they operate on higher levels of abstraction, but I would like to see empirical results of this by using caption models as baselines (e.g. zero-shot, fine-tuned).\n- What do generated descriptions look like? How long are the descriptions? What is the vocabulary size? How diverse are the descriptions?\n- What does the released dataset look like? Do you control for bias in the description? Diversity? Coverage?\n- This objective not only needs to maximize over all input space but also all description space. The former is talked about but the latter is skimmed over (by stating that beam search is performed). It's unclear whether beam search would yield useful description if the description is more complex (e.g. in a language task like question-answering). \n- It would be nice to show spurious feature filtering results on something that is not a contrived task (e.g. classification with spurious text labels on top left corner of image) and instead on something impactful (e.g. removing racial/gender bias by filtering out neurons with corresponding descriptions).\n\n\nMinor:\n- typo; convlutional network\n\nQuestions:\n- Why is there so much variance across architecture pairs? For example large gains on AlexNet -> Places and ResNet -> ImageNet but small gains on ResNet -> Places and AlexNet -> ImageNet.", "summary_of_the_paper": "This work proposes a framework for interpreting model behaviour by generating language descriptions of neurons in the model. The method is trained to maximize mutual information between the description and input examples that activate the neuron. The experiments show results for image classification, generation, and unsupervised representation learning. In particular, this method yields model description that achieve higher BERTScore than max-likelihood training when transferring to neurons in new architectures. Moreover the authors show how this technique can be used to compute correlations between descriptions and neuron importance (e.g. wrt classification accuracy), interpreting model behaviour (e.g. anonymized models still select for unblurred faces), and that one can remove spurious feature corrleations by removing models with specific descriptions.", "summary_of_the_review": "Strong paper demonstrating how to generation descriptions of image features and how to use method to analyze model behaviour and filter out neurons by description.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a method to interpret neurons in the vision neural models by generating natural language description that specifies the activation selectivity of a given neuron. The proposed method first identifies an exemplar set of input image regions that corresponds to a neuron, then searches a natural language description by optimizing the point-wise mutual information between descriptions and the exemplar set.

Strength:
- Reasonable method design and clear writing
- Important problem and broad applications
- Extensive experiments for evaluation of the proposed method

Weakness:
- Need more discussion on the limitations of the proposed method
- Elaboration on the human inter-annotation agreement
- Analysis on method transferability across tasks.
Decision: Accept (Oral)