Natural Language Descriptions of Deep Features

1 INTRODUCTION . A surprising amount can be learned about the behavior of a deep network by understanding the individual neurons that make it up . Previous studies aimed at visualizing or automatically categorizing neurons have identified a range of interpretable functions across models and application domains : low-level convolutional units in image classifiers implement color detectors and Gabor filters ( Erhan et al. , 2009 ) , while some later units activate for specific parts and object categories ( Zeiler & Fergus , 2014 ; Bau et al. , 2017 ) . Outside of computer vision , single neurons have been found to encode sentiment in language data ( Radford et al. , 2017 ) and biological function in computational chemistry ( Preuer et al. , 2019 ) . Given a new model trained to perform a new task , can we automatically find , describe , and catalog these behaviors ? Techniques for characterizing the behavior of individual neurons are still quite limited . Approaches based on visualization ( Zeiler & Fergus , 2014 ; Girshick et al. , 2014 ; Karpathy et al. , 2015 ; Mahendran & Vedaldi , 2015 ; Olah et al. , 2017 ) leave much of the work of interpretation up to human users , and can not be used for large-scale analysis . Existing automated labeling techniques ( Bau et al. , 2017 ; 2019 ; Mu & Andreas , 2020 ) require researchers to pre-define a fixed space of candidate neuron labels ; they label only a subset of neurons in a given network and can not be used to surface novel or unexpected behaviors . This paper develops an alternative paradigm for labeling neurons with expressive , compositional , and open-ended annotations in the form of natural language descriptions . We focus on the visual domain : building on past work on information-theoretic approaches to model interpretability , we formulate neuron labeling as a problem of finding informative descriptions of a neuron ’ s pattern of activation on input images . We describe a procedure ( called MILAN , for mutual-informationguided linguistic annotation of neurons ) that labels individual neurons with fine-grained natural language descriptions by searching for descriptions that maximize pointwise mutual information with the image regions in which neurons are active . To do so , we first collect a new dataset of fine-grained image annotations ( MILANNOTATIONS , Figure 1c ) , then use these to construct learned approximations to the distributions over image regions ( Figure 1b ) and descriptions . In some cases , MILAN surfaces neuron descriptions that more specific than the underlying training data ( Figure 1d ) . MILAN is largely model-agnostic and can surface descriptions for different classes of neurons , ranging from convolutional units in CNNs to fully connected units in vision transformers , even when the target network is trained on data that differs systematically from MILANNOTATIONS ’ images . These descriptions can in turn serve a diverse set of practical goals in model interpretability and dataset design . Our experiments highlight three : using MILAN-generated descriptions to ( 1 ) analyze the role and importance of different neuron classes in convolutional image classifiers , ( 2 ) audit models for sensitive demographic data by comparing comparing their features when trained on anonymized ( blurred ) and non-anonymized datasets , and ( 3 ) identify and mitigate the effects of spurious correlations with text features , improving classifier performance on adversarially distributed test sets . Taken together , these results show that fine-grained , automatic annotation of deep network models is both possible and practical : rich explanations produced by automated annotation procedures can surface meaningful and actionable information about model behavior.1 2 RELATED WORK . Interpreting deep networks MILAN builds on a long line of recent approaches aimed at explaining the behavior of deep networks by characterizing the function of individual neurons , either by visualizing the inputs they select for ( Zeiler & Fergus , 2014 ; Girshick et al. , 2014 ; Karpathy et al. , 2015 ; Mahendran & Vedaldi , 2015 ; Olah et al. , 2017 ) or by automatically categorizing them according to the concepts they recognize ( Bau et al. , 2017 ; 2018 ; Mu & Andreas , 2020 ; Morcos et al. , 2018 ; Dalvi et al. , 2019 ) . Past approaches to automatic neuron labeling require fixed , pre-defined label sets ; in computer vision , this has limited exploration to pre-selected object classes , parts , materials , and simple logical combinations of these concepts . While manual inspection of neurons has revealed that a wider range of features play an important role in visual recognition ( e.g . orientation , illumination , and spatial relations ; Cammarata et al . 2021 ) MILAN is the first automated approach that can identify such features at scale . Discrete categorization is also possible for directions in representation space ( Kim et al. , 2018 ; Andreas et al. , 2017 ; Schwettmann et al. , 2021 ) and for clusters of images induced by visual representations ( Laina et al. , 2020 ) ; in the latter , an off-the-shelf image captioning model is used to obtain language descriptions of the unifying visual concept for the cluster , although the descriptions miss low-level visual commonalities . As MILAN requires only a primitive procedure for generating model inputs maximally associated with the feature or direction of interest , future work might extend it to these settings as well . 1Code , MILANNOTATIONS data , and a trained MILAN model are available at [ anonymized ] . Natural language explanations of decisions Previous work aimed at explaining computer vision classifiers using natural language has focused on generating explanations for individual classification decisions ( e.g. , Hendricks et al. , 2016 ; Park et al. , 2018 ; Hendricks et al. , 2018 ; Zellers et al. , 2019 ) . Outside of computer vision , several recent papers have proposed procedures for generating natural language explanations of decisions in text classification models ( Zaidan & Eisner , 2008 ; Camburu et al. , 2018 ; Rajani et al. , 2019 ; Narang et al. , 2020 ) and of representations in more general sequence modeling problems ( Andreas & Klein , 2017 ) . These approaches require task-specific datasets and often specialized training procedures , and do not assist with interpretability at the model level . To the best of our knowledge , MILAN is the first approach for generating compositional natural language descriptions for interpretability at the level of individual features rather than input-conditional decisions or representations . More fundamentally , MILAN can do so independently of the model being described , making it ( as shown in Section 4 ) modular , portable , and to a limited extent task-agnostic . 3 APPROACH . Neurons and exemplars Consider the neuron depicted in Figure 1b , located in a convlutional network trained to classify scenes ( Zhou et al. , 2017 ) . When the images in Figure 1 are provided as input to the network , the neuron activates in patches of grass near animals , but not in grass without animals nearby . How might we automate the process of automatically generating such a description ? While the image regions depicted in Fig . 1b do not completely characterize the neuron ’ s function in the broader network , past work has found that actionable information can be gleaned from such regions alone . Bau et al . ( 2020 ; 2019 ) use them to identify neurons that can trigger class predictions or generative synthesis of specific objects ; Andreas & Klein ( 2017 ) use them to predict sequence outputs on novel inputs ; Olah et al . ( 2018 ) and Mu & Andreas ( 2020 ) use them to identify adversarial vulnerabilities . Thus , building on this past work , our approach to neuron labeling also begins by representing each neuron via the set of input regions on which its activity exceeds a fixed threshold . Definition 1 . Let f : X → Y be a neural network , and let fi ( x ) denote the activation value of the ith neuron in f given an input x.2 Then , the exemplar representation of the neuron fi is given by : Ei = { x ∈ X : fi ( x ) > ηi } . ( 1 ) for some activation threshold ηi ( discussed in more detail below ) . Exemplars and descriptions Given this explicit representation of fi ’ s behavior , it remains to construct a description di of the neuron . Past work ( Bau et al. , 2017 ; Andreas et al. , 2017 ) begins with a fixed inventory of candidate descriptions ( e.g . object categories ) , defines an exemplar set E′d for each such category ( e.g . via the output of a semantic segmentation procedure ) then labels neurons by optimizing di : = argmind δ ( Ei , E ′ d ) for some measure of set distance ( e.g . Jaccard , 1912 ) . In this work , we instead adopt a probabilistic approach to neuron labeling . In computer vision applications , each Ei is a set of image patches . Humans are adept at describing such patches ( Rashtchian et al. , 2010 ) and one straightforward possibility might be to directly optimize di : = argmaxd p ( d | Ei ) . In practice , however , the distribution of human descriptions given images may not be wellaligned with the needs of model users . Fig . 2 includes examples of human-generated descriptions for exemplar sets . Many of them ( e.g . text for AlexNet conv3-252 ) are accurate , but generic ; in reality , the neuron responds specifically to text on screens . The generated description of a neuron should capture the specificity of its function—especially relative to other neurons in the same model . We thus adopt an information-theoretic criterion for selecting descriptions : our final neuron description procedure optimizes pointwise mutual information between descriptions and exemplar sets : Definition 2 . The max-mutual-information description of the neuron fi is given by : MILAN ( fi ) : = argmax d pmi ( d ; Ei ) = argmax d log p ( d | Ei ) − log p ( d ) . ( 2 ) To turn Eq . ( 2 ) into a practical procedure for annotating neurons , three additional steps are required : constructing a tractable approximation to the exemplar set Ei ( Section 3.1 ) , using human-generated image descriptions to model p ( d | E ) and p ( d ) ( Section 3.2 and Section 3.3 ) , and finding a highquality description d in the infinite space of natural language strings ( Section 3.4 ) . 2In this paper , we will be primarily concerned with neurons in convolutional layers ; for each neuron , we will thus take the input space X to be the space of all image patches equal in size to the neuron ’ s receptive field . 3.1 APPROXIMATING THE EXEMPLAR SET . As written , the exemplar set in Equation ( 1 ) captures a neuron ’ s behavior on all image patches . This set is large ( limited only by the precision used to represent individual pixel values ) , so we follow past work ( Bau et al. , 2017 ) by restricting each Ei to the set of images that cause the greatest activation in the neuron fi . For convolutional neurons in image processing tasks , sets Ei ultimately comprise k images with activation masks indicating the regions of those images in which fi fired ( Fig . 1a ; see Bau et al . 2017 for details ) . Throughout this paper , we use exemplar sets with k = 15 images and choose ηi equal to the 0.99 percentile of activations for the neuron fi . 3.2 MODELING p ( d | E ) AND p ( d ) The term pmi ( d ; Ei ) in Equation ( 2 ) can be expressed in terms of two distributions : the probability p ( d | Ei ) that a human would describe an image region with d , and the probability p ( d ) that a human would use the description d for any neuron . p ( d | Ei ) is , roughly speaking , a distribution over image captions ( Donahue et al. , 2015 ) . Here , however , the input to the model is not a single image but a set of image regions ( the masks in Fig . 1a ) ; we seek natural language descriptions of the common features of those regions . We approximate p ( d | Ei ) with learned model—specifically the Show-Attend-Tell image description model of Xu et al . ( 2015 ) trained on the MILANNOTATIONS dataset described below , and with several modifications tailored to our use case . We approximate p ( d ) with a two-layer LSTM language model ( Hochreiter & Schmidhuber , 1997 ) trained on the text of MILANNOTATIONS . Details about both models are provided in Appendix B .

Summary: This work proposes a framework for interpreting model behaviour by generating language descriptions of neurons in the model. The method is trained to maximize mutual information between the description and input examples that activate the neuron. The experiments show results for image classification, generation, and unsupervised representation learning. In particular, this method yields model description that achieve higher BERTScore than max-likelihood training when transferring to neurons in new architectures. Moreover the authors show how this technique can be used to compute correlations between descriptions and neuron importance (e.g. wrt classification accuracy), interpreting model behaviour (e.g. anonymized models still select for unblurred faces), and that one can remove spurious feature corrleations by removing models with specific descriptions.