Interpreting Neurons in Deep Vision Networks with Language Models
Authors: Nicholas Bai
Rahul Ajay Iyer
Tuomas Oikarinen
Akshay R. Kulkarni
Tsui-Wei Weng
Venue: Accepted by TMLR ()

Abstract: In this paper, we propose Describe-and-Dissect (DnD), a novel method to describe the roles of hidden neurons in vision networks. DnD utilizes recent advancements in multimodal deep learning to produce complex natural language descriptions, without the need for labeled
training data or a predefined set of concepts to choose from. Additionally, DnD is training-free, meaning we don’t train any new models and can easily leverage more capable general purpose models in the future. We have conducted extensive qualitative and quantitative analysis
to show that DnD outperforms prior work by providing higher quality neuron descriptions. Specifically, our method on average provides the highest quality labels and is more than 2× as likely to be selected as the best explanation for a neuron than the best baseline. Finally, we present a use case providing critical insights into land cover prediction models for sustainability applications.

Published in Transactions on Machine Learning Research (01/2025)
Interpreting Neurons in Deep Vision Networks with Language Models
Nicholas Bai* UC San Diego
nicholaszyhai@@maalloom com
Rahul A. Iyer UT Austin
mui39668utezas.cmm edu
Tuomas Oikarinen UC San Diego
Srnkarinnennucsdde edu
Akshay Kulkarni UC San Diego
a2kulkarni@ucsd. edu
Tsui- Wei Weng
twengguucet edu
UC San Diego
Reviewed on OpenReview: https: optpr/vveeereeeeee.em net/ forum? id= æ1 dXvvEl Vd
Abstract
In this paper, we propose Describe-and aadbbee-nd-Dissee Disseet (DnD), a novel method to describe the roles of hidden neurons in vision worksrks. DnD utilizes recent advancements in multimodal deep learning to produce complex natural language descriptions, without the need for labeled training data a predefined set of concepts don't train any new models and can easily leverage more capable general purpose models in the future. We have conducted extensive qualitative and quantitative analysis to show that DnD outperforms prior work by providing higher quality neuron descriptions. Specifically, our method on average provides the highest quality labels and more than 2x as likely to be selected as the best explanation for a neuron than the best baseline. Finally, we present a use case providing critical insights into land cover prediction models for sustainabilityy applications. Our code and data are available at http:://ggttaaocoooomrrttttttttp
choose from. Additionally, DnD
bruining-free, meaning
1 Introduction
Recent advancementts Deep Neural Networks (DNNs) within machine learning have enabled unparalleled development in multimodal artificial intelligence. While these models have revolutionized domains across image recognition and natural language processing, they haven't seen much use in various safety-critical applications, such healthcare or ethical decision- This is in part due to their cryptic "black box" nature, where the internal workings of complex neural networks have remained beyond human comprehension. This makes it hard to place appropriate trust in the models and additional insight in their workings is needed to reach wider adoption.
Previous methods have gained a deeper understanding DNNs by examining the functionality (also known as concepts) of individual neurons" This includes works based on manual inspection (Erhan et al. 2009 Zhou et al. 2015 Olah al. 2020 Goh et al. 2021 which provide high quality description at the cost
N. Bai and R. Iyer contributed equally this work. Major work done in Summer outreach program UC San Diego. We conform to prior works' notation and use "neuron" to describe channel in CNNs.

---

Published in Transactions on Machine Learning Research (01/2025)
Layer
Layer 3
Layer Neuron Activating Images:
Layer Neuron 535
Top Activating Images:
DnD: blue aquatic objects (4.5)
CLIP-Dissect underwater(4. Net-Dissese blue (color)(4.0)
patierns (4.3)
CLIP-Dissece grid
MILAN: Blue areas/4.0)
MILAN: Horzontal lines (2:3)
Net-Dissect: pane (3.0)
Layer Neuron Activating Images:
Neuron 305
Top Activating Images:
colorful
P.Dissedt: pastel (4. (4.0)
fruits (4.7)
CL P...sst:::: fruits (4.3)
MILAN: Spots of the color white (4.0)
Net-Diesect: waffled (2.0)
MILAN: Faces of things (3.0)
Net: DisseDissctt head (2.0)
Layer
Layer 4
Layer
Top Activating Images:
Layer
Top Activating Images:
geometric patterns textures (4.3) CLIP-Dissect mosaic (2.7)
DnD: glassware and bubbles (4.3)
glassware (3.3)
MILAN: Red and while colored objects (2.3) Net-Dissect: dotted (2.3)
MILAN: Bottles (2.7)
Nel-Dissect: bubbly (4
Layer
Top Activating Images:
Layer
Top Activating Images:
patterns and scenes (4.3)
CLIP-Dissed: magenta (4.0)
DnD: inter architecture4..0
CLIP-Dissect; kitchens(4.0 0)
MILAN: The color blue (2.7)
Net-Diesect: tree (1.0)
MILAN: Vertical lines(2.0)
Net-Dissect headboard(2.0)
Figure I: Neuron descriptions provided by our method (DnD) and baselines CLIP-Dissect (Oikarinen & Weng 2023), MILAN (Hernandez et al., 2022)), ), and Network Dissection (Bau al. 2017) for random neurons from ResNet-50 trained on ImageNet. We have added the average quality rating from our Amazon Mechanical
Turk experiment described in Section next to each label and color-coded the neuron descriptions by whether we believed they were accurate, somewhat correct or vague/imprecise.
of being very labor intensive. Allernatively, Network Dissection (Bau et al. 2017) automated this labeling process by creating the pixelwise labeled dataset, Broden, where fixed concept set labels serve as ground truth binary masks for corresponding image pixels. The datasot was then used to match ncurons to a label from the concept set based on how similar their activation palterns and the coucept maps were. While earlier works, such as Network Dissection, were restricted to an annotated dataset. and predetermined concept. set, CLiP-Dissect (Oikarinen & Weng 2023) offered a solution by no longer requiring labeled concept data, but still requires predetermined concept set input. By utilizing OpenAI's CLIP model, CLIP- matches neurons to concepts based their activations in response to images, allowing for a more flexible probing dataset and concept set compared to previous works.
Concepts detected by certain neurons, especially in intermediate layers, prove to be difficult to encapsulate using the simple, often single-wor word descriptions provided in a fixed concept set. MILAN (Hernandez al.) 2022) sought to enhance the quality of these neuron labels by providing generative descriptions, but their method requires training new description model from scratch to match human explanations dataset of neurons. This leads to their proposed method being more brittle and often performs poorly outside its training data.
However, these methods still share a major limitation:
To overcome these limitations, we propose Describe-and-De -isseDtttcct (abbreviated as DnD) in Section a pipeline to dissect DNNs by utilizing an image-to-text model to describe highly activating images for corresponding neurons. The descriptions are then semantically combined by a large language model, and finally refined with synthetic images to generate the final concept of neuron. We conduct extensive qualitative and quantitative analysis in Section and show that. Describe-and- Dissect outperforms prior work by providing high quality
2

---

Published in Transactions on Machine Learning Research (01/2025)
Table 1: Comparison of existing automated neuron labeling methods and our method Describe-and Dissect (DnD). Green and boldfaced Yes or No indicates the desired property for a column. DnD has all the desired properties while existing work has some limitations.
Requires
Generative
Uses Spatial
Can easily
Training Free
Method|
property
Concept Annotations
Natural Language Descriptions
Activation Information
leverage better
future models
Network Dissection [Ba ct al.
2017
Yes
Yes
No
Yes
No
MILAN [ernandez al. 2022
Training only
No
Yes
Yes
No
CLIP-Dissect (Oikarinen Wong
2023
No
Yes
No
No
Yes
FALCON (Kalibhat et al.
2023
No
Yes
No
Yes
Yes
DnD (This work)
No
Yes
Yes
Yes
Yes
neuron descriptions. Specifically, we show that Describe-and- Dissect provides more complex and higher-quality descriptions (up to 2-4x better) on intermediate layer neurons than her contemporary methods in a large scale user study, Example descriptions from our method are displayed in Figure [ Additionally, we present a uso-case study demonstrating DnD's ability interpret and improve upon current sustainability models in Section 5
2 Background and related work
2.1 Neuron Interpretability Methods
Network Dissection [Bau et al. 2017) is the first method developed to automatically describe individual neurons' functionalities. The authors first defined the as ground-truth concept mask. The dataset is composed of various images I,, cach labeled with concepts at the pixel-level. This forms ground truth binary mask Lo(x;) which used calculate the intersection over union (IoU) score between La: and the binary mask from the activations of the neuron over all
dataset Broden, denoted as Drooden:
Eetepproden Mx(x))t.((a))
images T3 E DBrdden; denoted Ma(x)): ToUk,e E 2,,E Ma(x:)un,(a.)" The concept is assigned to
neuron 1 if IoUR,e > 77, where the threshold n) was set to 0.04. Intuitively, this method finds the labeled concept whose presence in the image is most closely correlated with the neuron having high activation.
Extensions of Network Dissection were proposed by Bau et al. (2020) and Mu & Andreas (2020).
However, Network Dissection is limited by the need of concept annotation and the concept set is a closed set that may be hard to expand. To address these linitations, a recent work CLIP-Dissecl (Oikarinen Weng 2023) utilizes OpenAl's multimodal CLIP (Radford et al. 2021) model to describe neurons automatically without requiring annotated concept data. They leverage CLIP to score how similar each image in the probing dataset Dprobe is to the concepts in user-specified concept set to generate concept activation matrix. To describe neuron, they compare the activation pattern of said neuron to activations of different concepts on the probing data, and find the concept that is the closest match using similarity function, such as SPM. Another very recent work FALCON (Kalibhat et al., 2023) uses a method similar to CLIP-Dissect but augments it via counterfactual images by finding inputs similar to highly activating images with low activation for the target neuron, and utilizing spatial information of activations via cropping. However, they solely rely on cropping the most salient regions within probing image filter spurious concepts that are loosely related to the ground truth functionality labels of neurons. This approach largely restrict their method to local concepts while overlooking holistic concepts within images, noted by Kalibhat et al. (2023). Their approach is also limited to single word/set of words descriptions that are unable to reach the complexity of natural language.
On the other hand, MILAN (Hernandez al. 2022) a different approach to describe neurons using natural language descriptions in generative fashion. Note that despite the concept sets in CLIP-Dissce and FALCON being flexible and open, they cannot provide generative natural language descriptions like MIL.NN
3

---

Published in Transactions on Machine Learning Research (01/2025)
The central idea of MILAN is to train an image-to-text model from scratch to describe the neuron's role based on 15 most highly activating images. Specifically, it was trained on crowdsourced descriptions for 20,000 neurons from selected networks. MILAN can then generate natural language descriptions to new neurons by outputting descriptions that maximize the weighted pointwise mutual information (WPMI) between the description and the active image regions. One major limitation of MILAN is that the method require training a model to imitate human descriptions of image regions on relatively small training dataset, which may cause inconsistency and poor explanations further from training data. In contrast, our DnD is training-free, generative, and produces higher quality of newron descriptions supported by our extensive experiments in Figure and Table 3 A detailed comparison between our method and the baseline methods is shown in
Table M
2.2 Leveraging Large Pretrained models
In our DnD pipeline, we are able leverage recent advances in the large pre-trained models to provide high quality and generative neuron descriptions for DNNSs in training-free manner. Below we briefly introduce the Image-to-Text Model, Large Language Models and Text-to-Image Model used in our pipcline implementation. The first model is Bootstrapping Language-Image Pretraining (BLIP) (Li et al.) 2022), which is an image-to-text model for vision- laniua-ang tasks that generates synthetic captions and filters noisy ones, employing bootstrapping for the captions to utilize noisy web data. While our method can use any image-to-text model, we use BLIP in this paper for our Step 2 in the pipeline due to BLIP's high performance, speed, and relatively low computational cost. However, we note that our method can be easily adapted to leverage more advanced models in the future.
The second model is GPT-3.5 Turbo, which is transformer model developed by OpenAl for understanding and generating natural language. It provides increased performance from other contemporary models due to its vast training dataset and immense network size. We utilize GPT-3.5 Turbo for natural language processing and semantic summarization in the Step 2 of our DnD. We use GPT-3.5 Turbo in this work as it's one of the SOTAs in LLMs and cheap to use, but our method is compatible with other future and more advanced LLMs. We provide a quantitative comparison analyzing the effect of GPT-3.5 Turbo, GPT-4.0, and LLaMA2 on DnD's label quality in Appendix [A.4.] as well evaluation on cost and usage limitations for each model.
The third model is Stable Diffusion (Rombach et al. 2022), which is text-to-imant latent diffusion model (LDM) trained on a subset from the LAION-5B database (Schuhmann et al. 2022). By performing the diffusion process over the low dimensional latent space, Stable Dillusiin is signilicantly more compulationally efficient than other diffusion models, such as DALLE (Ramesh et al. 2021 Due to its open availability, lower computational cost, and high performance, we employ Stable Diffusion for our image generation needs in the Step 3 of DnD.
3
Methods
Overview. In this section, we present Describe-and-Dee -isssect (DnD), a comprehensive method to produce generative neuron descriptions in deep vision networks. Our method is training-free, model-agnostic, and can be easily adapted to utilize advancements in multimodal deep learning. DnD consists of three steps:
Step 1. Probing Set Augmentation: Augment, the probing dataset with altention cropping to include both global and local concepts. This helps better describe localized neuron activations.
Candidate Concept Generation: Generate initial concepts by describing highly activating images and subsequently summarize them into candidate concepts using GPT. This is the main step that generates a set of possible explanations.
Step 2.
Step 3. Best Concept Selection: Generate new images based on candidate concepts and select the best concept based on neuron activations on these synthotic images with a scoring function. This refines the description by selecting the most accurate neuron description from descriptions produced in Step 2.
4

---

Published in Transactions on Machine Learning Research (01/2025)
Candidate Concept Generation
Probing Set Augmentation
neuron
GPT Model
BLIP
D
probe
propped
Target Model
BLIP Image-to-Text Extractor
GPT Caption Summarizer
Top Activating Images
Best Concept Selection
Swimming Shark
neuron
Stable Diffusion
Concept Scoring
Black and white Fish
Swimming Shark
Best Concept Label of neuron
Oceanic
Text-to-lmage
Target Model
Candidate
Model
Concepts
Figure 2: Overview of Describe and Dissect (DnD) algorithm. For given target model, DnD consists of three important steps to identify neuron concepts (e.g. 'Swimming Shark' for neuron n).
An overview of Describe-and-Dee (DnD) and these steps are illustrated in Figure
2
3.1 Step 1: Probing Set Augmentations
Probing dataset Pprobe is the set of images record neuron activations before generating description. As described in Section 2.1 one major limitation of Kalibhat et al. (2023) is the restriction to local concepts while overlooking holistic concepts within images, while one limitation of Oikarinen & Weng (2023) is not incorporating the spatial activation information. Motivated by these limitations, DnD resolves these problems by augmenting the original probing dataset with set of attention crops of the highest activating images from the original probing datasct. The attention crops can capture the spatial information of the activations and we name this set Peropped; shown in Figure We discuss the implementation details of our attention cropping procedure in Appendix A.1.I and perform an ablation study on its effects in Appendix A.4.1
3.2 Step 2: Candidate Concept Generation
The top K most highly activating images for a neuron are collected in set 1, K, by selecting K images 24 E Droobe U Deropped with the largest (xA)) Here g is a summary function (for the purposes of our experiments we define as the spatial mean) and (xx,) the activation map of neuron on input T4. We then generate a of candidate concepts for the neuron with the following two part process:
Step 2A Generate descriptions for highly activating images: We utilize BLIP imago-to-text model to generatively produce an image caption for each image in T. For an image [дE[K], we feed I into the base BLIP model to obtain image caption.
Step 2B Summarize similarities between image descriptions: Next we utilize OpenAl's GPT-3.5 Turbo model summarize similaritics between the image captions for each neuron being checked. GPT is prompted generate N descriptions which identify and summarize the conceptual similarities between most of the BL..-ggeeerteed captions.
5

---

Published in Transactions on Machine Learning Research (01/2025)
The output of Step 2B is set of N descriptions which call "candidate concepts'. We denote this set as T = (Ti,.., Tv). For the purposes of our experiments, we generate N = candidate concepts unless otherwise mentioned. The exact prompt used for GPT summarization is shown in Appendix A.1.3
3.3 Step 3: Best Concept Selection
The last crucial component of DnD is concept selection, which selects the concept from the set of candidate concepts T that is most correlated to the activaling images of neuron. We first use the Stable Dilfusion model (Rombach et al. 2022) from Hugging Face to generate images for each concept T Generating new images is important as it allows us to differentiate between neurons truly detecting a concept or just spurious correlations in the probing data. The resulting set of images is then fed through the target model again to record the activations of target neuron on the new images. Finally, the candidate concepts are ranked using a concept scoring function, as discussed in Section 3.4
Concept Selection Algorithm The algorithm consists of 4 substeps. For each neuron n, we start by:
1. Generate supplementary images. Generate Q synthetic images using a text-to-image model for each label [][[N-- The set of images from each concopt is denoted as Dj, [Dj| = Q. The total new datasct is then Dnew - D, {afe0 ... NN which represents the full of generated images. For the purposcs of the experiments in this paper, set = 10.
2. Feed new dataset Dnew, back into the target model and rank the images based on activation. We then evaluate the activations of target neuron n on images in Dnew and compute the rank of each image in terms target neuron activation. Given neuron activations An (xyee,, we define
In (I(An(=)""")) (2 (#W:)))) as the sot of scalar neuron activations.
3. Gather the ranks of images corresponding to concept Tj. Let Rank( 9) be function that returns the rank of an element in set 9, such that Rank(z';g) = 1 if x' is the largest element in G. For every concept T,, we record the ranks of images generated from the concept in Hj, where Hy = [Rank(g(A, (x)); 9m.) A r E Dj}, and Hj is sorted in increasing order, so Hzi is the rank of the lowest ranking element.
4. Assign scores to each concept. The scoring function score(Hl) assigns score to concept using the rankings of the concept's generated images, and potential additional information. The concept with the best (highest) score in T is selected as the concept label for the neuron. Concept scoring functions are discussed below in Section 34
In simpler terms, the intuition behind this algorithm is that if a neuron encodes for a concept e, then the images generated to encapsulate that concept should cause the neuron to activate highly. While only experiment with Best Concept selection within the DnD framework, it can be independently applied with other methods like Bau et al. (2017); Hernandez et al. (2022); Oikarinen & Weng (2023) to select the best concept out of their top-k best descriptions, which is another benefit of our proposed method. DnD can also used without Best Concept Selection (Step 3) to reduce computational costs.
3.4 Scoring Function
For a given neuron, we use scoring function to rate candidate concept accuracy during Best Concopt Selection (Step 3). Simple metrics such as mean are heavily prone to outliers that result in skewed predictions so we propose scoring function that weights the average rank of top activating images mapping to a candidate concept.
(,ore(R,,1,,))) = (N Ra,)
Here, the average rank of images for candidate concept j, Vj E (1, denoted R and Rank(R,) sorts the average rank R, of each candidate concept in increasing order. E(I,D; computes the average cosine similarity between image embeddings of DJ, and using CL.PPVVT-B/16 (Radford et al. 2021), with Dj C D,

---

Published in Transactions on Machine Learning Research (01/2025)
for highest activating images (for our experiments, we use t 10). In practice, R₁ is computed as the square of the ranks in top 3 = 5 ranking images for better differentiation ion between scores, Rg = {R) S (8..
Sections A.1.4 the details specifies behind the function. In Section A..155) we compare between various fanctions and show our algorithm works robustly with different options.
4
Experiments
In this section, we present extensive qualitative and quantitative analysis to show that DnD outperforms prior works by providing higher quality neuron descriptions. For fair comparison, we follow the setup in prior works to run our algorithm on the following two networks: ResNet-50 and ResNet-18 (He et al. 2016 trained on ImageNet (Russakovsky et al.) 2015) and Place365 (Zhou et al.) 2016) respectively. In Section 1.1 we qualitatively analyze DnD along with other methods on random neurons and show that our method we quantitatively show that DnD yiclds superior results to comparable methods. In Section 4.3 show that our method outperforms existing neuron description methods in large scale crowdsourced studies. Finally in Section 4.4 we study the importance of critical steps in our pipeline by ablating away Generative Image Captioning (Step 2A) and Concept Selection (Step 3). Supplementary results are presented in the appendix, including method details in Section A.1 additional qualitative examples in Section A.2 comparison to MILANNOTATIONS in Section A..3 extensive
provides good descriptions on these examples. Next in Section 4.2
ablation studics on each step of the DnD framework Scction A.4 an additional use case of DnD as an OOD classifier in Section A.6 and the capability to describe polysemantic neurons by producing multiple labels in Section A.7
4.1 Qualitative evaluation
We qualitatively analyze results of randomly selected neurons from various layers of ResNet-50, ResNet-18, and ViT-B-16. Sample results are displayed in Figure D and Figures 9 I0
13 15 and I6 in the 12
Appendix. We use the union of the ImageNet validation dataset and Broden D probe and compare to Network Dissection (Bau et al., 2017), MILAN (Hernandez et al.) 2022), and CLIP (Oikarinen & Weng 2023) as basclines. Labels for cach method are color coded by whether we believe they are accurate, somewhat correct, or vague/ fimprrcise. Compared to baseline models, we observe that DnD captures higher level concepts in a more semantically coherent manner. Specifically, methods such as CLIP-dissect and Network Dissection have limited expressability due to the use of restricted concept sets while MILAN produces labels confined lower level concepts. Additionally, we find that DnD can express multiple concepts within a single label owing to its generative nature.
4.2 Quantitative Evaluation
Final layer evaluation. Here we follow Oikarinen & Weng (0223 to quantitatively analyze description quality on the last layer neurons, which have known ground truth labels (i.c. class name) allow us to evaluate the quality of neuron descriptions automatically. To evaluate on accuracy, we measure the average CLIP and mpnet cosine similarity between ground trunh labels and generated descriptions on all output neurons of ResNet-50's 's final fully-connected layer. BERTScore metric mcasures the semantic interpretability of DnD descriptions. We focus this study on comparison with MILAN (Hernandez et al. 2022 as it is the other generative contemporary work in the baselines. Network Dissection (Bau et al. 2017) and CLIPP Dissect (Oikarinen & Weng 2023) are not included in this comparison because these methods have concept sets where the 'ground truth' class or other similar concepts can be inchuded, giving them an unfair advantage to the methods without concept sets like MILAN and DnD. We reported the results for all of the neurons of ResNet-50's final fully-connected layer in Table 2 Our results show that DnD outperforms MIL.NN, producing labels that are significantly closer to the ground truths than MILAN's.
4.3 Crowdsourced experiment
Setup. Our experiment compares the quality of labels produced by DnD against 3 baselines: CLIP-Dissoct, MILAN, and Network Dissection. For MILANN we used their most powerful base model in our experiments.
7

---

Published in Transactions on Machine Learning Research (01/2025)
Table 2:
Textual similarity between predicted labels and ground truths on the fully-connected layer of ResNet-50 trained on ImageNet. We can see DnD outperforms MILAN.
Metrie/ / Methods
MILAN
DnD (Ours)
CLIP
0.7080
0.7598
mpnet cos
0.2788
0.4588
BERTScore
0.8206
0.8286
Table 3: Averaged AMT results across four layers in ResNet-50 and ResNeet-18. We can see DnD outperforms existing methods on both ResNet-50 (ImageNet) and ResNet-18 (Places365). Our descriptions are consistently rated the highest and selected the best more than twice as often as the best baseline.
NesNet-50
ResNet-18
Metric/ / Method
NetDissect
MILAN
CLIP-Dissect
DnD (Ours)
NecDissect
MILAN
CLIP-Dissec
DnD (Ours)
Mean Rating
3.14+0.032
3.21-H0.032
3.67+0.028
4.152
3.35-
3.14++0.59
3.52+0.055
4.14+00.41
Selected Best
12.71%
13.29%
23.11%
50.89%
12.62
13.32%
19.39%
54.67%
We dissected both a ResNet-50 network pretrained on Imagenet-1K 1K and ResNet-18 trained on Places365, using the union of ImageNet validation dataset and Broden (Bau et al., 2017) our probing dataset. For both models evaluated 4 of the intermediate layers (end of each residual block), with 200 randomly chosen neurons per layer for ResNet50 and 50 per layer for ResNet-18. Each neurons description was evaluated by 3 different workers. In total, 3000 human ratings were conducted, 2400 evaluations on ResNet-50 and 600 evaluations on ResNet-18.
The full task interface and additional experiment details are available Appendix A.1.6 Workers were presented with the lop 10 highest activating images of neuron followed by four separate descriptions; each description corresponds to label produced by one of the four methods compared. The descriptions are rated a 1-5 scale, where rating of represents that the user "strongly disagrees" with the given description, and a rating of 5 represents that the user 'strongly agrees' with the given description. Additionally, we ask workers to select the description that best represents the 10 highly activating images presented. For these highly activating images, we used the images calculated by our method. As our probing dataset is a superset of the image sets used by prior methods, we believe our model is the most accurate for determining images to visualize since the probing dataset encapsulates the most concepts.
Results. Table 3 shows the results of large scale human evaluation study conducted on Amazon Mechanical Turk (AMT). Looking at *% time selected best" as the comparison metric, our results show that DnD performs over 2x better than all baseline methods when dissecting ResNet-50 or ResNet-18, being selected the best of the four up to 54.67% of the time. In terms of mean rating, our method achicves an average label rating over 4.1 for both dissected models, whereas the average rating for the second best method, CT.IP-Disect, only 3.67 on ResNet-50 and 3.52 on ResNet-18. Our method also significantly outperforms MILAN's generative labels, which averaged below 3.3 for both target models. Our method significantly outperforms existing methods in crowdsourced evaluation, and does this consistently across different models and layers.
4.3.1 MILANNOTATIONS evaluation
Though evaluation on hidden layers of deep vision networks can prove quite challenging as they lack 'ground truth' labels, one resource to perform such task is the MILANNOTATIONS dataset (Hernandez ct al.) 2022), which collects annotated labels to serve as ground truth neuron explanations. We perform quantitative evaluation by calculating the textual similarity between a method's label and the corresponding MILANNOTATIONS, Our analysis in Section A.3 found that if every neuron is described with the same
8

---

Published in Transactions on Machine Learning Research (01/2025)
constant concept 'depictions', it will achieve better results than any explanation on the dataset, but this is not a useful nor meaningful description. We hypothesize this is due to high levels of noise and interannotator disagreement, leading to low textual similarity between descriptions and generic descriptions scoring highly. We conclude that this dataset is unrcliable to serve as ground truths for comparing different methods.
4.4 Ablation Studies
4.4.1 DnD with fixed concept set
To analyze the importance of using generative image-to-text model, we explore instead utilizing fixed concept sets with CLIP (Radford al. 2021 to gencrate descriptions for cach image instead of BLIP, while use
the rest of the pipeline remains unchanged (i.e. using GPT to sunmarize etc). For the experiment, SLIP-VITB/BII 16, where we define L--) and E() text and image encoders respectively. From the initial concept set S {t,, 1 the best concept for image Im is defined as tt, where | = argmax; (L(44) Following CLIP-dissed (Oikarinen & Weng 2023). we use S = 20k (20, 000 most common English words] ]
(L(()) (((4)))(mm))
and D probe = TetgeNet U Broden.
To compare the performance, following Oikarinen & Weng (2023) we use our model to describe the final layer neurons of ResNet-50 (where we know their ground truth role) and compare description similarity to the class name that neuron detecting, as discussed in Section 4.2 Results in Table show that both methods perform similarly on the FC layer. In intermediate layers, we notice that single word concept captions from 20k significantly limit the expressiveness of DnD, suggesting generalive image descriptions is important for our overall performance. Qualitative examples and notable failure cases of CLIP descriptions can be found under Appendix A.4.2.
Table 4: Mean FC Layer Similarity of CLIP Captioning. Utilizing fixed concept set to caption activating images via CLIP (Radford et al. 2021), we compute the mean cosine similarity across fully connected layers of RN50. We find the performance of DnD w/ CLIP Captioning is slightly worse than BLIP generative caption.
Metric / Methods
DnD (Ours)
DnD w/ CLIP Captioning
% Decline
CLIP cos
0.7598
0.7583
0.197%
mpnet cos
0.4588
0.4465
2.681%
BERTScore
0.8286
0.8262
0.290%
4.4.2 Effects of Concept Selection
We use 50 randomly chosen neurons from each of the 4 layers of ResNet-50 to conducted an ablation study on the impact of Best Concept Selection (Step 3) on the pipeline. Evaluations are conducted on the first candidate concept. Each neuron was evaluated twice yielding a total of 400 human ratings. Table 5 shows the effect of Best Concept Selection on the overall accuracy of DnD. We see DnD performance is already high without Best Concept Selection, but Concept Selection further improves the quality of selected labels in Layer 2 through Layer 4, while having the same performance on Layer 1. One potential explanation is due to Layer detecting more limited low level concepts there is less variance in candidate descriptions identified in Concept Generation (Step 2), resulting in similar ratings across the set of candidate concepts T. We can see some individual examples of the improvement Concept Selection provides in Figure 3 with the new labels yielding more specific and accurate descriptions of the neuron. For example Layer Neuron 312 becomes more specific colorful festive sellings instead of generic Visual Flements.
2Suuree http://///tiiiiioo///////////o/o

---

Published in Transactions on Machine Learning Research (01/2025)
Layer Neuron 312
Layer Neuron 927
Activating Images
Activating Images
DnD Without Concept Selection: Visual Elements
DnD Without Concept Selection: Monochromatic and patterned designs.
Full DnD Pipeline: colorful festive settings
Full DnD Pipeline: abstract swirls
(a) Layer 2 Neuron 312
(b) Layer Neuron 927
Figure 3: Concept Selection (Step 3) supplements Concept Generation (Step 2) accuracy. We show that concept selection improves Concept Generation by validating candidate concepts.
Table Human evaluation results for DnD (w/o Best Concept Selection) versus full Describe-and- Dissect. Full pipeline improves maintains performance every layer in ResNet-50.
Method / Layer
Layer
Layer
Layer
Layer
All Layers
DnD (w/o Best Concept Selection)
3.54
3.77
4.00
4.02
3.84
DnD (full pipeline)
3.54
4.00
4.24
4.13
3.97
5 Use Case: Land-Cover Prediction
One important role of interprotability tools is the ability to create real-world impact. In this soction, we study applications in sustainability and climale change by applying DnD to the task of land cover prediction critical factor for managing water resources, conserving biodiversity, planning sustainable urban development, and mitigating climate change effects. DnD's concept descriptions for neurons not only improve the model performance, but also identify spurious correlations, suggesting a critical role of interpretability techniques in ensuring reliability and building trust in AT systems.
Setup. We evaluate our framework on two classification models. Tile2 (Jean et al. 2019) utilizes a modified ResNet-13 18 backbone trained to minimize triplet loss between anchor, neighbor, and distant land tiles from the NAIP dataset (Claire Boryan & Craig 2011). Following Jean al. (2019), we train random forest classifier (CDL) labels. We also evaluate ResNet-50 model trained on labeled EuroSAT images (Helber et. al. 2019 with 10 land cover classes. We experiment on two probing datasets. NAIP is an aerial imagery dataset updated anmually by the USDA. The dataset is composed crop cover data with annotated ground truth masks from CDL labels. However, due to the mne-grained nature of the 27 subclasses, we also categorize each subclass into six broad superclasses improve our understanding of the results: Herbaccous/Shhrutee /Shublandd, 3. Urban/Suburban, 4. Barren, Forest, 6. Water/wetlan wetlands. We use NAIP for all experiments the Tile2Vec ResNet-18 18 model. On EuroSAT ResNet-50, ImageNet U Broden datasets as the probing dataset.
ResNet-18 embeddings to evaluate accuracy 27 subclasses of Cropland Data Layer
Whantdd/Cultivalee
2.
the union of EuroSAT U
5.1 Locating Conceptual Groupings
We identify neuron clusters detecting similar concepts across Layer 2 of Tilc2Vec RosNet-18. For pair of candidate concepts sets for neurons n and m, we define their textual similarity Sn,m as the spatial mean of L(T,) L(T,)) where L(-) the LLIP-VIT-BIIC text encoder. A set of neurons considered similar if Sn. 2 x = 0.8, where & controls the minimum similarity threshold between concepts. For practicality, we confine each neuron to exactly one group highly similar neurons. Low-level concepts are then classified by GPT one-shot classification into the NAIP superclasses.
10

---

Published in Transactions on Machine Learning Research (01/2025)
Figure 4: Layer 2 Concept Profile. We cluster neurons with similar concepts and categorize them into NAIP superclasses. Interpretable concepts have more neurons associated with them. Some superclasses do not appear due to dataset bias or intrinsic similarities between classes.
Neurons per Concapt: layer2
plantedrculttivvee (21)
urban/suburban (21)
barren (3)
forest (1)
Neurons
Green feed with asigee:
Grean thrr day
Green stripied fanite
Grean Sorddr fotieg
Grean whili brordire
faiid birite
Greem berdered parterno
Green loorectsroee Bortiers
Green Block Itatured aeeciccppe
Grein and witte pisteert
Groen werthrbbrrineee conochhmmee
ferrain
Agric Agricultura Pazterm
sperts Held
Agricutinn sports Feeido
agricultur crec wwweeeerbbbrrrd brtbardd
muzedt acthvitess Lentttc
Mosaic
ppricuttuns mortiic
Andweade conttciinn fietie
Grem eotee patteric
Veriegered Natural Berraty
natural costure
acsosion toeturos
Blurred Neturel Sparreed
neut torons
Mixed Partor Natore
tartrganoe Green Laveiices cpee
Abviract Natural Elements
Uroan cootertionall Scoree
par
Corperate Beache widdie resteve
green bartter
Moreetrama Blue Accenti
Bescrrute facrrat
pattarty
Abstt fastracc mise carnnnnii por perdente freite
gran Neturr Pattee
nnggratie
Tuaree cood rature
stripcd spriuitura frests Forest with Cheerrngg
pathere
Green black criped
aemnocnna and
burdertdt tot
Conceot
Figure presents concepts from Layer 2, color-coded by superclass. We find that clusters (each bar in Figure containing more neurons are frequently associated with more interpretable features, while clusters with relatively less neurons are related to vague or irrelevant concepts. Based on these insights, we prune neurons with uninterprotable concepts in Section 5.2 We also note that "Herbaccons/Shrublaa
and "Water/wote wetlands' are not associated with concepts Layer 2. We hypothesize that this is likely due to the intrinsic similarity are closely
between the classes and bias in the NAIP dataset. Concepts related "Herbaceous/Shrubble superclass while 'Water /wetlawstwlss images only compriscs of 00.275% of
related to the the dataset.
5.2 Pruning Uninterpretable Neurons
Table Pruning uninterpretable neurons Tile2Vec ResNet18.
We conduct study to identify neurons within Vile2Vee ResNet-18 model that contribute minimally to model accuracy.
Layer
% Neurons Pruned Rand. Aoc.(%) Avg. Acc. (%)
Based on Section 5.1 we locate a subset of ungrouped neurons which correlate poorly to other neurons in the nerk. For our experiment, we define poorly correlating subsets as concepts that activate on only single neuron. Following the original implementation, train classifier embeddings from pruned
No pruning All pruned
0.00
71.63
100.00
35.96
Layer
23.44
69.3
71.07
Layer 2
50.00
68.96
71.54
Layer 3
26.95
70.13
71.75
Layer 4
58.98
69.95
72.04
Layer 5
56.45
70.99
71.76
Vile2Vec 18 model.
Table 6 shows model accuracy after identifying and pruning poorly correlating neuron subsets across each layer of the model. Due to high variance in the NAIP dataset, we evaluate the baseline accuracy for a fully pruned network. In this case, we find the prediction always the 'tomatoes" subclass which achieves an accuracy of 35.96% since this subclass comprises of ~3..2% of the data. Our results show that a significant proportion of neurons in the model do not contribute to the overall classification. Particularly in Layers 4 and 5, are able to prune over 50% of neurons in each layer while achieving result better than equal to the bascline performance (no neurons pruned). Across the entire network, the relative small difference in accuracy after pruning suggests human interpretable neurons account for more critical roles within image classification networks compared to unitterpretable neurons. Because we retrain the classifier on the pruned pipeline, the difference in accuracy using DnD pruning and that of random pruning fairly minor, but we can see random pruning consistently suffers an accuracy drop while DnD pruning does not. In Appendix A.5 conduct similar experiment with fixed classifier.
11

---

Published in Transactions on Machine Learning Research (01/2025)
5.3 Characterizing Spurious Correlations
We characterize spurious correlations in the intermediate layers of FuroSAT-trained ResNet-50 by determining common neuron labels through Term Frequency Analysis and studying their relationship to the task. We prune these neurons to further understand the class-wise correlation of spurious concepts, shown in Table 7 Though "fishing' is the most prevalent concept in Layer 4 (41.65% of all neurons), pruning them has no impact on model accuracy. In other words, these fishing neurons are irrelevant to the classification task. 'Pink' and "purple" account for 29.74% of Layer neurons and have a much greater impact. These concepts, which are seemingly unrelated the task, are spuriously correlated to the Forest, Herbaccous Vegetation, Industrial, Pasture, Residential, and Sea-Lake classes. However, the Annual Crop, Highway, and River classes have weaker correlations with these concepts.
Table 7: Pruning concepts from Layer 4 of FuroSAT-trained ResNet-50.
Concepts
% neurons pruned
Class-wise Accuracy (%)
Avg. Acc. (%)
pruned
Ann-Crrop
Forest
Herb- Herb-Veget. Highway Industrial Pasture Perm-CCrop Residential
River
Sea-Lake
No pruning Fishing
95.00
98.00
93.67
96.00
96.50
91.50
90,80
98.67
92.80
97.33
95.26
41.65
95.00
98.00
93.67
96.00
96.80
91.50
90.80
98.67
92.80
97.33
95.26
purppe
29.74
88.33
0.00
0.00
62.00
0.00
0.00
18.80
0.00
76.00
0.00
24.33
6 Conclusions
In this paper, we presented Describe-and and-Dissect (DnD), a novel method for automatically labeling the functionality of deep vision neurons without the need for labeled training data or a provided concept set. We accomplish this through three important steps including probing set sugmentation, candidate concept generation through the-tff- general purpose models, and best concept selection with carefully designed scoring functions. Through extensive qualitative, quantitative, and use-case analysis, we show that DnD outperforms prior work by providing higher-quality neuron descriptions, greater generality and flexibility, and significant potential for social impact.
Acknowledgments
This work is supported in part by National Science Foundation (NSF) awards CNS-1730158, ACI-1540112, ACI-1541349, OAC-1826967, OAC-2112167, CNS-2100237, CNS-2120019, the University of California Office of the President, and the University of California San Diego's California Institute for Telcommmications and Information Technology/ /Qualcomm Institute. Thanks to CENIC for the 100Gbps networks. This work used Expanse CPU, GPU and Storage at SDSC through allocation CTS230152 from the Advanced Cyberinfrastructtue Coordination Ecosystem: Services Support program, which is supported by National Science Foundation grants 2138259, 2138286, 2138307, 2137603, and 2138296. The authors thank REHS program (Research Experience for High School students) in San Diego Supercomputer Center. The authors are partially supported by National Science Foundation under Grant No. 2107189, 2313105, 2430539, Hellman Fellowship, and Intel Rising Star Faculty Award. The authors would also like to thank anonymous reviewers and the editor for valuable feedback to improve the manuscript.
References
David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Autonio Torralba. Network dissection: Quantifying interpretability of deep visual representations. In CVPR, 2017.
David Bau, Jun-Yan Zhu, Hendrik Strobelt, Agata Lapedriza, Bolei Zhou, and Antonio Torralba. Under- standing the role of individual units in deep neural network. Proceedings of the National Academy of Sciences, 117(48):30071-300 -3078, 2020.
12

---

Published in Transactions on Machine Learning Research (01/2025)
Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and Christopher Olah. Towards monose- manticity: Decomposing language models with dictionary learning. Transformer Circuils Thread, 2023. html.
Rick Mueller Claire Boryan, Zhengwei Yang and Mike Craig. Monitoring US agriculture: the US department of agriculture, national agricultural statistics service, cropland data layer program. Geocarto International, 26(5):341-38) -588 2011.
Dumitru Erhan, Yoshua Bengio, Aaron Courville, and Pascal Vincent. Visualizing higher-layed features of a doep network. 2009.
Gabriel Goh, Nick Cammarata, Chelsea Voss, Shan Carter, Michael Petrov, Ludwig Schubert, Alec Radford, and Chris Olah, Multimodal neurons in artificial neural networks. Distill, 6(3):e30, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2019.
Evan Hernandez, Sarah Schwettmann, David Bau, Teona Bagashvili, Antonio Torralba, and Jacob Andreas. Natural language descriptions of deep visual features. In ICLR, 2022.
Neal Jean, Sherrie Wang, Anshul Samar, George Azzari, David Lobell, and Stefano Ermon. Tile2Vec: Unsupervised representation learning for spatially distributed data. In AAAI, 2019.
Neha Kalibhat, Shweta Bhardwaj, Bayan Bruss, Hamed Firooz, Maziar Sanjabi, and Soheil Feizi. Identilying interpretable subspaces in image representations, 2023.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. BLIP: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In ICML, 2022.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping image pre-training with frozen image encoders and large language models. arXiv preprinl 2rXi::2301.1255 2023.
Jesse Mu and Jacob Andreas. Compositional explanations of neurons. Advances in Neural Information Processing Systems, 33:17153-17163, 2020.
Tuomas Oikarinen and Tsui- Wei Weng. CLIP-Dissect: Automatic description of neuron representations in deep vision networks. In ICLR, 2023.
Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction to circuits. Distill, 5(3):000024 001, 2020.
Nobuyuki Otsu. A threshold selection method from gray-level histograms. IEFE Transactions Systems, 197.
Mon, and Cybernetics, 9(1):62 66, 1979. doi:
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Tlya Sutskever. Learning transferable visual models from natural language supervision. In ICML, 2021.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Tlya Sutskever. Zero-shot text-to-image generation. CoRR, abs/2102.12092, 2021. URL https://arxiv.orgg abs/ abs/ 1ba22112.151 12092
13

---

Published in Transactions on Machine Learning Research (01/2025)
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, 2022.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet. large scale visual recognition challenge. International Journal of Computer Vision, 115:211-252, 2015.
Adam Scherlis, Kshitij Sachan, Adam S. Jermyn, Joe Benton, and Buck Shlegeris. Polysemanticity and capacity in neural networks, 2023.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. Laion-5b: An open large-scale datasct for training next generation image-text modcls, 2022.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Sounya Batra, Prajiwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. 2023.
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Object detectors emerge in deep scene CNNs. In ICLR, 2015.
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Antonio Torralba, and Aude Oliva. Places: An image database for deep scene understanding, 2016.
14

---

Published in Transactions on Machine Learning Research (01/2025)
A
Appendix
In the appendix, we discuss specifics of the methods and experiments presented in the paper and provide additional results. Appendix Section A.1 provides further details on the method pipeline, Section A.2 presents
additional qualitative evaluations, Section A.3 shows results for quantitative comparison to MILANNOTA,
TIONS, Section A.4 provides detailed ablation studies on each step of the DnD pipeline, Section A.5 extends
our land coverage use case experiment, Scction A.6 presents an additional use for DnD as an OOD classifier,
Section A.7 shows extension of DnD to return multiple concept labels, Section A.8 provides analysis of
the diversity of method's labels, Section A.9 showcasos failure cases of DnD, Section A.10 discusses the limitations and directions for future work, and Section
A.11 outlines the broader impact of our work.
App Appendix Outline
Method Details
(a)
A.1.1
Detailed Description of Attention Cropping
(b)
A.1.2
Holistic vs. Local Concepts
(c)
A.1.3
GPT Prompt
(d)
A.1.4
Scoring Function Details
(e)
A.1.5
Selecting a Scoring Function
(f)
A.1.6
Amazon Mechanical Turk Setup
2.
A.2
Qualitative Evaluation
3.
Quantitative Results: MILANNOTATIONS
4.
A
Ablation Studies
(a)
A.4.1
Attention Cropping
(b)
A.4.2
Image Captioning with Fixed Concept Set
(c)
A.4.3
Image-to- -enetto-Textttto Model
(d)
A.4.4
Multimodal GPT
(e)
A.4.5
Effects of Large-Tanguage-Mood Choice
(f)
A.4.6
Effects of GPT Concept Summarization
5.
A.5
5
Use Case: Fixed Classifier
A.6
Additional Use Case
7.
A.7
7
Multiple Labels
8.
A
g
Diversity Analysis
A.9
.9
Failure Cases
10.
A.
10
Limitations
11.
Broader Impact
15

---

Published in Transactions on Machine Learning Research (01/2025)
A.1 Method Details
A.1.1
Detailed Description of Attention Cropping
The allention cropping algorithm described in Section 3.1 is composed of three primary procedures:
Step 1.
Compute the optimal global threshold À for the activation map of given neuron using Otsu's Method
Otss 1979.
Step 2. Highlight contours of salient regions the activation map that activate higher than A.
Step 3. Crop a largest salient regions from the original activating image that have an IoU score less than y with all prior cropped regions.
Here, we provide more implementation details for each step:
Step 1. Global Thresholding using Otsu's Method. For activation map of neuron n, we define regions with high activations as the "foreground' and regions with low activations the "background'. Otsu's Method (Otsul 1979) automatically calculates threshold values which maximizes interclass variance between foreground and background pixels. Interclass variance of is defined by o = W&W (Ps - uy)², where Wo, denotes weights background and foreground pixels and μb, denotes the mean intensity of background and foreground pixels respectively. For global thresholding used in DnD, W, is the fraction of pixels in activation map (x,) above potential threshold X while W, is the fraction of pixels below A. A value of À that maximizes z is used the global threshold.
Step 2. Highlight Contours of Salient Regions. DnD utilizes OpenCV's contour detection algorithm to identify changes within image color in the binary masked activation map. Traced contours segments are compressed into four end points, where each end point represents a vertex of the bounding box for the salient region.
Step 3. Crop Activating Images. Bounding boxes recorded from Step are overlaid on corresponding activating images from Dprobes which are cropped to form Deropped- To prevent overlapping attention crops, the IoU score between every crop in Deropped is less than an empirically set parameter 7), where high 1) values yield allention crops with greater similarities. We use n 4 across all experiments.
We also visualize these steps in the below Figure for better understanding.
16

---

Published in Transactions on Machine Learning Research (01/2025)
Original Activation Map
Step 1: Otsu Thresholding
Step 2: Draw Bounding Boxes from Contours
Step 3: Crop Activating Images
Activating Images in D probe
D Deropped
Figure 5: Detailed Visualization of Attention Cropping Pipeline. All three steps of attention cropping are shown for Layer 2 Neuron 165. Steps 1 and 2 illustrate the derivation of bounding boxes from salient regions in the original activation map and are overlaid on original activating images from D probe in Step 3.
Cropped images are added back to D probe to form proobe U D Deropped-
17

---

Published in Transactions on Machine Learning Research (01/2025)
A.1.2 Holistic Local Concepts
improve the accuracy of DnD, we form the probing dataset with cropped images representing local concepts and uncropped images representing holistic concepts. We define local concepts as concepts contained to a portion of the image and holistic concepts as emergent concepts represented by the whole image. This is necessary solely using image crops results in vague spuriously correlated descriptions. We show two examples in Figure D In Layer Neuron 64, uncropped images help produce nuanced concepts: cropped images only represent "blue" while uncropped images reflect the "aquatic" concept. Layer Neuron 120 presents an example where cropped images activate on highly localized regions, making the image-to-text model prone to produce spurious captions. Uncropped images (last two images) make the "canine" concept more evident.
Layer 1 Neuron 64
Layer 3 Neuron 120
DnD: blue aquatic objects
DnD: canine and animal features
(a) Layer Neuron 64
Layer Neuron 120
Figure 6: Holistic vs Local Concepts. We show two cases where a union of local and holistic concepts helps improve model accuracy. Augmenting the dataset reduces vague and spurious captioning.
18

---

Published in Transactions on Machine Learning Research (01/2025)
A.1.3 GPT Prompt
The Concept Selection process utilizes GPT to identify and coherently summarize similarities between image captions of every image Due to the generalive nature DnD, engineering precise prompt is crucial the method's overall performance. The use of GPT in DnD can be split into two primary cases, each with individual prompt: 1) Summarizing image captions to better bring out the underlying concepts, and 2) Detecting similarities between the image captions. We explain each part separalely.
1. Summariking Image Captions. BLIP's image captions frequently contain extraneous details that detract from the ideas represented. To resolve this problem, captions are fed through GPT and summarized into simpler descriptions to represent. the concept. We use the following prompl:
"Only state your answer without a period and quotation marks. Do not number your answer. State one coherent and concise concept label that simplifies the following description and deletes any unnocessary details:"
2. Detecting Similarities Between Image Captions. To describe the similarity between image captions, we design the following prompt:
"Only state your answer without a period and quotation marks and do not simply repeat the descriptions. State one coherent and concise concept label that 1-5 words long and can semantically summarize and represent most, not necessarily all, of the conceptual similarities in the following descriptions:
In addition to the prompt, we use few shot-prompting to feed GPT two of example descriptions along with the respective numan-identtiiee similarities:
Example 1:
Human-Identified Similarity: 'multicolored textiles" Image Captions:
purple background with a very soft texture"
brown background with a diagonal pattern of lines and lines"
'a white windmill with red door and red door in the middle of the picture'
beige background with a rough texture of linen"
'a beige background with rough texture and very soft texture'
Example 2:
Human-Identified Similarity: 'red-themned scenes"
Image Captions:
a little girl is sitting in a red tractor with the word sofy on the front"
toy car sits on red ottoman in play room"
red dress with silver studs and a silver belt"
'a red chevrolet camaro is on display at show'
rod spool of a cable with the word red on it'
We also include the prompt used for our Use Case one-shot-clasiffcation.
19

---

Published in Transactions on Machine Learning Research (01/2025)
3. Use Case Superclass Classification. we use the following prompt:
To classify neurons into broader NAIP superclasses
'State one coherent and concise concept label 1-5 words long related to land- scapes/satellite imagery that semantically summarizes and represents most, not necessarily all, of the conceptual similarities in the following descriptions. Focus on colors, textures, and patterns. After, print most likely natural landscape pe described by the satellite imagery captions, in parenthesis, the same line. Be confident and do not be vague:
20

---

Published in Transactions on Machine Learning Research (01/2025)
A.1.4 Scoring Function Details
Though scoring functions all seek to accomplish the same purpose, the logic behind them can vary greatly and have different characteristics. Simplest scoring functions, such as mean, can be easily skewed by outliers in Hj, resulting in final concepts that are loosely related to the features detected by neurons. In this section, discuss three different scoring functions and combination of them which we experimented with in Section The concept with highest score is chosen as the final description for neuron n.
A.1.5
Mean. Simply score concept j as the negative mean of its image activation rankings Hj. Concepts where each image activates the neuron highly will receive low ranks and therefore high score. We use the subscript M denote it's the score using Mean.
Q
scoTe (HG) = Q 1m1
Hy
TopK Squared. Given Hj. the score is computed as the mean of the squares of 3 lowest ranking images for the concept. For our experiments, we use = 5. This reduces reliance on few poor images. We use the subscript TK to denote the score using TopK Squared:
9
ScoreT (OoTTTK(HH,3)) = - Σ
1
H3
1=1
Image Products. Let set Dj C Dj, such that it keeps the & highest activating images from Dj- From the original set of activating images I and Dj, the Image Product score is defined as the average cosine similarity between original highly activating images and the generated images for the concept j. We measure this similarity using L.IP-ViT-B/ B/16 Radford et al. [2021] as our image encoder E(-):
score)((1. D))
Σ
Σ
(E(x) E(xasw)
1 IDJ
RET gROWED!
See Figure 7 for illustration of Image Product. Intuitively, Image Products selects the candidate concept whose generated images are most similar to the original highly activating images. However, Image Product doesn't actually account for how highly the new images activate the target neuron, which is why we chose to use this method to supplement other scoring functions. The enhanced scoring method, TopK Squared + Image Products, is described below.
TopK Squared + Image Products. This scoring function uses both image similarity and neuron activations to select the best concept. The method combines Image Products and TopK Squared by multiplying the Image Product score with the relative rankings of each concept's TopK Squared score. NJ} as the set of TopK Squared scores for different We define RTK- = scoтeTk (H,, 3), V descriptions. The final score is then:
(N Rank(soorerT (H,, RTK)) scorer DJ),
where we use N - -Rank(-) to invert the ranks TopK Squared so low ranks result in a high score.
In Section A.I.5 we compare the different functions and note that our model largely robust the choice of scoring function with all performing similarly well. We use the TopK Squared + Image Products scoring function for all experiments unless otherwise specified.
21

---

Published in Transactions on Machine Learning Research (01/2025)
Image Encader
P
P
P
Px
W
W
P
W
W.P.
W P
W
P.
wPP.
wpp
P
Candidate Concept
Image Encoder E
PP
PP
W Px
Score
Compute Mean
Dj
:
W.P.
W.P.
WPP
WPP K
Figure 7: Image Product Scoring Function. In the diagram, we let W, = E(Dj) and P. E(4). Image Products computes the mean of W; P..
22

---

Published in Transactions on Machine Learning Research (01/2025)
A.1.5 Selecting Scoring Function
We again use ResNet-50 with Imagenet U Broden as the probing dataset. 50 neurons were randomly chosen from each of the hidden layers, with each neuron evaluated twice, again rating the quality of descriptions on a scale 1-5. The participants in this experiment were volunteers with no knowledge of which descriptions were provided by which methods. Table shows the performance of different scoring functions described in Section A.1.4 We observe that all three scoring functions presented perform similarly well, and shows our algorithm is robust and not dependent on the choice of scoring function.
Table 8: Comparison of scoring functions. Total of 276 evaluations were performed on interpretable neurons (neurons deemed uninterpretable by raters were excluded) from the first 4 layers of ResNet-50 on scale from to 5. We observe a small margin of difference between the total averages of each scoring function's 's ratings. Across all 4 layers, the worst and best performing scoring functions differ in average rating by only 0.04 so our algorithm not dependent on the choice of scoring function.
Scoring Function
Layer
Layer
Layer 2
Layer
Layer 4
All Layers
Mean
3.95
3.92
4.14
3.82
3.95
Topk Squared
3.73
3.91
4.23
3.80
3.91
TopK Squared + Image Products
3.77
3.94
4.28
3.78
3.93
23

---

Published in Transactions on Machine Learning Research (01/2025)
A.1.6 Amazon Mechanical Turk Setup
In this section we explain the specifics of our Amazon Mechanical Turk experiment setup in Section 4.3
the interface shown Figure [ we display the top highly activating images for a neuron of interest and prompt workers to answer the question: "Does description accurately describe most of the above 10 images?" Each given description is rated on 1-5 scale with 1 representing "Strongly Disagroe', 3 representing 'Neither Agree nor Disagree', and 5 representing "Strongly Agree'. Evaluators are finally asked to select the label that best describes the set of 10 images.
We used the final question to ensure the validity of participants' assessments by ensuring their choice for best description is consistent with description ratings. If the label selected as best description the final question was not (one of) the highest rated descriptions by the user, we deem the response invalid. In our analysis we only kept valid responses, which amounted to 1744/2400 total responses for Resnet-50 and 443/600 responses for ResNet-18 (around 75% of the responses were valid)..
We collected responses from workers over 18 years old, based in the United States who had > 98% approval rating and more than 10,000 approved HI'ss to maximize the quality of the responses. We paid workers $0.08 per response, and our experiment was deemed exempt by the relevant IRB board.
One downside of our experimental sctup that not clear which highly activating images we should display to the users and how display them, although these choices may be important to the final results. In our experiments, displayed the images in probe with the highest mean activation done by CLIP-Dissect
Oikarinen & Weng, 2023), but displaying images with highest max activation would likely be better for MILAN (Hernandez et al. 2022). Unfortunately there is no simple answer to which images we should display that would be good for all methods. See Limitations A.10 for more discussion.
24

---

Published in Transactions on Machine Learning Research (01/2025)
Study Information
Click to View Study information
By checking this box indicate that am least years old, have reed the study information above, and agree to per cearricipstee this research study.
Task
Rate your agreement with the statement: "description' accurately describes most of the above 10 images, for each of the descrittios below. If you don't understand the description we encourage you to find out using for example dictionary or Google image search.
Does description accurately describe most the above 10 mages?
Description "blinds"
Strangly
Nelther
Strongty Agree
Description "srid"
Strongly Disagree
Disagree
Neither Agree Disagree
Agree
Strongly Agree
Description "Red and white colored
Strangly Disagron
O Disagroe
O Nether Agran nor Disagron
O Agron
O Strangly Agron
Description "monochrome patlerns"
Strongly Disagree
Disegree
Nelther Agree nor Disagree
Agree
Strongly Agree
Which description is the best match for the sat of images?
"binnss"
"grid"
"Red and while colored objects"
"monactrome patterns"
Submit
Figure 8: Amazon Mechanical Turk experiment user interface. For each neuron, present participants with the top 10 highly activating images and prompt them to rate each method's label based by how accurately the description represents the set of images. We also ask participants to select the best label from the four descriptions presented.
25

---

Published in Transactions on Machine Learning Research (01/2025)
A.2 Qualitative Evaluation
Figures 9 10 and 12 provide supplementary qualitative results for DnD along with results produced by baseline methods for ResNet-50 and ResNet-18. Figure [3 showcases our descriptions on random neurons of the ViT-B/16 16 Vision Transformer trained on ImageNet, showing good descriptions overall and higher quality than those CLIP-Dissect. Figure 14 extends DnD to ResNet-152, demonstrating the ability to scale to larger vision networks. Figures] 15 and 16 we dissect ResNet-50 using the CTFAR-100 training image set. D probes showcasing our method still perfoms well with different probing dataset. Finally, we present a qualitative comparison with FALCON Kalibhat al. 2023 in Figure 17
ResNet-50 (ImageNet)
Layer
Layer 2
Neuron: 108
Neuron:
DnD: blse aquatic and artistic elements IP-Diewwt underwaler
MILAN Blue oijjects Nad- Nwi-Drioeez:: mountein (soins)
DnD: patterned designs
ML AN Spots of the oolor while
CL. P-Disowd:: dots
Not-Dinsect while (ctler)
Neuron: 241
Neuron 158
black white animal Dissact
Horizondal
DnD: and orange
Not-Dissect: (color)
Neuron:
Nauron:
DnD: yellow and white patterns
MILAN:
Spots of the color while
DnD: nighttime Imagery
MILAN: Edges
CLIP-D Diseect defenses
Beo--D chequeeeed
CL P.DDDDasdd:: Blumination
Net-Diseect black (color)
Nouron: 109
Neuron: 150
DnD: lnxtile and metal patterns
MILAN: Stripes
DnD: patterns and textures
MILAN: Vertical lines NotDissedt
CLIP-D Disectt lodile
Net-Dissest: agoagged
DiDsset blinds
link-chemed backgrounds
MLAN: Nes-Dlesedt:ne (color)
objects
scenio objects
MILA: Specific
CLIP D--Dssect magenta
CLP P-Diooedt: stmospherio
Nee-Dlssect painting
Neuron:211
Nouron: 478
Bnb: dark imagery with lights and naturn CLIP iewdi darknwse
MILANI Harizonaal ines Nel-Diseect while (ooker)
BnD: patterns and objects colors
Mi AN: Metel objects Not-Dissect: stripe
CL CLi-Diowd:: zubra
Neuron:
black white audio-visuals
Words
display miscollansous
Dissesttct
NeteOiseect:
Ble-elssect:oo
Figure 9: Additional examples of DnD results from Layer and of ResNet-50. We showcase set. of randomly selected neurons and their descriptions from Layer and 2 of ResNet-50 trained on ImageNet-IKK 1K. Labels are color-coded by whether we believed they were accurate, somewhat correct correct/vague vaguee or imprecise.
26

---

Published in Transactions on Machine Learning Research (01/2025)
ResNet-50 (ImageNet)
Layer 3
Layer
Nauron 838
Neuron: 360
OnD: structures and materials
AN: The or walls Diss-ttstctttoo leather
DnD:: tools and activities
MILAN: Curved bamns Net Bes-Dissect: spralled
CLIP P-Dissect latiic
CLIP-OIssect physiclans
patterns
objects
geometric patterns CUI- Cls-Olssect: leatle
of things
CLIP CUP-Dissect pattem
Net-D issect digragged
Net-Dtooed: algragged
Neuron: 780
Neuron: 1989
DnD: pattemss and textures IP-Dissect
MILAN: Faces of animals
DnD: monochrome and metailic objects
MILAN Animals and circuar objects
Ne--Disectt wheel
CLI--Disset: zirconta
Net-Deesact:
428 hallways
MILAN: Diagonal
CLP--Disect: comiooes
Net-Oissect: florr
CUP-Diseect: hortioullure
Net-Drowed:: plant
Nauron: 187
Neuron: 1674
DnD: various Images of subjects UP-Dbewect regard
MILAN: Bird heads
DnD: colortul objectsiscenes
MIAAN Rounced edges in pictures
Nel-Dloeech nack
GLIP-Dised toester
Nat-Daeadd: car
sightings CLI--Dissec::
CLiP-Disect quils
Nel-Diesech paper
Net-Disend:
Nouron: 977
Neuron: 1210
DnD: red and white polka dots
MILAN: Red objects Net-Dieeech red (color)
DnD: hotel accommodations
MIAN Beds
CL Dieeed red
CLI--Disse:: bedrooma
Noi-Desedd bed
Figure 10: Additional examples of DnD results from Layer 3 and 4 of ResNet-50. We showcase set of randomly selected neurons and their descriptions from Layer 3 and 4 of ResNet-50 trained ImageNet-1K. Labels are color-coded by whether we believed they were accurate, somewhat correct /vague imprecise.
27

---

Published in Transactions on Machine Learning Research (01/2025)
ResNet-18 (Places365)
Layer
Layer 2
Neuron:1 19
Neuron: 106
OnD: striped pattems various colors
DnD: colortul pattems and textures CL P-Drssect garrison
CLP-Daowwd seectt berade
MILAN: Dots the
MIL AN: Red dote
objects
social
objacts
surveying
Crean objects
Neuron:2
Neuron
DnD: colorful holiday pattemn
DnD: stripes and rainbows
CLIP P--Disae seedt greed
MILANc Pink colored objects
CLP:Diseest hostels
MILN: Colorful dots
Neuron:11 18
Neuron: 51
black white patterns
DnD: textures of Industrial materials P-Dissed:
Neuron:
DnD: colorful patterns
BnD: various green stripes
CLIP D'socot: q.ils
MILAN: Rod blue objects
CLIP Disoott stripd
MILAN: Vertical lhhes
Neuron: 26
Neuron 103
DnD: wallets and patterns
DnD: patterned surfaces
CLIP-Disocc wallet
Red
Dissect texdlle
Circular
patterned backgrounds
CLIP-D
Aquarium scenes
CLi-Disecct greed
The right side
00000000
Figure 11: Additional examples of DnD results from Layer and 2 of ResNet-18. We showcase sot of randomly selected neurons and their descriptions from Layer and 2 of ResNet-18 trained Places365. Labels are color-coded by whether we believed they were accurate, somewhat correct/ correct/vague vague imprecise.
28

---

Published in Transactions on Machine Learning Research (01/2025)
ResNet-18 (Places365)
Layer 3
Layer 4
Nouron: 121
Neuron
DnD: iestures and atructures CLIP-Dieed moeque
DnD: buildings and rooma CLIP-Dieldd conidors
MILAN Arches and arches
MILAN Windows
everyday objects
CLIP-Deewct dipboaad
ground
CLiP-Diseet: compartment
Nouron: 208
Neuron:
DnD: water-relsted activities
DnD: military and aviation equipment; furniture
CLIP-Da Di-sestttt: rafting
MILAN: Water the water
CLIP DUP--Disect: aviation
MILAN: Water and planes
243
DnD: settingg
CLIP-Dissact:
CLI--Disoct: furnished
Nouron:
Neuron
DnD: black and white textures/pattems
DnD: assorted visualls
CLIP Diss-dtttttt mash
MILAN: The ground
CLI- Dip--it sihouette
MILAN: Green
hormy-inspired
objectt
featured/pattermss
ground
Dissect:
pattem objects
Nouron: 197
Neuron: 42
DnD: earthy fextures and objects
DnD: bedroom decor
CLIP-Disect: drought
MILAN: Grass
CLIP-Disoact bodrooms
MILAN: Bods
D0
Figure 12: Additional examples of DnD results from Layer and 4 of ResNet-18. We showcase a set of randomly selected neurons and their descriptions from Layer and ResNet-18 trained on Places365. Labels are color-coded by whether believed they were accurate, somewhat. porrect/vaguue imprecise.
29

---

Published in Transactions on Machine Learning Research (01/2025)
ViT-B/16 (ImageNet) Encoder
Neuron: 185 DnD: kitchen activities
Neuron:
CLIP-Dissect kitheens
DnD: kitchen and stone themes
CLIP-Dissecs kitchens
Neuron: 325 dogs
CLIP-Diseect kitchen
arrangements
CLIP-Dissect room
Neuron: 522
Neuron: 122
DnD: bathroom household activities
CLIP-Dissect: bathroom
DnD: flowers and flelds
CLIP-Dissect fields
DnD: bedroom furnishings
CLi-DDisect: bedrooms
DnD: kitchen scones
CLip-Diseect kitchess
Neuron: 226
Neuron: 507 home
livingg room settings
DisseDtstett
CLIP-Dissect:
Neuron: 457
Neuron: 665
DnD: their
CLiP-Dissect
DnD: bathroom and kitchen fixtures CLIP-Dissect: bathroom
DnD: domestic activities
CLIP-Dissec: kitchen
DnD: food and dining scenes
CLIP-Dissect: kitchen
Figure 13: Examples of DnD results from the encoder layer of ViT-B/16 (ImageNet). We showcase set of randomly selected neurons and their descriptions from the encoder layer of ViT-B/16 trained ImageNet. DnD is model agnostic and can be generalized to any convolutional network. Labels are color-coded by whether we believed they were accurate, somewhat correct paaect/vagge /vague or imprecise.
30

---

Published in Transactions on Machine Learning Research (01/2025)
ResNet-152 (ImageNet)
Layer 1
Layer 3
129
Neuron 301
DnD: purple and pink visuals
DnD: mountain scenery
Neuron 153
Neuron 500
DnD: assorted texiiles and patterns
DnD: water scenes
Neuron 235
Neuron 766
DnD: blue and green backgrounds
DnD: interior spaces
Layer 2
Layer 4
Neuron 67
Neuron 216
DnD: puzzle theme
DnD: theater/stage decor
Neuron 144
Neuron 907
DnD: blue-themed accessories pattems
DnD: aviation and transportation
Neuron 373
Neuron 1732
DnD: textiles and
Figure 14: Examples of DnD results from ResNet-152 (ImageNett. We showcase set of randomly selected neurons and their descriptions across all layers of ResNet-152 trained on ImageNet. DnD is scalable to larger vision networks.
31

---

Published in Transactions on Machine Learning Research (01/2025)
ResNet-50 (CIFAR100 Train)
Layer
Layer
13
Neuron
DnD: dark-Ihemed Imagery
Nothing
DnD: oceanic and celestinl scenes
MILAN: The edges things
CLIP-Dissect: darkness
Dissect: black-o
CLIP-D DiP-Distect: habitat
Nat-Dissect grass
Neuron 42
Neuron 30
backgrounds
CLIP-Dissec:: stripes
Nat-Dissect: blacked
CLIP-Dissed:: processor
Nat-Disseci: perforated
Neuron 108 CLIP-Dissect: shark
Neuron 69
objects
colorful objects
Spots
Net-Dissect: mountain mnuurtainssnoyy--
CLIP-Dissac: quitts
Nel-Dissect: detied
Neuron 159 yellow CLIP-Dissect: yellow
objects Net Dissect:::::ooo orange-o
DnD: yellow and orange motitss CLIP-Diesect zebra
MILAN: Orange aroas in picture Net-Dissect haneycombed
DnD: blue themed scenes
MILAN: and red lines Ned-Dissect banded
DnD: yellow-themed flora and patterns CLIP-Dissect:
MILAN: Yellow objects
CLIP-Diseedt snorkaling
Nos- Nos-Dissest
Neuron 223
Neuron 311
DnD: green-themed imagery
MILAN: Neon
human portraits
MILAN: Brighter objects
CLIP-Dissect: green
CLIP-Diasect wicker
Nel-Dissect striped
DnD: colorful nature acanes CLiP-Dissect: pattern
MIL ANI Horizontal lines Net Dissect: zigsagged
DnD: blua-themed designs
MIL INI Spots the oolor while Net-Dissect datied
CLIP-Dissect: dats
Figure 15: Examples of DnD results from Layer and of ResNet-50 using the training dataset of CIFAR100 as the probing image set. We showcase set of randomly selected neurons and their descriptions from Layer and of ResNet-50 trained on ImageNet- DnD is probing dataset agnostic and can maintain its high performance on other image sets. Labels are color-coded by whether we believed they were accurate, somewhat correct/ vague or imprecise.
32

---

Published in Transactions on Machine Learning Research (01/2025)
ResNet-50 (CIFAR100 Train)
Layer 3
Layer 4
Neuron 85
Neuron 200
backgrounds
sky scenes CL IPD-Dassdd aking
D.i--itstct: grid
Net- Mos--issact:noo lined
Net- Med-Diescct: mountain
objects verticality
DnD: scenic views CLIP-Dissect: captive
MIL AN: Dog noses Net Dissect:
MILAN: Rounded edges In pictures
CLIP-Dissect fungi
Neb Net-Dissect: lighthouse-s
Neuron
Neuron 942
DnD: pink and yellow flowers/fruits
MILAN: Babies and white skin Net-Dissect:
DnD: wooden furniture CLiP-Diseed room
MILAN: Edges of objects
CLIP-Dissedt: carbohydrates
Nel- Dissect headbooad
Nouron 231
Neuron 1122 CLIP Dissocts pond
DnD: monochrome and color accents
MILAN: Cirouler objects
MILAN: Not Dissect: bathtub
Neuron 384
1725
DnD: bathroom
MILAN: Circular objects
DnD: table lamp and potted plants CLIP-Diseect bethroom
MILAN: Rounded edges pictures Mek-Dieeect: tollet
CLIP-Diseec: bottles
Net Dissedt
Neuron 857
Neuron 1877 DnD: display screens CLiP.Dssecct radiator
DnD: white animall and objects
MILAN: White skin
MILAN Animals and grates
Neuron 847 DnD: sunset
Neuron 2038
MILAN: the hollowed
DnD: Interlor design elements I.i-Dissect bathroom
MILAN: Fire
Net- shade
Net-Dissect skyscraper
Figure 16: Examples of DnD results from Layer 3 and 4 of ResNet-50 using the training dataset of CIFAR100 as the probing image sct. We showcasc set of randomly selected neurons and their descriptions from Layer and of ResNet-50 trained on ImageNet- DnD is probing dataset. agnostic and can maintain its high performance on other image sets. Labels are color-coded by whether we believed they were accurate, somewhat parrect/vaguu or improcise.
33

---

Published in Transactions on Machine Learning Research (01/2025)
Layer
Layer
Layer
Top Activating Images:
Layer
Top Activating Images:
DnD: yellow themed eloments objects
Dis-eDssed:: yallow
DnD: green leaf variations MILAN: green areas picture FALCON: leaf
CLIP-Dissect leaf Nat-Dissech:
Net-Dissect: striped
FALCON: bamboo graes, reed, green Layer Neuron 163
Top Activating Images:
Layer Neuron 145
Top Activating Images:
DnD: pallerned lextures
CLIP GLIP-Dissedt: dotted Net-Dissect: banded
DnD: chockered pattoms
CLIP-Dissact: homepago Net-Dissact:
MIL AN:
numbers and lellers
MILAN: words FALCON: horizontal, note,
FAI CON: texture, dark, traeline
Layer
Layer
Nouron 322
Top Activating Imeges:
Leyer
Activating Images:
MILAN: vertical lines
FALCON: bar, fonco. window Layer
stripe patterns
Net- Dissect: white-c
CLIP. Dissect stripes
FALCON: burning fire
MILAN: fire
concopts
Nat-Dissect: body
CLIP-Dissact combustion
Top Activating Images:
Layer Activeting Images:
FALCON: blue, lank, school
dots
snorkeling Net- Dissact: zigraggad
FALGON: blue jean, denim pant
clething
fashion
Nor- Doei-Diasctt: walartal-aiocka
CLIF-Dissect jeens
Figure 17: Examples of DnD results compared with FALCON results from layers across ResNet-50 (ImageNet). We showcase a set of randomly selected neurons provided in the appendix of Kalibhat et al. (2023) and their descriptions from layors across ResNet-50 trained on ImageNet-1K. DnD performs well compared to FALCON and other contemporary methods.
34

---

Published in Transactions on Machine Learning Research (01/2025)
A.3 Quantitative Results: MILANNOTATIONS
In this section we discuss using the MILANNOTATIONS dataset (Hernandez et al. 2022) as set of ground truths to calculate quantitative results on the intermediate layers of R55 Table 9 displays an experiment donc on 103 randomly chosen 'reliable' neurons across the 4 intermodiate layers of ResNct-152. A neuron was deemed 'reliable" if, out of its three corresponding MILANNOTATIONS, two had a CLIP cosine similarity exceeding a certain threshold (for the purposes of this experiment we set that threshold to 0.81). Additionally, we define our summary function g be spatial max, as this what was used to calculate the highest activating images for the MILANNOTATIONS We compare against MILAN trained Places365 and CLIP-Dissect, using the spatial max summary function for CLIP-Dissect as well. Our results are generally mixed, with DnD performing the best (average of 0.7766), then MILAN (average of 0.7698), and finally CLIP-Dissect (average of 0.7390) using CLIP cosine similarity. These results match with BERTScore, as using that metric DnD performs the best (average 0.8489), then MILAN (average of 0.8472), and finally CIIIP-issect (average of 0.8368). With mpnet cosine similarity, CLIP-Disscct is caleulated as the best (average of 0.1970) followed by MILAN (average of 0.1391) and then DnD (average of 0.1089). However, the MILANNOTATIONS dalaset is very noisy seen in Figure 18 and thus is largely unreliable for evaluation. The dataset provides three annotations per neuron, each made by different person. This causes them to often be very different from one another, not representing the correct concept of the neuron. We can see in the table that the average CLIP cosine similarity between the annotations for each of these reliable neurons is 0.7884, the average mpnet cosine similarity is 0.1215, and the average BERTScore is 0.8482. Due to this noisiness, we found that the best performing description is as generic as possible as it will be closest to all descriptions. In fact, we found that trivial baseline describing each neuron as 'depictions' irrespective of the neuron's 's function outperforms all description methods, including MILAN. This leads to believe MILANNOTATIONS should not be relied upon to evaluate description quality. Figure 18 showcases descriptions and MILANNOTATIONS for some neurons.
Table 9: Similarity between predicted labels and "ground truth" MILANNOTATIONS on the 4 intermediate layers of ResNet-152 trained on ImageNet. We observe that when using CLIP cosine similarity and BERTScore, DnD performs the best followed by MILAN and then CLIP-Dissoct, but when using MPNet. cosine similarity, CLIP-Dissect performs the best followed by MILAN and then DnD. Simply labeling every neuron as "depictions' outperforms all other methods, demonstrating the unreliability of MILANNOTATIONS as an evaluation method. The set's noisiness is also shown by the average similaritics between the different annotations for each neuron, demonstrating that the annotations are not consistent. We round results to the nearest 4 decimal places, but in instances of ties within row, we observe further digits and bold accordingly.
Method
Metric
Layer
MILAN
CLIP-Dissoc
Describe-and-DDiee (Ours)
"depiction"
Avg Cos Sim
Between Amnotations
CLIP cos
Layer 1
0.7441
0.7398
0.7454
0.7746
0.7643
Layer 2
0.7627
0.7472
0.7872
0.7988
0,7976
Layer 3
0.7893
0.7414
0.7858
0.7900
0.8026
Layer 4
0.7739
0.7291
0.7825
0.7820
0.7845
All Layers
0.7698
0.7390
0.7766
0.7864
0.7884
ippuet cos
Layer 1
0.0801
0.1962
0.0950
0.2491
0.1124
Layer 2
0.1143
0.1850
0.1075
0.2513
0.1167
Layer 3
0.1462
0.2039
0.1160
0.2483
0.1279
Layer 4
0.1973
0.1995
0.1132
0.2501
0.1252
All Layers
0.19
0.1970
1089
0.2496
0.1215
BERTScore
Layer I
0.8426
0.8282
0.8395
0.8513
0.8392
Layer 2
0,8472
0.8348
0.8475
0.8597
0.8467
Layer 3
0.8519
0.8438
0.8519
0.8580
0.8490
Layer 4
0.8456
0.8374
0.8539
0.8475
0.8501
All Layers
0.8472
0.8368
0.8489
0.8541
0.8482
35

---

Published in Transactions on Machine Learning Research (01/2025)
Layer
Layer
Layer Neuron 55
Top Activeting Images:
Layer Neuron 107
Top Activating Images:
DnD: green and yellow imagery
DnD: pubíi transportation
MILAN:
Yallow and blue colored objects
MA1: grass and foliage MA2: monies some trees
MA1: windows or window like things
MAZ: glass windows on aus typa vahicies, with swing set and house
thrown
MA3: animals
MA3: yellow areas plotures
Layer 2
Layer
Layer 2 Neuron 489
Top Activating Images:
Layer Neuron 1801
Top Activating Imagee:
DnD: monochrome objects
DnD: nature scenes with rocks
MILAN: Stripes
MILAN: Nature
ballons the flowers pasture, the lense of a camera side of a bus MA2: striped or dotted objects stripes
gravel near river, people dining trail,, rocks surface, fountains
MA2: items with curved features
MA3:
similar patterns
Figure 18: MIANNOTATIONS from random neurons in ResNet-152 (ImageNet). We showcase a set of randomly selected neurons alongside their corresponding DnD label, MILAN label, and three MILANNOTATIONS labels (MA). We can see that the three MILANNOTATIONS for cach neuron often don'l match up well and don't accuralely describe the neuron.
36

---

Published in Transactions on Machine Learning Research (01/2025)
A.4 Ablation Studies
A.4.1 Ablation: Attention Cropping
Attention cropping is a critical component of DnD due to generalive image-lo-text captioning. Unlike models that utilize fixed concept sets such CLIP-dissect (Oikarinen Weng 2023), image captioning models are prone to spuriously correlated concepts which are largely irrelevant to neuron's 's activation. To determine the effects of attention cropping subsequent processes in the DnD pipeline, evaluate DnD on D probe without augmented image crops from D Peropped-
We show qualitative examples of this effect in Figure 19
Layer Neuron 475
Layer Neuron 475
Uncropped
Cropped
Potential Candidate Concepts:
Potential Candidate Concepts:
Colorful patterns and objects.
Red and colorful patterns.
Colorful objects and scenes.
Red and multicolored patterns.
Colorful patterns and objects.
Red-themed patterns and objects.
Colorful patterns and objects.
Red-themed pattems and objects.
Geometric shapes colorful objects.
Red-themed patterns and objects.
Layer
Layer
Uncropped
Cropped
Neuron
Neuron 451
Potential Candidate Concepts:
Potential Candidate Concepts:
Patterned designs.
striped patterns
Patterned textiles and people.
striped patterns
Patterned designs.
striped patterns
Patterned clothing and decor.
striped patterns
Different patterns
Striped patterns and clathing.
Figure 19: Examples where Attention Cropping Improves DnD. The left panel shows the result without attention cropping (Uncropped) while the right panel show the result with attention cropping (Cropped). It can be seen that Attention cropping climinates spurious features in highly activating images and improves accuracy of potential candidate concepts.
37

---

Published in Transactions on Machine Learning Research (01/2025)
A.4.2 Ablation: Image Captioning with Fixed Concept Set
In Section 4.4.1 we explored using fixed concept sets with CLIP (Radford et al. 2021 to generate descriptions
for highly activating images rather than BLIP. We present. qualitative examples fixed concept caplioning below in Figure 20 Additionally, we also note particular failure cases caused by a. lack of expressiveness encapsulated in static concepts, demonstrated in Figure 21
Layer Neuron Activating Images:
Layer Neuron Activating Images:
DnD (BLIP): underwater marine life
DnD (BLIP): cleaning and tools supplies
DnD (CLIP): underwater exploration
DnD (CLIP): various products
Neuron 300
Top Activating Images:
Neuron 106
Top Activating Images:
DnD (BLIP): perforated metal sheets and grills DnD (CLIP): dotted cylinders
DnD (BLIP): chevron and pattern designs
DnD (CLIP): optical illusions with patterns
Layer Neuron 244
Activating Images:
Layer Neuron 37
Top Activating Images:
DnD (BLIP): checkered items and backgrounds
DnD (BLIP): black and white designs
(CLIP): geometric designs
DnD (CLIP): repetitive patterns
Figure 20: Examples of DnD with CLIP Image Captioning Compared to DnD (with BLIP). Despite producing similar results on the FC layer, we see that DnD (with BLIP) outperforms DnD with CLIP image captioning, especially on intermediate layer neurons. Single word captions fail to fully encapsulate concepts expressed these layers, resulting poor DnD performance.
Layer
Layer FC Neuron 173
DnD w/ CLIP Output:
DnD w/ CLIP Outout:
Repelition of "hummer'
provide enough
language model, cannot any additional information or change the eet of descriptions, piesse provide more context or information for to generate coherent and concise concept label
Information to determine coherent and concise concept label. pleese provide more context or descriptions
DnD w/ BLIP Output:
Ground Truth:
DnD w/ BLIP Output:
Ground Truth:
hummingbird sightings
hummingbird
dogs and their activities
Ibizan hound, Ibizan Podenco
(a) CLIP Image Captioning Failure Case
(b) CLIP Image Captioning Failure Casc 2
Figure 21: Failure Cases of CLIP Image Captioning. Due to the lack of expressiveness of static concept sets, GPT sunmarization fails to identify conceptual similarities within CLIP image captions. With dynamic concept sets generated by BLIP, the issue is largely avoided.
38

---

Published in Transactions on Machine Learning Research (01/2025)
A.4.3 Ablation: Image-to- Text Model
In light of advancements in image-to-text models, we compare BLIP Li et al, 2022) to a more recently released model, BLIP-2 (Li et al. 2023). Unlike BLIP, BLIP-2 leverages frozen image encoders and LLMs while introducing Querying Transformer to bridge the modality gap between two models. We experiment with using BLIP-2 the imago-to-text model and quantitatively compare with BLIP by computing the mcan cosine similarity between the best concept chosen from Concepl Selection. As discussed in the Appendix A.3 LLUP-VIT-B16 cosine similarity is a stronger indicator of conceptual connections than all-mpnet-basse similarity for generative labels. Accordingly, CLIP-VIT-B/IK cosine similarity is used as the comparison metric. We evaluate on 50 randomly chosen neurons from each intermediate layer of ResNet-50 and results of the experiment are detailed in Table TO. From Table TO BLIP and BLIP-2 produce highly related concepts across all four layers of ResNet-50 and a 87.0% similarity across the entire network.
Table 10: Mean Cosine Similarity Between BLIP and BLIP-2 Labels. For each layer in RN50, we compute the mean CLIP cosine similarity and BERTScore between BLIP and BLIP-2 labels for 50 randomly chosen neurons. Similar conceptual ideas between both models reflected in the high similarity scores.
Metric
Layer 1
Layer 2
Layer
Layer 4 All Layers
CLIP cos
0.864
0.848
0.875
0.891
0.870
BeRTS
0.883
0.880
0.894
0.891
0.887
We also show examples of similar description produced Figure 22
Somewhat counter-intutivell, our qualitative analysis of neurons in preliminary layers of ResNet-50 reveals examples where BLIP-2 fails to detect low level concepts that BLIP able to capture. Such failure cases limit DnD accuracy by generating inaccurate image captions and adversely affecting the ability of GPT summarization identify conceptual similarities between captions. We show two such examples in below Figure 23 In general this experiment shows that our pipeline is not very sensitive to the choice of image-captioning model.
Layer 2 Neuron 25
Layer 4 Neuron 600
Best ELIP Concept: checkerboard pattems
Best BLIP Concept: school buses
Best RL RLIP2 Concent: Checkerboard patterms.
Bast BLIP2 Concagt: school bus transportation
BLI2 Potential Candidate Concepta
BLP2 Potential Candidate Concepts
Checkerboard patterns.
Checkerboard patterns and beach scenes.
School transportation.
bus transportation.
Checkerboard patterns..
school bus transportation
School bus transportation.
Checkered and Polka Dot Patters..
Checkerboard patterna.
School bus transportation.
(a) BLIP-2 Similar Example
(b) BLIP-2 Similar Example
Figure 22: Examples of Similar BLIP and BLIP-2 Labels. BLIP and BLIP-2 detect similar concepts across all layers of ResNet-50. Two such examples detailed above.
39

---

Published in Transactions on Machine Learning Research (01/2025)
Layer Neuran
Layer Neuron 202
Best BLIP Concept, gold and yellow odjectstbackroonds
Best BLIP Concept, grid pattems and related items
Best ILIP2 Concept: textures and objects
Best ILIP2 Concent: pattems and textures
BLIP2 Potential Candidate Concepts
BLIP2 Potential Candidate Concepts
Textures and Objects
Textures and Objects.
Pattems and textures.
Patterns and textures. Patterned surfaces.
Textures and Objects.
Textures Objects
Pattems and textures.
Textures and Objects
Close-up.
Pattems and textures.
(a) BLIP-2 Failure Case Example
(b) BLIP-2 Failure
Case Example 2
Figure 23: Examples of BLIP-2 failure cases. BLIP-2 overlooks crucial low level concepts in early layers of ResNet-50. Potential Candidate Concepts generated are vague and spuriously correlated, yielding poor results the final DnD label.
40

---

Published in Transactions on Machine Learning Research (01/2025)
A.4.4 Ablation:
Multimodal GPT
To justify using two large models instead of one during Step in our pipeline, we conduct an additional experiment by exchanging BLIP + GPT Step 2 of DnD with the image caption capabilities of GPT-4o. Results in Figure 24 show that due the large overlap of the GPT models, performance is similar in many examples. However, the original DnD pipeline seems to be better than DnD w/ GPT-4o when BLIP's image captioning capabilities out purperform that of GPT-4o. Though including two foundation models in Step 2 may reduce the flexibility of our pipeline, it can increase performance by specifying fine-tuned models for each designated task.
Layer Activating Images:
Layer 4
1 Neuron
Layer Neuron 436
Top Activating Images:
BLIP): objects (GPT4o): blue patterns and
DnD (wi BLIP): metal objects
DnD (GPT4o): public access
Layer 4 Neuron 726
Top Activating Images:
Layer
Layer 2 Neuron 170
Top Activating Images:
DnD (wl BLIP): food and drink DnD (GPT4o): fast food and fishing
Layer Neuron 1693
Top Activating Images:
DnD (wl BLIP): vibrant nature scenes DnD (GPT4o): natural landscapes and activities
DnD (wi BLIP): celtic-inspired designs
Layer
DnD (GPT4o): omnat designs and parteems
Layer Neuron 298
Top Activating Images:
Layer Neuron 1734
Top Activating Images:
DnD (wl BLIP): animal and city scenes DnD (GPT4o): urban landscapes and nature
DnD (wl BLIP): gultar-related items DnD (GPT4o): musical instruments and accessories
Figure 24: Neuron examples of GPT-4o replacing BLIP and CPT-3.5-Turbo in the DnD pipeline.
41

---

Published in Transactions on Machine Learning Research (01/2025)
A.4.5 Ablation: Effects of Large-Languager Model Choice
test the robustness our pipeline at the concept summarization stage, we swap out GPT-3.5 Turbo for GPT-4 Turbo and LLaMA2 (Touvron et. al. 2023). For reference, our paper utilizes GPT-3.5 Turbo for all of its primary experiments rather than GPT-4 Turbo because GPT-4 Turbo has 20x the cost of GPT-3.5 Turbo. At the time of developing pipeline, GPT-3.5 Turbo was the most advanced GPT model provided by OpenAI. LLaMA2, although cost free, is restricted to the public and requires specialized access for use in research.
Our experiment follows the precodence of the setups of our previous experiments in our paper and of the setups utilized in prior works. We evaluate 26 neurons on the FC Layer of ResNet-50 and compare to the ground truth labels provided by the classes. In this experiment, we can see that utilizing other models actually significantly increases DnD's performance all metrics. As shown in Table [1 not only is DnD able to keep up its performance with other models but also improve with new and more advanced ones.
Table 11: Comparison of LLMs on label quality. Our experiment shows that GPT-4 Turbo and LLaMA2 significantly increase the cosine similarity score between DnD labels and ground truth classes FC layer neurons, suggesting the ability for our framework to incorporate new advancements in LLMs.
Metric
DnD
DnD w/ GPT-4 Turbo
DnD w/ LLaMA2
CLIP cos
0.6377
0.7607
0.7407
mpnet cos
0.1403
0.4506
0.4726
BERT Score
0.8172
0.8340
0.8105
42

---

Published in Transactions on Machine Learning Research (01/2025)
A.4.6 Ablation: Effects of GPT Concept Summarization
DnD utilizes OpenAl's GPT-3.5 Turbo to summarize similarities between the image caption generated for each highly activaling images of neuron n. This process crucial component to improve accuracy and understandability of our explanations, as shown in the ablation studies below.
As mentioned in Appendix A.1.3. GPT summarization is composed of two functionalities: 1. simplifying image captions and 2. detecting similarities between captions. Ablating away GPT, we substitute the summarized candidate concept set T with the image captions directly generated by BLIP for the K activating images of ncuron n. We note two primary drawbacks:
Poor Concept Quality. An important aspect of GPT summarization the ability to abbreviate muanced captions into semantically coherent concepts (function 1.). Tn particular, failure case for both BLIP and BLIP-2, shown in Figure 25,, is the repetition of nonsensical elements within image captions. Processing these caplions are computationally inefficient and can substantially exlend runtime. DnD resolves this problem by preprocessing image captions prior to similarity detection.
Concept specific to single image. Without, the summarization step, concepts only describe single image and fail capture the shared concept across images. We see for example that the neuron in Figure 25 is described as 'pile of oranges', which only applies to the fourth image and misses the overall theme of orange color.
Layer 155
Layer Neuron 418
Best BLIP Concept: black and white che che...
Best BLIP Concept: pile of oranges
Potential Candidate Concepts
Candidale Concepls
perfored perfored sheet
bi bi
bunch of colorful oolored wires
pile of oranges
perfored perfored...
perfored sheet perfored sheet perfored sheet...
close up of a light on a table
person wearing
spider web in the middle of spider web
holding a cell
a baby bi bi bi...
(a) GPT Ablation: Poor Quality Concept
(b) GPT Ablation: Concept Specific to Single Image
Figure 25: Failure Cases in GPT-Ablation Study. Figure (a) illustrates potential failure cases of DnD
without GPT summarization. BLIP produces illogical image captions that hinder overall interpretability. Figure (b) shows additional failure case where the omission of concept summarization causes individual image captions to generalize to the entire neuron.
43

---

Published in Transactions on Machine Learning Research (01/2025)
A.5 Use Case: Fixed Classifier
The Tile2Vec model evaluated contains two parts: 1. A ResNet model to compute embeddings of land tiles, 2. A random forest classifier to predict the land coverage of the output embedding. We use the originally proposed Telc2Vec pipeline by first pruning ncurons in the ResNet model and then training the classifier on embeddings of the pruned model. Because we retrain the classifier on the pruned pipeline, the difference in accuracy using DnD pruning and that of random pruning fairly minor. In this section, we instead evaluate the Vice2Vec model using fixed classifier. We train the classifier using embeddings from the full unpruned pipeline and evaluate by fixing the classifier while pruning the ResNet model only. We observed a moderate (13.63%) increase in speed after performing pruning on the first layer of Tile2Vec ResNet-18 and further gains are possible if running inference more optimized for sparse models. Table 12 shows results using similar percentage of pruned neurons as Table B
Table 12: Pruning uintterpretable neurons in Tece2Vec ResNet18. Due to GPT's generative captioning, the similarity between neuron labels can vary slightly between runs, causing slightly different number of neurons prunod.
Layer
% of Neurons Pruned
Random Pruning Acc. (%)
Avg. Acc. (%)
No pruning
0.00
71.63
All pruned
100.00
35.96
Layer
23.44
45.0
51.0
Layer
50.0
47.0
49.5
Layer
26.95
66.6
70.5
Layer
50.78
49.5
56.0
Layer
56.64
56.0
60.5
44

---

Published in Transactions on Machine Learning Research (01/2025)
A.6 Additional Use Case
To showcase potential use case for neuron descriptions (and provide another way to quantilatively compare explanation methods), we experimented with using neuron descriptions to find a good classifier for class missing from the training set. Our sctup was as follows: we explained all neurons Layer of ResNet-50 (ImageNet) using different methods. We then wanted to find neurons this layer that could serve as the best classifiers for an unseen class, specifically the classes in CIFAR-TO, CTFAR- R-100, and Places365 datasets. Although there some overlap, these classes are typically much more broad than ImageNet classes. To find classifier, locate the neuron whose description is closest to the CIFAR/ Plaaee/6 class
neuron to name in text. embedding space. We then measure how well that neuron (its average activation) performs single class classifier on the respective validation dataset, measured by area under ROC curve. For cases where multiple neurons share the closest description, we average the performance of all neurons with that description.
Results are shown in Table 13 We can see DnD performs quite well, reaching AUROC values around 0.74, while MILAN performs much worse. We believe this may be because MILAN descriptions are very generic (likely caused by noisy dataset discussed in Section 4.3.1p making it hard to find classifier for specific class. We believe this is good mcasure of explanation quality, as different methods are dissecting the same network, and even if no neurons exist that can directly detect a class, better method should find a closer approximation.
MILAN
DnD (Ours)
CIFAR10
0.5906
0.7036
CIFAR100
0.6514
0.7396
Places365
0.650
0.709
Table 13: The average classification AUC on out of distribution dataset when using neurons with similar description as a classifier. We can see that our DnD clearly outperforms MILAN, the only other generative description method.
45

---

Published in Transactions on Machine Learning Research (01/2025)
A.7 Multiple Labels
As discussed in the Appendix A.10 many neurons in intermediate layers can be considered 'polysemantic" meaning that they encode multiple unrelated concepts. Our primary pipeline only produces one label, and though this label can encapsulate more than one concept, providing multiple labels can better account for issues polysemanticity. We accomplish this by taking the top candidate concepts selected by Concept Selection as our final labels. If the final labels have CLIP cosine similarity exceeding 0.81, we only take the top labels and eliminate the others from the set of final labels. This allows use bring out accurate labels that account for distinct concepts prevalent in the neuron. We can see this in Figure 26 For Neuron 508 from Layer 2, DnD captures not only the polka dot and metal texture concepts of the neuron, but adds that the textures are primarily black and white. For Neuron 511 from Layer 3, DnD labels the neuron as detecting not only interior elements, but also specifying that these elements are mostly black and white.
Layer 1
Layer 3
Layer Neuron
Top Activating Images:
Layer
Activating Images:
Label black and white stripes Label ped alathing
Label 1: interior design elemn
Label 2: black and white decor
Layer Neuron 199
Top Activating Images:
Layer Neuron 550
Top Activating Images:
while
objects
Label 2: checke board and texture
Label 2: reptiles and palterns
Layer 2
Layer
Layer 2 Neuron 508
Top Activating Images:
Layer 4 Neuran 883
Top Activating Images:
Label 1: polka dot and metal textures Label black and white patterns
Label purple and tropical aroduce Label
Layer 2 Neuron
Top Activating Images:
Layar 4 Neuron 963
Top Activating Images:
Label textured dark materials
textured
Label and while Label 2: blue and white irons on various backgrounds
Figure 26: Example results on ResNet-50 (ImageNet) from the DnD pipeline being modified to produce multiple labels. We showcase a set randomly selected neurons to exemplify DnD's capability provide multiple descriptions. We can see that though some of these neurons can't be described by just a singular label, DnD's multiple labels can describe their various aspects.
46

---

Published in Transactions on Machine Learning Research (01/2025)
A.8 Diversity Analysis
To more deeply understand the effectiveness and outcomes of our method, perform an experiment analyzing the relationship between similar highly activating images and similar labels. We follow the setup in Section Over the entircty of the FC Layer of ResNet-50 (1000 neurons), we took random pairs of neurons from those 1000, making 500 pairs in total. We then calculated the similarity between each pair's top 10 activating images and calculated the CLIP cosine similarity between each pair's labels outputted by DnD. Finally, we calculated the correlation coefficient between image similarity and label similarity with each pair being a datapoint and found that value to be 0.4406. We can see there is positive correlation between similar highly activating images and similar DnD labels. In general, DnD provides similar labels for neurons with similar highly activating images while more varying labels for neurons with differing highly activating images.
4.2
47

---

Published in Transactions on Machine Learning Research (01/2025)
A.9 Failure Cases
In this section analyze few examples in Figure 27 that can cause DnD to produce suboptimal neuron labels. Neuron 106 from Layer 3 exemplifies how DnD can occasionally focus on one specific element from the highly activating image set when the truc concept less obvious. Neuron 193 from Layer 3 shows how when given an unitterpretable or polysemantic neuron, DnD defaults to generic label like "patterned items." These neurons are further discussed in sections A.7 and A.10 Neuron 515 from Layer demonst how DnD may produce a more generic and altered version of the specific. We can see that the true concept represented by the neuron is likely something along the lines of 'animal ears," but DnD is not able to identify the ears and adds in incorrect portion of "resting' to the label. Finally, neuron 1880 from Layer 4 showcasos how it's hard for DnD to detect more spacial and abstract representations of the concept. We can see that the true concept should be likely along the lines of 'line patterns' but since the lines are embedded into other concepts like nature elements, DnD isn't 't able to detect
concept when the true concept more
Layer Neuron 106
Top Activating Images:
Layer Neuron 515
Top Activating Images:
Offtvit
DnD Label: rad-thamed Btams
DnD Label: varius animals resting
Layer 3 Nouron 193
Top Activating Images:
Layer 4 Neuron 1880
Top Activating Images:
DnD Label: patterned items
DniD Label: natural elements
Figure 27: Failure cases of DnD. We showcase specific examples from dissecting ResNet-50 that. exemplify instances where DnD produce ambiguous inaccurate labels.
48

---

Published in Transactions on Machine Learning Research (01/2025)
A.10 Limitations
One limilation of Describe-and-Dissee is the relatively high computational cost, laking on average about 38.8 seconds per neuron with a Tesla V100 GPU. However, this problem can be likely well-addressed as the current pipeline has not been optimized for speed-purposes yet (o.g. leveraging multiple GPU etc). Another potential limitation of our method is that since it first converts the images into text and then generates its labels based on the text, DnD likely cannot detect purely visual elements occasionally found lower layers of networks, such as contrast or spatial elements. Additionally, our method only takes into account the top & most highly activating images when generating labels. Excluding the information found in lower activating images may not allow our method to gain full picture of neuron. However, this problem of only focusing on top activations is shared by all existing methods and we are not aware of good solutions to it. Finally, DnD is limited by how well the image-to-text, natural language processing, and text-to-image models are able to perform. On the other hand, this also means that future innovations in these types of Machine Learning models can increase the performance of our method.
Polysemantic neurons:
Existing works Olah et al. (2020) : Mu & Andreas (2020); Scherlis et al. (2023)
have shown that many neurons in common neural networks polysemantic, i.e. represent several unrelated concepts or no clear concept at all. This is challenge when attempting to provide simple text descriptions to individual neurons, and is limitation of our approach, but can be somewhat addressed via methods such as adjusting DnD to provide multiple descriptions per neuron as we have done in the Appendix AA.7 We also note that due to the generative nature of DnD, even its single labels can often encapsulate multiple concepts by using coordinating conjunctions and lengthier descriptions. However, polysemantic neurons still remain a problem to us and other existing methods such as Bau al. (2017), Hernandez al. (2022), and Oikarinen & Weng 2023) One promising recent direction to alleviate polysemanticity is via sparse autoencoders explored by Bricken et al. (2023).
Dut-of-Distribution Synthetic Images: While generated synthetic images in Step 3 (Best Concept Selection) may be out of distribution, we note that an explanation should explain the behavior of the model on all imputs, not just in-distribution examples. A neuron explained by "car" should activate on all kinds of cars, not just in-distribution ones, otherwise, a better explanation would be more specific.
Challenges in comparing different methods: Providing fair comparison between descriptions generated by different methods is very challenging task. The main method we (and previous work) utilized was displaying highly activating images and asking humans whether the description matches these images. This evaluation itself has some flaws, such as only focusing on the top-k activations and ignoring other neuron behavior. In addition, the question of what are the highly activating images and how to display them is a surprisingly complex one and requires multiple judgment calls, with different papers using different methods with little to no justification for their choices. First, which images are most highly activating. When our neuron is a CNN channel, its activation is 2D sorting requires summarizing to a scalar, typically using either max average pooling. In our experiments we used average, but max gives different sets of results. Second, the choice of probing dataset is important, as it will affect which images are displayed. Finally choosing if/ if/how to highlight the highly activating region to raters will have an effect. We chose not to highlight as we crop the highly activating regions which can provide similar information. If highlighting is used, choices like what activation threshold to use for highlights and how to display non-activating regions will also affect results. Different choices for all these parameters will generate varying results. Consequently, it is hard to provide fully fair comparisons, so results such Section 4.3 should be (aken with a grain of salt.
A.11 Broader Impact
Our work aims to improve our underslanding of machine learning models, which we expect to have positive societal impact as more understanding will decrease the chance of harm from deploying unsafe or unreliable modcls. However, one potential downside may be that if explanations look convincing but are not reliable enough, they may cause a user to place unwarranted trust on unreliable model. Our methods can be used to help reveal and mitigate biases in existing vision models by, for example, identifying neurons detecting sensitive attributes such as skin-color.
49


Reviews:
[{"review_id": "3pw6EESmV4", "reviewer": "Reviewer_63zx", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall, the paper clearly explain the proposed method, experiments and discussions.\n1. Related work section is clear and comparisons with existing method is presented clearly.\n2. Authors break down the proposed method into several subsections and introduce each step clearly.\n3. Experimental results can validate that using the proposed method can give the superior performance compared with existing method.\n4. Authors have included extensive experimental results in supplementary materials.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a training-free framework, named Describe-and-Dissect (DnD), to interpret and explain neurons in deep networks. The proposed DnD leverages large multimodal models to provide natural language descriptions for neurons. As the proposed method is training- free, it can be easily adapted to general models. Besides, the proposed method overcomes the existing limitation by providing more complex and higher-quality descriptions rather than representing concepts with simple descriptions. Authors have conducted extensive experiments and ablation studies and results can validated the effectiveness of the proposed method.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I have concerns which are expected to be resolved by the authors listed in the following:\n\n1. At the beginning of section 3, authors can briefly explain the motivations (1 short sentence) when introducing each step.\n2. In step 2, this paper utilizes both the BLIP and GPT models to generate candidate concepts, which is less flexible because it requires 2 large models (Table 5 shows that the improvements with BLIP is not sigfinicant). I wonder have authors tried to replace GPT model with similar language models in BLIP family or even re-use text tower in BLIP? \n3. I wonder authors have further scaled the method with larger ResNet models as well as variants, e.g., VGGNet, etc?\n4. In Table 7, I am curious the efficiency given by introducing the pruning operations? Since authors show that the certain percent of neurons have been prunes, whether it can bring some speedup? \n5. One open discussion question: is the proposed method limited to CNNs? Have authors consider any extensions in transformers?", "broader_impact_concerns": "Authors have not included any broader impact discussions, I encourage authors include necessary discussions.\n1. When leveraging large language models, are there any biased contained in these models?\n2. Since the proposed method is providing explanations for neurons, authors should discuss use cases that includes detecting features related to humans (e.g. facial recognition).", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "mzumDmGB8b", "reviewer": "Reviewer_zajy", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- Innovative approach to pruning: The pruning experiments in Section 5 are a compelling application of DnD\n- Interesting human evaluation: The AMT human study is well-designed, and the results demonstrate DnD’s interpretability benefits\n\nWeaknesses\n- Lack of clarity on holistic vs. local concepts: The paper mentions the importance of holistic versus local concepts, especially when discussing Kalibhat et al. (2023). However, this distinction is not clearly defined, leaving it unclear about what makes holistic concepts valuable or how they are incorporated.\n- D_probe and D_cropped: Details on how D_probe is constructed and why D_cropped is needed are lacking. This absence makes it difficult to assess the technical correctness of Step 1 and its effect on downstream steps. The paper would benefit from explicitly stating the role of D_probe and its limitations, followed by a justification of D_cropped. I looked into A.1 in Appendix and did not find that info.\n- Potential Out-of-Distribution (OOD) activation in step 3: In Step 3, feeding synthetic (generated) images into the model can return OOD activations because the model could see the generated images drawn from a different distribution than the distribution of its training set. This could make the activation not reliable to evaluate the generated concepts. The reviewer encourages the authors to explore this limitation (i.e. Do we have better alternative than generating synthetic images and feed back to the model? I know that there are a bunch of papers following the same approach but it has the OOD limitation as mentioned. I would love to see the authors discuss this problem and possible solution in Discussion/Limitations.)\n- Last-layer neurons in evaluation: In Section 4.2.1, the authors assume that last-layer neurons correspond to class names, yet it is unclear why this would be the case. In [1,2], people show evidence that the deep layers of NNs are responsible for high-level concepts (e.g. nose, tail), that together form the object, but not the class name. Without a clear justification, this assumption may be problematic, as high-level concepts do not always equate to specific class labels.\n- Writing and consistency issues: Typos, inconsistencies (e.g., “Figure” vs. “Fig.,” “Section” vs. “Sec.”), and a lack of stylistic coherence impact readability. Tables with only a few rows, such as Tables 3 and 4, could be combined to improve presentation.\n- Lack of experiment details and justification in key sections:\n\n** In Section 4.2.1, the setup for the textual similarity evaluation is missing key details, such as the number of neurons evaluated and input specifics for the similarity function.\n\n** Section 4.4.1 does not clarify how concepts are generated without using an image-to-text model, which would be crucial information for replication.\n\n** In Table 5, the minimal performance gain from including the image-to-text model raises questions about its value. The 2.6% when using mpnet is not enough evidence to demonstrate the importance of the img-to-text model.\n\n** In Section 4.4.2, it is unclear how concept selection is achieved if not using Best Concept Selection. The authors do not specify how they chose a concept among the generated ones.\n\n[1] Visualizing and Understanding Convolutional Networks\n\n[2] Understanding Neural Networks Through Deep Visualization", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents Describe-and-Dissect (DnD), a method for generating natural language descriptions of hidden neurons in vision networks. The DnD pipeline has 3 main steps: \n- Probing Set Augmentation: Building a set of images to record neuron activations on before generating a description for the neuron\n- Candidate Concept Generation: Given a neuron, feed the highly-activating images to BLIP to generate descriptions and use GPT-3.5 Turbo for summarizing neuron descriptions \n- Concept Selection: Given possible neuron descriptions (concepts), synthetic images are generated for each concept and fed back to the model to select the most representative concept. \n\nThe method is evaluated through qualitative and quantitative studies, ablation analyses, and a use-case application in land cover prediction.\nThe reviewer encourages the authors to revise the writing carefully and provide missing key details.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Clarify holistic vs. local concepts: Provide a clear distinction between holistic and local concepts to contextualize the importance of including holistic concepts in neuron interpretation. Explain how DnD captures both types and why holistic concepts matter for understanding neuron function.\n- Justify D_probe and D_cropped: In Step 1, clearly describe how D_probe is constructed and its purpose, along with its limitations, and justify the inclusion of D_cropped. Without these explanations, Step 1 lacks sufficient grounding for technical evaluation.\n- Potential OOD activation in step 3: Discuss the potential for OOD activation when using synthetic images in Step 3. Consider exploring alternative approaches or at least acknowledging this as a limitation in the Discussion section to provide a balanced view of DnD’s applicability.\n- Explain assumptions on last layer neurons in Section 4.2.1: Provide justification for assuming that last-layer neurons correspond to class labels. Clarifying this assumption is necessary for readers to interpret the validity of the textual similarity evaluation results.\n- Qualitative analysis: I like the AMT human study setup, yet, I hope authors can give more statistics about the numbers (e.g. std for mean rating) in Tables. Also, showing qualitative examples where each method is the best should be insightful to know where DnD works/does not work.\n\n- Threshold justification in section 5.1: In Section 5.1, provide rationale for the threshold value (0.8) used to determine concept similarity/dissimilarity in land cover prediction.\n\n- Revise writing for consistency and clarity: Correct typos and ensure stylistic consistency across the paper, such as standardizing “Figure” and “Table” usage. Additionally, combining tables with limited rows (e.g., Tables 3 and 4) would streamline presentation. Examples of typos in writing: (a) compare descriptions similarity to the class name that neuron is detecting; (b) For a pair of candidate concepts sets. Also, the Limitations are placed in Appendix A.8 should be moved to the main text (the authors could briefly discus the main points to save spaces).\n\n- Provide additional experiment details:\n\n(a) In Section 4.2.1, include details on the experimental setup, such as the number of neurons evaluated and input specifics for the textual similarity function.\n\n(b) Clarify the concept selection approach in Section 4.4.2 for cases where Best Concept Selection is not used.\n\n(c) Add statistical details for Tables, such as standard deviation for mean ratings in the AMT study.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "asOZkXaDtB", "reviewer": "Reviewer_Vcjh", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The presented method is simple and training-free, using widely available off-the-shelf models.\n1. The method performs better than previous methods, as shown in the quantitative preference evaluation by raters.\n1. Overall, the paper is well written and easy to follow.\n\nWeaknesses:\n\n1. The paper only discusses the application of the method to ResNets.\n1. The method is missing some ablations, such as the used image-to-text model or the text-to-image model, as well as the used thresholds. Having good ablations of these parameters would give confidence that the method is tuned sufficiently to be used by the community.\n1. I found the Use Case Section hard to follow and I'm not convinced by the results presented in that section. This section makes the submission harder to read and adds little insight in its current form.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents a novel XAI technique called Describe-and-Dissect (DnD). The technique describes the \"role\" of hidden neurons in ResNets by generating \"concepts\" that best describe images that lead to high activations of these hidden neurons.\n\nThe network is first probed with random crops of images from a probing set. The crops leading to largest activations of neurons are then captioned using BLIP (\"candidate concept generation\"). These candidate concepts are then selected by generating images using Stable Diffusion and scoring the concepts by the resulting activation of these hidden neurons (\"best concept selection\").\n\nThe authors compare the effectiveness of the proposed method with other methods (Network Dissection, CLIP-Dissect, MILAN), and then also examine the effect of neuron pruning based on the outputs of the method with two land-cover prediction networks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Critical requests:\n\n1. The PDF is missing the Appendix, but the Appendix is referenced in many places.\n1. Section 3.4 \"Scoring Function\" should be revised for improved clarity (the other parts of Section 3 are easy to follow). Some examples of what I found unclear in this section: Are `t` and `β` free parameters? How are they set? Is the \"average rank\" the defined as $\\text{avg}(\\mathcal{H}_j)$? What does it mean that \"Rank($R_j$) sorts $R_j$\"?\n\nImprovements to Section 5:\n\n1. Making list of NAIP and EuroSAT labels more prominent would help readability of this section\n1. Make it clear which model is evaluated in which subsections.\n1. Section 5.2. claims that the results after pruning are \"better than the baseline\" – but the difference is very small, and there is no information on statistical variance.\n1. In Table 7, the observed changes should also be compared to pruning a random subset of neurons. Otherwise the observation that \"human interpretable neurons account for more critical roles\" does not hold.\n1. The first paragraph of Section 5.1 is not clear: how is the \"spatial mean\" computed from two text embeddings? how are the neurons grouped together by using a single threshold? what was the prompt used with GPT (which version?) for \"one-shot classification into the 6 NAIP superclasses\"?\n1. Section 5.1. claims that \"clusters containing more neurons are frequently associated with more interpretable neurons\" – looking at the concepts in Figure 4, I don't find this finding obvious\n1. in Section 5.3. it is not clear how \"Term Frequency Analysis\" is used, or how the \"relationship\" is studied\n1. Not sure what Section 5.3 shows. It seem that using probing images from ImageNet and Broden has lead to concept generation that is not useful for the EuroSAT dataset, and that this leads to a rather random selection of two neuron groups, one of which has a large effect, and the other does not.\n\nRequests to strengthen the submission:\n\n1. Show some failure cases and examine why they happen and how they could be improved.\n1. Add a code link so the community can use the method.\n1. Add FALCON results to Figure 1, and compare with FALCON in Section 4.2\n1. Missing table of intermediate neurons (Section 4.1.1) – Table 5 only has results of FC layer\n1. Add results for a non-ResNet model, or explain why this is impossible.\n1. Ablate BLIP and Stable Diffusion models – how sensitive is the method to these models?\n1. what `K` was used? (Section 3.2)\n1. Clarify (Section 4.3) \"we used the images calculated by our method\" – does this mean images from $\\mathcal{D}_\\text{probe}$? are these the same 10 images that were presented to workers to rate the description? (asking because the same paragraph mentions \"10 images\" twice, but only mentions the \"images calculated by our method\" on the second occurrence)\n1. Add the prompt that was used for summarization (Step 2B)\n\n\nTypos\n\n1. The abbreviation \"WPMI\" is used before it's spelled out as \"weighted pointwise mutual information\".\n1. Section 3.1 \"and an perform\".", "broader_impact_concerns": "The authors claim that \"One important role of interpretability tools is the ability to create real-world impacts\" in Section 5.\n\nThis aspect might be elaborated in more detail in a \"Broader Impact Statement\". I see this though more of an opportunity for the authors if they want to highlight this aspect of the work, and not as a requirement.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a new method, Describe and Dissect (DnD) to assign textually describable "concepts" to individual neurons in DNNs. The idea is to identify the most highly activated images (along with augmented, cropped images obtained through self attention), then use Image-to-Text model such as BLIP, together with GPT, to summaries the similarities between the generated images in one caption. The result is then fed through a stable diffusion to generate new synthetic images corresponding to the each candidate concept. Crucially, the best performing candidate concept is then extracted by verifying that it activates the target neuron strongly. 

I find the success of this pipeline convincing, especially due to the final candidate concept selection step, which guarantees that the LLM-generated concepts are not off topic/hallucinatory. It is also clear that substantial effort has gone into designing the appropriate scoring functions for that last step. Lastly, the experiments are very extensive, all reviewers recommend acceptance, and the paper is reasonably well written. 


Minor points: 

As mentioned by reviewer zajy, there is only one subsection in section 4.2, resulting in awkward formatting, which can be improved during the camera-ready stage. 

I would also recommend adding an additional pointer to Appendix A.1.5 in Section 3.2 (fourth step).
Decision: Accept (poster)