CLIP-Dissect: Automatic Description of Neuron Representations in Deep Vision Networks
Authors: Tuomas Oikarinen
Tsui-Wei Weng
Venue: ICLR 2023 notable top 25% (2023)
Keywords: Interpretability
Explainability
Network Dissection

Abstract: In this paper, we propose CLIP-Dissect, a new technique to automatically describe the function of individual hidden neurons inside vision networks. CLIP-Dissect leverages recent advances in multimodal vision/language models to label internal neurons with open-ended concepts without the need for any labeled data or human examples. We show that CLIP-Dissect provides more accurate descriptions than existing methods for last layer neurons where the ground-truth is available as well as qualitatively good descriptions for hidden layer neurons. In addition, our method is very flexible: it is model agnostic, can easily handle new concepts and can be extended to take advantage of better multimodal models in the future. Finally CLIP-Dissect is computationally efficient and can label all neurons from five layers of ResNet-50 in just 4 minutes, which is more than 10$\times$ faster than existing methods. Our code is available at https://github.com/Trustworthy-ML-Lab/CLIP-dissect.

Published as a conference paper at ICLR 2023
CLIP-DISSECT: AUTOMATIC DESCRIPTION OF NEU- RON REPRESENTATIONS IN DEEP VISION NETWORKS
Tuomas Oikarinen UCSD CSE toikarinen@ucsd.ed edu
Tsui-Wei Weng UCSD HDSI
Lweng&ucsd. edu
ABSTRACT
In this paper, we propose CLIP-Dissect, a new technique to automatically describe the function of individual hidden neurons inside vision networks. CLiP-Dissect leverages recent advances in multimodal vision/language models to label internal neurons with open-ended concepts without the need for any labeled data or hu- man examples. We show that CLIP-Dissect provides more accurate descriptions than existing methods for last layer neurons where the ground-truth is available as well as qualitatively good descriptions for hidden layer neurons. In addition, our method very flexible: can be extended to take advantage of better multimodal models in the future. Fi- nally CLIP-Dissect is computationally efficient and can label all neurons from five layers of ResNet-50 in just 4 minutes, which is more than 10x faster than existing methods. Our code available at htttts://gittuub..oo http::/ggihuu..oooooo wtwpgghuubbooootuuuttttty.y wwtpwoooooh---L//CCLLLLI ML-Lab/CLIP- dissect
model agnostic, can easily handle new concepts and
INTRODUCTION
Deep neural networks (DNNs) have demonstrated unprecedented performance in various machine learning tasks spanning computer vision, natural language processing and application domains such as healthcare and autonomous driving. However, due to their complex structure, it has been chal- lenging to understand why and how DNNs achieve such great success across numerous tasks. Un- derstanding how the trained DNNs operate is essential to trust their deployment in safety-critical tasks and can help reveal important failure cases or biases of a given model.
One way towards understanding DNNs is to inspect the functionality of individual neurons, which the focus of our work. This includes methods based on manual inspection (Erhan et al. 2009) Zeiler & Fergus, 2014. Zhou et al. 2015 Olah et al. 2017 2020 Goh et al. 2021), which provide high quality explanations and understanding of the network but require large amounts
manual effort. To address this issue, researchers have developed automated methods to describe the func- tionality of individual neurons, such as Network Dissection (Bau et al.) 2017) and Compositional Explanations (Mu & Andreas 2020). In (Bau al. 2017), the authors first created new dataset named Broden with pixel labels associated with a pre-deterd set of concepts, and then use Bro- den find neurons whose activation pattern matches with that of a pre-defined concept. In (Mu) & Andreas 2020), the authors further extend Network Dissection to detect more complex concepts that are logical compositions of the concepts in Broden. Although these methods based on Network Dissection can provide accurate labels in some cases, they have few major limitations: (1) They require: a densely annotated dataset, which is expensive and requires significant amount of human la- bor to collect; (2) They can only detect concepts from the fixed concept set which may not cover the important concepts for some networks, and it is difficult to expand this concept set as each concept requires corresponding pixel-wise labeled data.
To address the above limitations, we propose CLIP-Dissect, a novel method to automatically dissect DNNs with unrestricted concepts without the need of any concept labeled data. Our method is training-fret free and leverages pre-trained multi-modal models such as CLIP (Radford et al., 2021 to efficiently identify the functionality of individual neuron units. We show that CLIP-Dissect (i) provides high quality descriptions for internal neurons, (ii) is more accurate at labeling final layer neurons where know the ground truth, and (iii) is 10x-200x more computationally efficient.

---

Published as a conference paper at ICLR 2023
ResNet-50 Layer 1
ResNet-50 Layer 4
Neuron stries
watfled
MILANIb:: that nursery
objects
Neuron
CUP-Dissect MILANibi: Red colored objerts
NetDissect: banded
Neuron CUP-Olesect: graduating MILANibj: Red and blue objects
NetDissect: wriklled
MILANíph: Red and white objects
MILAN(a) Faces of human beings
Neuron 242:
CLP Dissect: aquarium
NetDiisectt: blue-4
Nearon 683: ELP--Oiesett: Barrier
NetDisset:: bus
MILANibi: F'luorescent blue objects
MILANiph: Blue and purple objects
MILANibj: Animals and circular abjects
MILANoo) Groular objects
Neuron gnd
perforated MILANiph
Neuron MUANibb: Diagonal
NetDissect: objects
Figure Labels generated by our method CLiP-Dissect, NetDissect (Bau et al., 2017) and MI- LAN (Hernandez et al.) 2022) for random neurons of ResNet-50 trained on ImageNet. Displayed together with most highly activating images for that neuron. We have subjectively colored the descriptions green if they match these 5 images, yellow if they match but are too generic and red they do not match. this paper we follow the torchvision (Marcel & Rodriguez, 2010) naming scheme of ResNet: Layer 4 is the second to last layer and Layer I is the end of first residual block. MILAN(b) is trained on both ImageNet and Places365 networks, while MILAN(p) is only trained on Places365.
than existing methods. Finally, we show how one can use CLIP-Dissed to better understand neural networks and discover that neurons connected by a high weight usually represent similar concepts.
2 BACKGROUND AND RELATED WORK
Network dissection. Network dissection (Bau et al. 2017) is the first work on understanding DNNs by automatically inspecting the functionality (described concepts) of each individual neu- ron They identify concepts of intermediate neurons by matching the pattern of neuron activations to the patterns of pre-defined concept label masks. In order to define the ground-truth concept la- bel mask, the authors build an auxiliary densely-labeles dataset named Broden, which is denoted as Dardenn The dataset contains a variety of pre-determined concepts e and images 2; with their associated pixel-level labels. Each pixel of images x; is labeled with set of relevant concept which provides a ground-truth binary mask (z,)) for a specific concept c. Based on the ground- truth concept mask (xx),, the authors propose to compute the intersection over union score (IoU) between (x:) and the binarized mask M (xx) from the activations of the concerned k-tth neuron En,ePaaien (a(a))nLeee
unit over all the images T,4 in Drooden: loUk,c
EPPrrnn (c(.))U
loU,.c > 77. then the neuron 1 is identified to be detecting concept c. In (Bau et al., 2017). the authors set the threshold 77 to be 0.04. Note that the binary masks M2 (2x)) computed via thresholding the spatially scaled activation Se(x;) > E, where & is the top 0.5% largest activations for the neuron over all images I; DRedden and S& (x:) has the same resolution as the pre-defined concept masks by interpolating the original neuron activations (x.)).
(Bau al. 2020 propose another version of Network Dissection, which replaces the human lated labels with the outputs of a segmentation model. This gets rid of the need for dense annotations
We follow previous work and "neuron" describe channel in CNNs.
2

---

Published as a conference paper at ICLR 2023
in Dprobes but still requires dense labels for training the segmentation model and the concept set is restricted to the concepts the segmentation model was trained on. For simplicity, we focus on the original Network Dissection algorithm (with human labels) in this work unless otherwise mentioned.
MILAN. MILAN (Hernandez et al. 2022) is a contemporary automated neuron labeling method addressing the issue of being restricted to detect predefined concepts. They can generate unrestricted descriptions of neuron function by training generative images-to-text model. The approach of (erraanezz et al. 2022) is technically very different from ours as they frame the problem as learning to caption the set of most highly activating images for a given neuron. Their method works by collecting a dataset of human annotations for the set of highly activating images of a neuron, and then training generative model to predict these human captions. Thus, MILAN requires collecting this curated labeled data set, which limits its capabilities when applied to machine learning tasks outside this dataset. In contrast our method does not require any labeled data for neuron concepts and is training-free.
CLIP. CLIP stands for Contrastive Language-Image Pre-training (Radiord et al., 2021), an effi- cient method of learning deep visual representations from natural language supervision. CLIP is designed to address the limitation of static softmax classifiers with a new mechanism to handle dynamic output classes. The core idea of CLIP is to enable learning from practically unlimited amounts of raw text, image pairs by training an image feature extractor (encoder) E, and a text encoder Er simultaneously. Given a batch of N image Iy and text t, training example pairs denoted as {Zz,, ti) Nj with [N] defined as the set {1. 2..,.....NN N.. CLIP aims to increase the similarity of the (æ:, t.) pair in the embedding space as follows. Let 1; = (x;)),, = (t;), CLIP maximizes the cosine similarity of the Ti) in the batch of N pairs while minimizing the cosine similarity of using a multi-class N-pair loss (Shnn 2016 Radford et al. 2021). Once the image encoder E₁ and the text encoder Er are trained, CLIP can perform zero-shot classification for any set of labels: given a test image T1. we can feed in the natural language names for a set of M labels {4i)SE The predicted label of xi is the label 4 that has the largest cosine similarity among the embedding pairs: (41, Tk).
(I,, . f
METHOD
In this section, we describe CLiP-Dissect, an automatic, flexible and generalizable neuron labeling method for vision networks from popular convolutional neural networks (CNNs) to SOTA vision transformers ( (ViT). An overview of CLIP-Dissect algorithm is illustrated in Figure 2 and described in detail in See 3.1 We then introduce and discuss few theoretically inspired choices for similarity
function in Sec 3.2 Finally in Sec 3.3
we discuss how our method can benefit from more powerful
models in the future.
3.1 CLIP-DISSECT OVERVIEW
Inputs & Outputs. The CLiP-Dissect algorithm has 3 inputs: (i) DNN be dissected/prohee, denoted as ((x) (ii) a set of probing images, denoted as D Ppoube where 'protbel = N, (iii) set of concepts, denoted S. |S| M.
The output of CLIP-Dissect is the neuron labels, which identify the concept associated with each individual neuron. Compared with Network Dissection, our goals are the same we both want to inspect and detect concepts associated with each neuron. The input (i) is also the same, as we both want to dissect the DNN fz); however, the inputs (ii) and (iii) have differences. Specifically, in CLIP-Dissect, our Dprote does not require any concept labels and thus can be any publicly available dataset such as CIFAR-100, ImageNet, combination of datasets or even unlabeled images collected from the internet. On the other hand, Network Dissection (Bau et al. (017)) can only use a Dprote that has been densely labeled with the concepts from the concept set S. As a result, users of Network Dissection are limited to the D Prote DBoden and the fixed concept set S of Broden unless they are willing to create their own densely labeled dataset. In contrast, the concept set S and probing can any text corpus to form the concept set
dataset Drrebe in framework are decoupled and any image dataset D prohe independent of S in CLiP-Dissect, which significantly increases the flexibility and efficiency to detect neuron concepts.
3

---

Published as a conference paper at ICLR 2023
CLIP-Dissect
Input
Concept $ burning, oceanic...
Ey
Probing dataset: Dprobe
E
127
8s
1yT.
Image encodor
Compute concegt- activation matrix
Output
Successfully dissect NN!
Network being probed: (x)
Record activations
car
((A)(s)))
Generate neuron labels
a(Agtra)) 3
argmax sim((... P) we|k]
(CA6
4 = accanic
oceanic
Figure Overview of CLIP-Dissect: a 3-step algorithm to dissect neural network of interest
Algorithm. There are key steps in CLiP-Dissect:
1. Compute the concept- concept-activation matrix P. Using the image encoder E, and text encoder Er of CLIP model, we first compute the text embedding T, of the concepts in the concept set S and the image embedding I; of the probing images n; in the probing dataset calculate the concept-activation matrix P R. whose (i,j)-th ele-
Pprohe- Next,
ment is the inner product 4, Ty. i.c. P., = 4. Tz.
2. Record activations of target neurons. Given neuron unit k, compute the activation A& (x;) of the k-th neuron for every image E "mohe Define a summary function g. which takes the activation map Ag (xr)) as input and returns real number. Here we g be the mean function that computes the mean of the activation map over spatial dimensions, but 9 can be any general scalar function. We record (AAx(;)), for all k.
3. Determine the neuron labels. Given neuron unit k, the concept label for l is de- termined by calculating the most similar concept f.g. with respect to its activation vec- tor 9k (A(A)(2)))).... (A(c)))))) defined as s sim(tm, (k; P). In other words, the label of neuron k is t, where / = arg max, sim(fm, qk; P). Below discuss different ways define sim.
9k E RRN. The similarity function s sim is
3.2 SIMILARITY FUNCTION
There are many ways to design the similarity function sim, and this choice has a large effect on the performance of our method. In particular, simple functions like cosine similarity perform poorly, likely because they place too much weight on the inputs that don't 't activate the neuron highly. We focus on the following 4 similarity functions and compare their results in the Table
Cos. Cosine similarity between the activation vector (qk) of the target neuron & and the concept activation matrix P.m. from CLIP with the corresponding concept t.m
PT
sim(tm, 9ki
(I)
P
Rank reorder. This function calculates the similarity between 9k and P.m. by creating vector de which has the values of 9: in the order of P.m-- Le. is generated by reordering the elements of q according the ranks of the elements P.m. The full similarity function is defined below, and is maximized when the and P.m have the same order:
mmmmm gk; P) A -11 -x-lp
4

---

Published as a conference paper at ICLR 2023
WPMI (Weighted Pointwise Mutual Information). A mathematically grounded idea to derive a sim based on mutual information as used in (Wang et al. 2020). where the label of neuron is defined as the concept that maximizes the mutual information between the set of most highly activated images on neuron k, denoted as Bg, and the concept 4m- Specifically:
s (4m; 0k: P) A wpmi(I qs) = log p(mml Bx)) Algg((mm)
where p(fm. = B. (( (z)) and À is a hyperparameter.
SoftWPMI. Finally, we propose a generalization of WPMI where we use the probability
p(z B&) to denote the chance an image x belongs to the example set Bg- Standard
WPMI corresponds the case where p(r E B5) is either
for all x E Dprobe while SoftWPMI relaxes the binary setting of p(æ; E Bx) to real values between 0 and 1.
This gives us the following function:
sim(tm, 9k: P) _ soft wofittt 9&) log [lp(Em m B&)l - llgg((tm) (4) where we compute log Elp B&)] log([IzED [1 + p(z (x))(((ml2z) - 1)])- As shown in our experiments (Table B). we found WPII give the best results among the four and thus we use in all our experiments unless otherwise mentioned.
Due to page constraint, we leave the derivation and details on how to calculate WPMI and So using only CLIP products matrix P. as well as our hyperparameter choices to Appendix A.I
3.3 COMPABILITY WITH FUTURE MODELS
The current version of our algorithm relies on the CLIP (Radford ct al. 2021) multimodal model. However, this doesn't have to be the case, and developing improved CLIP-Iike like models has received lot of attention recently, with many recent works reporting better results with an architecture similar to CLIP YYu et al., 2022 Yuan et al. 2021] Zhai al. [022] Pham al. (02T). If these models are released publicly, we can directly replace CLIP with a better model without any changes to our algorithm. As result, our method will improve over time as general ML models get more powerful, while existing works (Bau et al. 20177; Hernandez et al. 2022) can't 't really be improved without collecting a new dataset specifically for that purpose. Similar to ours, the segmentation version of Network Dissection (Bau et al. 2020) can also be improved by using better segmentation models, but each improved segmentation model will likely work well for only a few tasks.
EXPERIMENTS
In this section, we provide both qualitative and quantitative results of CLiP-Dissect in Sec and 4.2 respectively. We also provide an ablation study on the choice of similarity function in Sec 4.3 and compare computation efficiency in See 4.4 Finally, we show that CLIP-Dissect can detect concepts that do not appear the probing images in Sec 4.5 We evaluate our method through analyzing two pre-trained networks: 5e (He et al. 2016) trained on ImageNet (Deng et al.) 2009), and ResNet- 18 trained on Places-365 (Zhou et al.) 2017). Our method can also be applied to modern architectures such Vision Transformers discussed in Appendix A.5] Unless otherwise mentioned we use 20,000 most common English words as the concept set S.
Due to the page limit, we leave additional 9 experimental results in the Appendix. Specifically, Ap- pendix sh2oss additional qualitiative results discussed in Section [.1 Appendix [A.3] showcaaes our ability to detect low-level concepts but also discusses some limitations, such as sometimes out- putting higher level concepts than warranted. Appendix [.4 shows how our method can be applied to generate compositional concepts, and Appendix A.S shows that our method can be applied to Vision Transformer architecture and provides qualitative results. In Appendix A.6 we experiment with another potential method measure quality of neuron explanations and show it also favors CLss-Dissect. Appendix A.7 discusses the limitations of only displaying top-5 images for qualita- tive evaluations and showeases wider range for some neurons. In Appendix [A.8] we discuss how our method can be used to decide whether neuron is interpretable or not. Appendix [A. shows the qualitative effect of different similarity functions. Finally, in Appendix AA.10] we evaluated our description quality for 500 randomly chosen neurons, and found descriptions generated by CLIP- Dissect to be a good match for 65.5% of the neurons on average.
"Surree: http://wwwmttatoooo//oooooooooom

---

Published as a conference paper at ICLR 2023
4.1 QUALITATIVE RESULTS
Figure shows examples of descriptions for randomly chosen hidden neurons in different layers generated by CLIP-Dissen and the two baselines: Network Dissection (Bau et al., 2017) and MI- L.NN (Hernandez et al. 2022). We do not compare against Compositional Explanations (Mu & An- dreas 2020j as much more computationally expensive (at least 200 times slower) and comple- mentary to our approach as their composition could also be applied to explanations. We observe that not every neuron corresponds to clear concept and our method can detect low-level concepts on early layers and provide more descriptive labels than existing methods in later layers, such as the 'graduating' and nursery' neurons. These results use the union of ImageNet validation set and Broden as Drrobe- In general observe that MILAN sometimes gives very accurate descriptions but often produces descriptions that are too generic or even semantically incorrect (highlighted as red labels), while Network Dissection good at detecting low level concepts but fails concepts missing from its dataset. We compared against two versions of MILAN: MILAN(() was trained to describe neurons networks trained on "both" ImageNet and Places365, and MILAN(p) was only trained on Places365 neurons to test its generalization ability. Additional qualitative comparisons for interpretable neurons are shown in Figures and in Appendix
4.2 QUANTITATIVE RESULTS
Besides the qualitative comparison, in this section we propose the first quantitative evaluation to compare our methods's 's performance with baselines. The key idea to compare the neuron labels generated for neurons where we have access the ground truth descriptions - i.e. the final layer of a network, as the ground truth concept of the output layer neuron is the name of the corresponding class (class label). This allow to objectively evaluate the quality of the generated neuron labels, which avoids the need for human evaluation and uses real function of the target neurons while human evaluations are usually limited describing a few most highly activating images. We propose below two metrics for measuring the quality of explanations:
a)
Cos similarity: We measure the cosine similarity in a sentence embedding space between the ground truth class name for the neuron (e.g. "sea lion" in Fig 3 and the explanation generated by the method. For embeddings, we use two different encoders: the CLIP VIT-B/16 6 text encoder (denoted as CLIP cos) and the all-mpnet-base-ve sentence encoder (denoted as mpnet cos). See Figure 3 for an example of the similarity scores for descriptions of single neuron.
b) Accuracy:
We compute accuracy for method the percentage of neurons that the method signs the exact correct label i.e. the class name. Note that we only measure accuracy in situations where the method chooses from concept set that includes the exact correct label, such as Net- work Dissection for models trained on Places365 (not for ImageNet models since ImageNet labels are missing from Broden). We also did not measure accuracy of MILAN as MILAN generates explanations without a concept set and thus is unlikely to match the exact wording of the class name.
The cosine similarity of predicted labels compared to ground truth labels on final layer neurons of ResNet-50 trained on ImageNet. The higher similarity the better. We can see that our method performs better when D probe and concept set are larger and/or more similar to training data.
Table 1:
Method
Prrobe
Concept set
CI.IP cos
mpnet cos
Network Dissection (baseline) MILAN(() (baseline)
Broden
Broden
0.6929 0.7080
0.2952 0.2788
ImageNet val
ImageNet val
Broden
0.7393
0.4201
ImageNet val
3k
0.7456
0.4161
ImageNet
10k
0.7661
0,4696
ImageNet val
20k
0.7900
0.5257
CLIP- -Disectt (Ours)
ImageNet val
ImageNet
0.9766
0.9458
CIFAR train
20k
0.7300
0.3664
Broden
20k
0.7407
0.3945
ImageNet val
20k
0.7900
0.5257
ImageNet val Broden
20k
0.7900
0.5233

---

Published as a conference paper at ICLR 2023
Table Performance when labeling final layer neurons of a ResNetl8 8 trained on Places365, Accu- racy measured on 267/365 neurons whose label is a directly included in Broden labels.
Method
D Proobe
Concept set
gt label annotation
Topl Ace
CLIP cos
mpnet cos
Net-Disseet (baseline)
Broden
Broden
Yes
43.82%
0.8887
0.6697
CLIP- P-Disseectt (ours)
Broden
Broden
No
58.05 %
0.9106
0.7024
Final layer neuron (#150) in ResNet-50 (ImageNet), Ground truth: sea lion
Methods
Generated neuron label
CLIP cos
mpnet cos
Accuracy
Network Dissection:
0.7754
0.3473
NIA
MILAN (b):
Animals the watcr
0.8037
0.3291
N/A
CLIP-Dissecipp
scals
0.8735
0.5151
N/A
1.0000
1.0000
(exactly matches ground truth!)
Figure 3: Example of a final layer neuron: we compare the descriptions generated by different meth- ods and our metrics. Accuracy only evaluated for CLiP-Dissect with ImageNet labels as concept set since it is the only method where exact correct answer is possible choice and therefore accuracy makes sense.
In Table we can see that the labels generated by our method are closer to ground truth in sentence embedding spaces than those of Network Dissection or MILAN regardless of our choice of D Prrobe or concept set S. We can also see that using larger concept set (e.g. 3k v.s. 20k) improves performance of our method. Table 2 shows that our method outperforms Network Dissection even though this task is favorable to their method (as the Places365 dataset has large overlaps with Broden). We highlight that CLIP-Dissect can reach higher accuracy even though Network Dissection has access to and relies the ground truth labels in Broden while ours does not.
4.3 CHOICE OF SIMILARITY FUNCTION
Table 3 compares the performance of different similarity functions used in CLIP-Dissect. We use accuracy and cos similarity in embedding space defined in Sec 4.2 to measure the quality of descriptions. We observed that SoftWPMI performs the best and thus it is used in all other ex- periments unless otherwise mentioned. The effect of similarity function shown qualitatively in Appendix A.9 Table 3 also showcases how CLIP-Dissect can give final layer neurons the correct label with a very impressive 95% accuracy.
Table Comparison of the performance between similarity functions. We look at the final layer of ResNet-50 trained on ImageNet (same as Tab IT. We use S = 20k for cosine similarity and ImageNet classes for topl accuracy. We can see SoftPMI performs best overall.
D prrobe
CIFAR100
Broden
ImageNet val
ImageNet val Average
Metric
Similarity function
train
Broden
mpnet
0.2761
0.215
0.2823
0.2584
0.2580
similarity
Rank reorder
0.3250
0.3857
0.4901
0.5040
0.4262
WPMI
0.3460
0.3878
0,5302
0.5267
0.4477
SoftWPMI
0,3664
0,3945
0,5257
0.5233
0.4525
Topl accuracy
8.50%
5.70%
15.90%
11.40%
10.38%
Rank reorder
36.30%
57.50%
89.80%
89.90%
68.38%
WPMI
23.80%
47.10%
87.00%
86.90%
61.20%
SoftWPMI
46.20%
70.50%
95.00%
95.40%
76.78%

---

Published as a conference paper at ICLR 2023
4.4 COMPUTATIONAL EFFICIENCY
Table 4 shows the runtime of different automated neuron labeling methods when tasked to label all the neurons of five layers in ResNet-50. We can see our method runs in just 4 minutes, more than 10, 60 and 200+ times faster than the baselines MILAN (Hernandez et al. 2022). Network Dissection (Bau et al. 2017 and Compositional Explanations (Mu Andreas 2020) respectively.
Table 4: The time it takes to describe the layers ['convl', 'layerl', 'layer2', 'layer3', 'layer4'] of ResNet-50 via different methods using our hardware(Tesk)) P100 GPU). We can see CLIP-Dissect is much more computationally efficient than existing methods.
Method CLIP-Dissect
Network Dissection
Compositional Explanations
MILAN
Runtime
3min50s
> hrs
hours
55min 30s
4.5 DETECTING CONCEPTS MISSING FROM D Drrabe
One surprising ability we found is that our method able to assign the correct label to a neuron even if D probe does not have any images corresponding to that concept. For example, CLIP-Disseds was able to assign the correct dog breed to 46 out of 1 118 neurons detecting dog breeds, and correct bird species to 22 out 59 final layer neurons of ResNet-50 trained on ImageNet, while using CIFAR- 100 training set as Drobe. which doesn'l 't include any images of dogs or birds. This is impossible for any label based methods like NetDissect (Bau al. 2017) and Compositional Explanations (Mu & Andreas 2020) (as IoU will be 0 for any concept not in D Drrobe). ). and unlikely for methods based on captioning highly activated images like MILAN (Hernandez et al. 2022) (as humans won't assign caption missing from activated images). Example labels and highest activating probe images can be seen in Figure
Final layer neuron #131
example picture
Final layer neuron #132
example picture
Ground truth: little blue heron
Ground truth: great white heron
CLIP-dissect: little blue heron
CLIP-dissect: great white
Highest activating images D paie (no bird images):
Highest images in D (no bird images):
CLIP-dissect matches the Ground truth label (bird classes) exactly, while using probing data that does NOT contain any birds images!
Figure 4: Example of CLIP-Dissect correctly labeling neurons that detect the little blue heron and the great white heron based on pictures dolphins and dinosaurs in CIFAR. CIFAR100 does not contain any bird images but CLIP-Dissed. can still get correct concept.
USE CASE OF CLIP-DISSECT
In this section, we present simple experiment to showcase how we can use CLIP-Dissect to gain new insights on neural networks. By inspecting the ResNet-50 network trained on ImageNet with CLIP-Dissect, we discover the following phenomenon and evidence for it: the higher the weight between two neurons, the more similar concepts they encode, as shown in Figure This makes sense since high positive weight causally makes the neurons activate more similarly, but the extent of this correlation is much larger than we expected, as each final layer neuron has 2048 incoming weights would not expect any single weight to have that high of an influence. A consequence of the similarity in concepts is that the second-to-lat layer already encodes quite complete repre- sentations of certain final layer classes in individual neurons, instead of the representation for these
8

---

Published as a conference paper at ICLR 2023
classes being spread across multiple neurons. For example Fig 5h shows that the neurons with highest outgoing weights already seem to accurately detecting the final layer concept/class label they re connected to.
To make these results more quantitative, in Figure Sp we measure the similarity of concepts encoded by the neurons connected via highest weights in the final layer of ResNet-50. For layer4 neurons, we used CLIP-Dissect to determine their concept, while for the final layer neurons we used the ground truth i.e. class label in text form. We can clearly see that higher weights connect more similar concepts together, and the average similarity decreases exponentially as a function of & when averaging similarities of neurons connected via the top weights. To further test this relationship, we found that the mpnet cos similarity between concepts encoded by two neurons and the weight connecting them are correlated with 0.1 and p-value < 10-300 (003300(probbbb (probability of no correlation is practically 0) when calculated over all 2 million weights in the final layer. If we only look at the highest 50000 weights, the correlation even higher with 0.258, p-value < 10-300 300
Layer 4
Weight
Final layer
Neuron #24, CLIP-Disses joans
x0.7371
Nauron #608
class:
Average top k weights
0.8
jean, blue jean, denim
Neuron #251, CLIP-Dissect rooling
Nsuron #858 class:
0.4
tile roof
CLIP similarity
#539, CLiP.Dissadt buses
x0.5910
Neuron #779
0.2
mpnet similarity
gl clase:
10
10a
10
10
10a
school
(a) Visualization of 3 highest weights of final layer.
(b) Average cosine similarity between concepts.
Figure 5: a) 3 highest weights of the final layer of ResNet-50 trained on ImageNet, we can see neurons connected by the highest weights are detecting very much the same concept. b) Cosine similarities between the concepts of neurons connected by highest weights. The higher the weight between neurons, the more similar concept they represent.
LIMITATIONS AND CONCLUSIONS
Limitations: The main limitation of our method compared to previous work that it's not taking advantage of the spatial information of neuron activations. This causes some difficulties in detecting lower level concepts, but we are still able to detect many low level/localized patterns as discussed method currently works well only concepts and images that CLIP
in Section A..3] Secondly, works well on, and while this already covers larger set tasks than what existing neuron label- ing methods perform well on, CLIP-Dissect may work of the box networks trained tasks that require highly specific knowledge such as classifying astronomical images. However, our method is compatible with future large vision-language models long as they share a similar struc- ture to CLIP, and CLIP-like models trained for a specific target domain. Finally, not all neurons can be described well by simple terms such as single word explanations. While we can augment the space of descriptions using a different concept set S, or creating compositional explanations as dis- cussed in Appendix [A.4, some neurons may have a very complicated function or perform different functions at different activation ranges. For the most part, current methods including ours will be unable to capture this full picture of complicated neuron functions.
In this work, we have developed CLIP-Dissect, novel, flexible and computationally efficient framework for automatically identifying concepts of hidden layer neurons. We also pro- posed new methods to quantitatively compare neuron labeling methods, which is based on labeling final layer neurons. We have shown CLIP-Dissect can match or outperform previous automated la- beling methods both qualitatively and quantitatively and can even detect concepts missing from the probing dataset. Finally we used CLIP-Dissect to discover that neurons connected by high weight often represent very similar concepts.
Conclusions:

---

http://biillllu2222220cceeie//mo Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENT
The authors would like to thank anonymous reviewers for valuable feedback to improve the manuscript. The authors also thank MIT-IBM Watson AI lab for computing support in this work. T. Oikarinen and T.W.. Weng are supported by National Science Foundation under Grant No. 2107189
REFERENCES
David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection: Quantifying interpretability deep visual representations. Computer Vision and Patlern Recognition, 2017.
David Bau, Jun-Yan Zhu, Hendrik Strobelt, Agata Lapedriza, Bolei Zhou, and Antonio Torralba. Understanding the role of individual units in deep neural network. Proceedings of the National Academy of Sciences, 117(48):30071-333
2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-248225555 255. leee, 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint 2020.
arXiv:2010. 11929, 1929,
Dumitru Erhan, Yoshua Bengio, Aaron Courville, and Pascal Vincent. Visualizing higher-layer features of deep network. University of Montreal, 1341(3):1, 2009.
Gabriel Goh, Nick Cammarata t, Chelsea Voss t. Shan Carter, Michael Petrov, Ludwig Schubert, Alec Radford, and Chris Olah. Multimodal neurons in artificial neural networks. Distill, 2021.
doi: 10.2391 siddistill.00300. https://distilllpub//22 /multimodallneuron..
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 770-778, 2016.
Evan Hernandez, Sarah Schwettmann, David Bau, Teona Bagashvili, Antonio Torralba, and Jacob Andreas. Natural language descriptions of deep visual features. In International Conference on Learning Representations, 2022.
Sébastien Marcel and Yann Rodriguez. Torchvision the machine-vision package of torch. In Pro- ceedings of the 18th ACM International Conference on Multimedia, MM '00 pp. 1485-1488, New York, NY, USA, 2010. Association for Computing Machinery. ISBN 9781605589336. doi: 10. 00.1145/1873951.187429
Jesse Mu and Jacob Andreas. Compositional explanations of neurons. Advances in Neural Informa- tion Processing Systems, 33:17153-17163 2020.
Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Feature visualization. Distill, 2(11);e7, 2017.
Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction to circuits. Distill, 2020. doi:
Hieu Pham, Zihang Dai, Golnaz Ghiasi, Hanxiao Liu, Adams Wei Yu, Minh- Thang Luong, Mingx- ing Tan, and Quoe Le. Combined scaling for zero-shot transfer learning. arXiv preprint arXiv:21 I. 10050, 2021.
Minh-Thang
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh. Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021.
10

---

Published as a conference paper at ICLR 2023
Kihyuk Sohn. Improved deep metric learning with multi-class n-pair loss objective. In D. Lee, M. Sugiyama, U. Luxburg. I. Guyon, and R. Garnett (eds.), Advances Neural Information Processing Systems. Curran Associates, Inc., 2016.
Zeyu Wang, Berthy Feng, Karthik Narasimhan, and Olga Russakovsky. Towards unique and in- formative captioning of images. In European Conference on Computer Vision, pp. 629-644. Springer, 2020.
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.01917, 2022.
Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, al. Florence: A new foundation model for computer vision. arXiv preprint 1:X1v 2021.
Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European conference on computer vision, pp. 818-833 Springer, 2014.
Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. Lit: Zero-shot transfer with locked-image text tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18123-18133, 2022.
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Object detectors emerge in deep scene cnns. In ICLR, 2015.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 mil- lion image database for scene recognition, IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017.
11

---

Published as a conference paper at ICLR 2023
A
APPENDIX
A.1 SIMILARITY FUNCTION DETAILS AND DERIVATION
Rank reorder hyperparameters:
The results of Table are using top 5% of most highly activating images for each neuron and using p 3 for the nn-rm..
WPMI:
In this section, we show that choice of similarity function s sim(km; 9k: P) can be derived based on the weighted point-wise mutual information (wpmi). Note that wpmi is also used in (Hernandez) ct al. (022)) but in different way -ur approach can compute wpmi directly from the CLIP products P and does not require any training, while (Hernandez et al. 2022) train two models to estimate wpmi.
To start with, by definition, the wpmi between concept fm and the most highly activated images Bg of neuron can be written as
wpmi(/m: (x)) log BB) - A plgm(mm)
(5)
Here Bg the set of images that most highly activates neuron k: i.e. the top indices of qe- First we can compute p(fmml [z)) = softmax(a
where softmax(2),, Σ 2 e with 2 E RR Pt. the
e'n
i-th row vector of the concept-activanl activationn matrix and is a scalar temperature constant. This the probability that CLIP assigns to a concept im for image z; when used as classifier.
We then define pm B&) as the probability that all images in Bş have the concept tm, which gives us pm BBx) = NT, Ma,,eB,Plle (lm (x.). Thus, we have
log (xx)
Σ Z.EB.
logp(tmlee
(6)
which is the 1st term in Eq Next, we can approximate the 2nd term p((m) in Eq (5) as follows: p(tm) is the probability that random set of images B will be described by fm- Since we don't 't
know the true distribution for a set of images, an efficient way to approximate this is to average the probability of fm over the different neurons we are probing. This can be described by the following equation:
p(tm) = EB Ep(p((tmBB)) ~ EmecD(tmlB.) Ejec ((m
(7)
|C|
CC]
where is the set of neurons in the layer we are probing. Eq. (5) to compute wpmi through the CLIP model:
Thus we can plug Eq. (6) and Eq.
in
wpmi(tm, (k)
=
log logp((m|/z;) - A log
Σ
+ A log [C]
(8)
B.EB
jEC
So the above Eq 8 in our CLIP-Dissect and set sim(tm qai P) wpmi(tms 9k) in the algorithm.
For our experiments we 2, À 0.6 and top 28 most highly activating images for neuron k as B& which were found to give best quantitave results when describing final layer neurons of ResNet-50.
SoftWPMI:
SoftWPMI is an extension of wpmi as defined by Eq. { into settings where we have uncertainty over which images should be included in the example set Bg. In WPMI the size of example set is defined beforehand, but it is not clear how many images should be included, and this could vary from neuron to neuron. In this description, we assume that there exists a true Bg which includes images from D probe if and only if they represent the concept of neuron We then define binary indicator random variables X [x, E B&] which take value if the ith image is is in set the Bg. and we define X (XF, xk1
12

---

Published as a conference paper at ICLR 2023
Our derivation begins from the observation that rewrite (Bkm from above
px) 11., ((ml 11., prron p( [e 1l P(
(9)
Now:
Exxlp(tm (o:(tt//:;)) x11 = p(zs E (k)p((ml(ss) + +1 p(a; E B&)) = p(a, E Bx)(p(tmlzz) 1)
(10)
If we assume the X are statistically independent, we can write:
E [xx(p(tml [B)]] E prraha Ext (p(tml/xs) [x) II, E p(a, E Bx)(p(tm(xz) 1)]
(11)
log x (p(t [B)]]
E
log(1 + p(a; E (e)(p(tmlzz)) - 1))
(12)
24 E S.EDDproe prrbbe
Note Equation
10 goes to if p(æ; ==)= (i.e. no effect a product) and p(tm|a)) (xy)
p(a; E [Bk) So Eq. [2 reduces to Eq. (6 of standard WPMI if p(æ; B&) is either or 0 for
all 2; E Drrohe- other words, we are considering "soft" membership in B₃ instead "hard" membership of standard WPMI.
To get the second term for wpmi, p((m), i.e. probability that text 4m describes a random example set Bk, approximate it like we did in Eq. (7) by using the example sets for other neurons are interested in.
Ejec Ex (p(tm B,)]
pllm) = EB, [x (m(mm B:)]l [.)]l
101
Ejeo Ex. ())]
Ejec IoE [1 p(x E B, (p((m (z) - 1)
(13)
IC1
|C|
Finally, we can compute full SoftWPMI with Eq. 12) and Eq. in CLIP-Dissect:
13
and use as similarity function
soft_wpmi(lm, (b))
log(1 + p(a; E Bx)(p((m(zt) 1 1))
I.E pprahe
-Alog
Σ II Derocc
[1 + 11pppx B,)(p(mmlz)) - 1}] + Aloog
(14)
jEC
One thing we haven't discussed is the choice of p(a: Bg). There is flexibility and this prob- ability could be derived from the activations of neuron it on image æ, by for example by taking scaled sigmoid, or it could be based on the ranking of the image.
For our experiments we found ranking based probability to perform the best, and used p(x E Bx) linearly decreasing from 0.998 of the most highly activating image for neuron & to 0.97 for 100th most highly activating image and for all other images. Thus in practice we only have to use the 100 images when calculating SoftWPMI instead of full D Dprote which is much more computationally efficient. For other hyperparameters we used a = 10 and 1.
A.2 ADDITIONAL QUALITATIVE RESULTS
Additional visualization on ResNet-16 and ResNet-50 in Figs 6 and continued from Section 4.1
13

---

Published as a conference paper at ICLR 2023
Resnet-18(Places 365) Layer 4
Most interpretable neurons (Network Dissection)
Most interpretable neurons (CLIP-Dissect)
ClisDds
MILAMIS::
MLLANiC: siricies
MILANDO
Roams
VIL MELAAGI Windowe
MLAMBI:
Tran trace train tracke
MLANIC Regins emaighe fratured
MLANNDS
Water fountarn
HILANG:: Founteirs and water
Kwaan aal: DIP-DRANCE
CLP Ornsct: attic
NetDissect:
HILAMI Dog heads
MLANIO Doga
MLANISO Buidings
VELANG:: Windows
Neuron CLI Diseet: doseet Closets
NetCkeseet: Ebsets
Neuron CLisDisset:: auditonu
NetDisect: conference
MLLAMI
MILANII dathing
MLLANEN
VILANG] fumiture
Neuron
CLIP DisecE bike vetscies
NetEROSeCT: wheel MIL ANID Whenls
Neuron 350:
CLP Dissect battroom
NetDiseet: bathrooms
MILAMAA Fumiture
VELANGH Ragira with straight feature
Reuron 201:
CLI DuP-Dusect--omm mutititoons
NatDhssect: art gallerys
Neuron 476
Dipsect worketations NetDisect: affices
HILAMII:
MILANIS
Furiiture
VILANGI: fummture
Neuron 303:
CLP Dosett buses
HeEDDsseet:: Dus
Neuron 416
CLI Diesect Mnchenss
NetDissect: Michenn VILANNI: Numiture
MLLASS::
MLANIBO
Neuron CUI--itrect buses
NetCkSSACT bes
Neuron 3:
CLP Diseet: audiorium
NcDossedt: auditoium
MLANEBO
Nouron Disect kayaking
Neurnn Clssett amii
NetDissech antics
Nourn
styscrazers MLLNND. towers and
Neumn
CLP Dissect mosque
NetDissech mosque outiioon
Towers
Figure 6: Explanations of most interpretable neurons in the second to last layer of ResNet-18 trained on Places365. Displayed together with 5 most highly activating images for that neuron. We have subjectively colored the descriptions green if they match these images, yellow if they match but generic and red if they do not match. Both Network Dissection and CLIP-Dissect do very well while MILAN struggles to explain some neurons. MILAN(() is trained on both ImageNet and Places365 networks, while MILAN(i) is only trained ImageNet. Both MILAN networks perform similarly here but the ImageNet version misses/is too generic for more neurons, such labeling bus neuron "vehicles", The neurons on the left have highest IoU according MILAN while neurons on the right have highest similarity to the concept according to our similarity function.
14

---

Published as a conference paper at ICLR 2023
Resnet-50(lmageNet) Layer 4
Most interpretable neurons
Most interpretable neurons (CLIP-Dissect)
(Network Dissection)
kitchens
MILANIBI: Antemals
MILANpp:: Arimals
MLAXIbb:
Fumbtrr and electroncc MIANNOK Talevtsion monitors
Neuron 465:
CLIP- nxwect
MenDiesect: cobwebbed
Neuron 516:
Diss booksshop
NenDheeet:: beslestore-s
MI MIJANIbI:
Items with straight features
MILANiph ttams of central focus
MLANUb
Stadlees and boots
MLANNsh Bookkaasss
Neuron 632: AMthi
CUP-Orasect: lattice
NenDissect: heneycombed MILAMip:: objects
Neuron 210:
CLP Diesect coliter
NacDlesect: spiraled MLAVipt Bama focus
MIANDS
Items that connected
76:
MenDissect: robaetbed
972:
CLI--Bissect gymnastics
Nettlisect: bexing
Disseis Animale food
ML MIAANip: Food,
MLANIai: People
1966: sueet MIAANib: Simiar color potterss
NeDisseec initted
Dissect
MIAANlb cream and animals
MLANNII:
Books
MLANNoo: Shevees
Neuron 2742: suect latice
MecDissect: waffled Mi Amip): hood
Neuron 1108: CLIP Dissect braccol
Netoisect filly MLANiph Marns
MILANII:: Items with circuaar features
MLANIbE:
oreen onsects
Neuron 2919; CUP siseect spider Bugs
MesDissect: cobwebbed MLAMMpp::
Neuron 76:
CLI Dissect: web
Nccoissec:: cebwebbed
CLIP latice
nentiiseet:
Dissect: kitchens
stove
MLANNih
MI ANbbI Rourded
MLANIp): Cirtular cloetts
MIANDS
Fabriic
MIAANio: Food, people, ciam
Figure 7: Explanations of most interpretable neurons in the second to last layer of ResNet-50 trained on ImageNet. Displayed together with 5 most highly activating images for that neuron. We have subjectively colored the descriptions green if they match these 5 images, yellow if they match but are too generic and red they do not match. Both CLIP-Dissect and Network Dissection perform well on these most interpretable neurons except for few failures by Network Dissection, while MILAN often gives concepts that are too generic. MILAN(b) is trained on both ImageNet and Places365 networks, while MILAN(p) is only trained on Places365. We struggling more with concepts like spiders, indicating issues with generalization.
the Places trained model is
15

---

Published as a conference paper at ICLR 2023
A.3 Low LEVEL CONCEPTS
In this section we show additional results of probing low level concepts, using two networks trained on ImageNet, ResNet-50 and ResNet-115. 152. For ResNet-152 we also compare against human annota- tions for these neurons from MILAnnotations (Hernandez et al., (022).
The results for ResNet-152 can be seen in Figure 8 We can see CLIP-Dissect able to accurately detect many lower level concepts, such as colors in Conv neurons 1,3,10 and Layerl neuron 4, well detecting that neuron 3 of Layerl activates specifically for the text/label, without having access to the activation pattern, while MILAN fails to detect this.
However, CLIP-Dissect does also have some failure modes on lower level patterns.
This leads to higher level outputs than desired. For example: CLIP-Dissect gives the concept underwater to convl neuron 16, while the concept probably more similar blue (human annotators also made this mistake), or conv1 neuron 28 where CLIP-Dissed outputs zebra while the neuron is likely just detecting stripes. The worst example of this is neuron convl neuron 24, which simply activates on white background, but this is entirely missed by CLIP-Dissect as it's not good focusing on the background.
Failure to differentiate between concept and correlated objects:
Unintrepretable neurons: Some neurons to be not interpretable, e.g. Conv1 neuron 2. CLIP-Dissect outputs music which seems incorrect, but neither human annotators or MILAN were able assign clear concept for the neuron either. See Appendix A.8 for more analysis on uninterpretable neurons.
Similar observations also hold for the ResNet-50 model as can seen in Figure Note we do not have human annotations to compare against for this network.
16

---

Published as a conference paper at ICLR 2023
ResNet-152(ImageNNtt Conv1
Highly acthveing images (masked by activation)
Highly activeting images
Human: 'camic bank text and drawings, flower pedale, yellew flowess and 'yellow the coiours'
heuron
CLI Dissect: yallow
MILAN: tellow and green spots pictures
music
Human landio eculpment, 'nothing highllghted!, 'nothing'
brown colored objects
WARNING
CUP Dissecti red
ared "the
MILAM: objects
bright
Human: "lower pedais' flowers, peosle, and birds are colorful and are all action', biids Bying a sky
Keuron 10:
CUP Discect: purple
Human: 'shpppin Huorescent and yellow objecte! 'green
MILAN: Green objects
in
Keuron 16:
CUP-Diesact: underwater
Human: lunderwater- realine', tablles and shirt!' 'antmals the see!
MILAN:
"white background white
MILAN: White spaces on top geometrical
WARNING
WARNNNGG
heuron 28:
CLI Dissect: aebra
Human: animall, 'sleck zebro
MILANI Black colored
ResNet-152(ImageNet Layer1
Human: top bottles; words, 'the bottles, shark, food and
label
lanaffams ane different shapes
left side
Human: gultar, animal', lorange objects including snakes and vehicles, lrange tint scenery'
Neuron 4.
CUP-Dissect orange
MILAN: Orange and yellow objetts
Figure 8: Descriptions for select neurons in early layers of ResNet- 152, showcasing both the suc- cesses and failure modes of CLIP-Dissect. For this figure used max the summary function for CLIP-Dissectro be comparable to MILAnnotations. For evaluation we used the MILAN model trained only Places models avoid overfitting.
17

---

Published as a conference paper at ICLR 2023
ResNet-50 (ImageNet) Layer1
Most interpretable
Most interpretable neurons (CLIP-Dissect)
(Network Dissection)
Neuron CUP Diosechh Netbesech banded MLAN
P-- D-Ouseech snorkeling kettissect wined
AEAME venical Eses
MILAMRS prorescent bue and green ab MLLANBl slut 20lects
gress
ABLAN//) Green aro35 atcure
NLAWS::
MLLAMCS Green arbas picure
HILANB) Green oblects
Neuron CLi--Disoect strpes
NetDissect, lined
Neuron
SuP-Oissect purple
kettlesect, parplett
Neuron 253: CLi--Diseet: magerta
NetDissect: redic MLA pink
Nearon 147: MLAANS
Culggiseet zebre
hetDissect: chequered MLAAN
CLi-otssect ME AXIbA unjects
MLAARSS Red objects
MILAMSS Oreen prez paiure
MLAMet: Bhi
CLi-Oiieet: shorkeling MI AMtot Underwatte Uh
AMtt brein areas skcuare
LAW els Groem
MLANish: Sue aboects
Neuron (iP-Dttsert pumle
Netthosech: purple-
Neuron CuP--lssect: sporleling
kettlisect:: velned
Neuron 19:
CLI Dissert pink
NetDisoects redi MLAN pi nee oblects
Nouron 63: C.P Dissect: stipes
ketDiszect: banded
MILAMI to oojects
MLANG):: 2trpes
Dii--issect:
diequred
Mil ANIio Dreen oblects
Hi MLA gi: green and blue objects
MI MLANSS) spets the calor wate
MLANis: 200 or poles
t'eaa orange
prange
Diseet MLANCS Pursle, rel Bue Bupresce MLAMDE: lod and pank cbietts
Ni LLA Red
Figure 9: Descriptions of most interpretable neurons (highest similarity/loU) of early layer in ResNet-50.
18

---

Published as a conference paper at ICLR 2023
A.4 COMPOSITIONAL CONCEPTS
In the sections above our method has focused on choosing the most fitting concept from the pre- defined concept set. While changing the concept CLiP-Dissect is easy editing text file, we show it can also detect more complex compositional concepts. We experimented with generating explanations by searching over concatenations of two concepts in our concept space. To reduce computational constraints, we only looked at combinations of 100 most accurate single word labels for each neuron. Example results are shown Fig 10 While the initial results are promising, some challenges remain to make these compositional explanations more computationally efficient and consistent, which is an important direction for future work.
Loyer4, neuron 386
Loyer4, neuron 1617
Original: textile
Compositional: flores embroldery
Original: tournament
Compositional: ping athlete
J0G
Figure 10: An example of compositional explanations generated by our method for two neurons of ResNet50 trained on ImageNet.
A.5 VISION TRANSFORMER
Since our method does not rely on the specifics of CNNs in its operation, we can easily extend it to work on different architectures, such as the Vision Transformer, specifically ViT-B/16 (Dosovitskiy et al. 2020) model trained on ImageNet. We have visualized most interpretable neurons, their highly activating images and their descriptions Figure Interestingly, we found the highly interpretable neurons to be very location focused, i.c kitchens or highways despite the network being trained on object level labels (ImageNet).
19

---

Published as a conference paper at ICLR 2023
ViT-B/16(lmageNett Most interpretable neurons
Neuron Diesect bethroom
Dissect: flelds
Neuron 82:
CLIP Dissect: Michens
CLI Dissect: becrooms heuron 748:
auditorum
Neuron 185:
CLI Dissect: ktchens
Neuron CLI--Dtsect:: fumished
Ri--haasedhhhhhauay highoays
hearon 752:
CLUP Dissect: room
bathroom
Neuron 727:
CUI-Dissect: kitchee kitchen
heuron
CLI Dissect: titchens
heuron 236:
CLI Dissect: Michen
Neuron 317:
CLi Dissect: Desrrcdtttesroos Redrooms
Neuron 363
Dissectstttt.coo roo
Neuron 457: CLI Dissact:
Figure 11: Most interpretable neurons after the 'encoder' module (one of the last layers) of Vision Transformer.
20

---

Published as a conference paper at ICLR 2023
A.6 PREDICTING INPUT CLASS FROM NEURON DESCRIPTIONS
In this section we follow reviewer suggestion and study whether the class of the image can be pre- dicted based on descriptions of highly activating neurons. We perform an experiment the neurons in the second-to-last layer (last hidden layer) and study if the descriptions of highly activating neu- rons in this layer can be used to predict what class the input is from.
As we show in below experiments, the neuron description generated from CLIP-Dissect gives overall higher prediction accuracy than the neuron descriptions from prior work (Network Dissection and MILAN), which suggests our proposed method generates better neuron description than prior work. Below we outline the steps we used to predict the class of an image based on the internal neuron descriptions from different neuron labelling methods:
1. Following our notation in Sec 3, for an image Tú, record the activations of neurons in the second-to-last layer g( ((j((z))) (where denotes the neuron index) well the predicted class c.
2. Find the neuron with the highest positive contribution to the predicted class c, which can be computed as k arg max, Wa, (xA)(
3. Obtain the description 4 for neuron k using an automated description method.
4. Find the class name that is most similar to the description of the highest contributing neu- ron, and predict this class as the images class. Similarity measured using an average of cosine similarities in the sentence embedding space of CLIP text encoder and mpnet sentence embedding space discussed in section 4.2
We performed this experiment ResNer-50 trained ImageNet, and found that predicting with the above algorithm and CLIP-Dissect neuron descriptions we were able to correctly predict the class of 10.28% of the images. In contrast, when we used descriptions from Network Dissection we were only able to predict the class of 3.36% of the images, and with MILAN(base) only 2.31 % of the time. This gives evidence towards our CLIP-Dissect descriptions being higher quality than Network Dissection and MILAN. It's worth noting that overall we would not expect to reach very high predictive accuracy using the above method as the most contributing neuron often does have completely different role than the target class. However if we study the same network with different neuron description methods, we would expect a better neuron description method would return a higher prediction accuracy, assuming at least some of the important neurons indeed do have similar role to the class itself. This method has the benefit of being automated, going beyond visualizing few most highly activating images and being able to analyze hidden layer description quality. We think methods like this are an interesting future direction for evaluating neuron description methods.
A.7 VISUALIZING WIDER RANGE OF ACTIVATIONS
So far our qualitative evaluation has focused on whether the description matches the 5 or 10 images in D Prooe that activate the neuron the most. However this does not give full picture of the function of the neuron, and in this section we explore how the neurons activate on a wider range of input images. In particular, in Figures 12 13 we visualize the most interpretable neurons of two layers of ResNet-18 (Places 365) and ResNet-50 (ImageNet) by uniformly sampling images from the top 0.1%, 1% 5% of most highly activating images. We can that the descriptions tend to match quite well for the 0.1% of most highly activating images, but the top 1% and top 5% images start to be of quite different concepts only slightly related to the description. We also notice that low level concepts like colors tend to be more consistently represented in top 1% and 5% images while higher level concepts are not. This result is somewhat expected the Prrohe does have that many images for each higher level concept, but highlights the need to explore neuron activations beyond just the few most highly activated images.
21

---

Published as a conference paper at ICLR 2023
ResNet-18
Images
ResNet-18 Layer
Neuron 23: CLI Dissect: snorkeling
Neuron 432: CUP Diss-ctssett: attic
Top
%% (-113)
Top
(-1130) 301
Top 5% (-5650)
Neuron 22: LUP-Dissect. green
Neuron CLI Dissect: fountains
Top
0.1% % (-113)
Top 1% (-1130)
Top 5% (-5650)
stripe
Top
Too 0.1% (-113)
Top 1% (-1130)
5% (-5660)
D--sisectt
Neuron CUP-Dissect: auditorium
Top 5
Top 0.1%) (-113)
Top 1% (-1130)
(-5650)
Figure 12: Most interpretable neurons ResNet-18, showcasing randomly sampled images of wide range of most highly activating images for that neuron.
22

---

Published as a conference paper at ICLR 2023
ResNet-50 (ImageNet) Layer
Images
ResNet-50 (ImageNet) Layer
Neuron 38: CLI Dis-ettsect:: snarkeling
Neuron 720: CLIP ELP--issecct kitchens
Top
(-113)
Top 1% (-1130)
Top 5% (-5653)
Neuron 73: CLis-Dissect: green
Neuron 516: CLIP Dissect bookshop
Top
Top 0.1% (-113)
1% (-1130)
(-5850)
Neuron CLIP- CLI--Disect: purple
Neuron 210: CLiP-Dissect: collier
Top
Top 0.1% (-11a)
1% (-1130)
Teo 5% (-5850)
Neuron 147: CLIP-Dissect: zebra
Neuron 972: CL CLil-Dissect gymnastics
Top 5
Top 0.1% % (-113)
Top 1% (-1130)
WM
(-5650)
Figure 13:
Most interpretable neurons ResNet-50, showcasing randomly sampled images of wide range of most highly activating images for that neuron.
23

---

Published as a conference paper at ICLR 2023
A.8 INTERPRETABILITY CUTOFF
Our method can also be used to quantify which neurons are "interpretable', Since each description for a neuron is associated with similarity score, and the higher the similarity the more accurate the description, we consider neuron & with description / as interpretable if sim(4, ys; P) > т. To choose the best cutoff т, we leverage our experiment section A.10 particular, we choose the lowest such that interpretable neurons will have an average description score of 0.75 or higher (compared 0.655 of all neurons). This gives us a cutoff threshold 7 0.16 with the proposed SoftWPMI similarity function. In contrast, the neurons with SoftWPMI - have an average de- scription score of 0.5257, which is lower than that of interpretable neurons (0.75) and neurons (0.655). Thus, it suggests that the similarity score of SoIIWPMI is useful indicator of description quality. Using this cutoff т, we find 69.7% of neurons in ResNet-18 (Places-35) 365) and 77.8% of neurons in ResNet-50 (ImageNet) to be interpretable, indicating that around 20-30% of neurons do not have a simple explanation for their functionality, i.c. are 'uninterpretable..
A.9 QUALITATIVE EFFECT OF SIMILARITY FUNCTION
Figure 14 shows the descriptions generated by our CLiP-Dissect when using simple cosine similarity as the similarity function. As we can see the performance is very poor, only adequately describing one of the 8 neurons displayed, highlighting the need for more sophisticated similarity functions we introduced.
ResNet-50 Layer
1
ResNet-50 Layer
Neuron 46:
cos: cont
SofWWPM:: stripes
Neuron 1203: cos: cabinets
SoftWPMI: nursery
fittings
graduating
Neuron 242:
cos: blue
SoftPPMI: aquarium
Neuron 683:
cos: gartner
SoftWPMI: terrier
Figure 14: Explanations for same neurons as Figure I showeasing the qualitative difference between using simple cos similarity the similarity function for CLiP-Dissect best performing SoftWPMI similarity function.
24

---

Published as a conference paper at ICLR 2023
A.10 LARGER SCALE EXPERIMENT ON DESCRIPTION QUALITY
In this section we perform a larger scale analysis of the neuron description quality provided by CLiP-Dissect. We evaluated the description quality of 50 randomly selected neurons for each of the 5 layers and models studied, for total of 1000 evaluations. Each evaluator was presented with 10 most highly activating images, and answered the question: "Does the description: '(" match this set of images?" An example of the user interface is shown in Figure 13 options which we used with the following guidelines:
Each evaluation had three
Yes Most of the 10 images are well described by this description
Maybe Around half (i.e. 3-6) of the images well described, most images de- scribed relatively well (accurate but too generic, or slightly inaccurate)
No Most images are poorly described by this caption
Question 1/50. Description: red
Does the description: 'red' match this set of images?
yes
maybe
no
Submit
Reset
Figure 15: Example of the user interface for our evaluation of description quality.
These evaluations were turned into numeric score with the following formula: yes: maybe:0.5, no:0. shows the average description score across different neurons and evaluators for each of the layers evaluated. This average score can be thought of as the percentage of neurons well described. We observed that overall the descriptions are good for 55-80% of neurons depending on the layer, with the average score across all evaluations being 0.655. In addition we notice that the very early and very late layers are most interpretable, corresponding clear low or high level concepts, while the middle layers seem to be harder to describe. It is worth noting that we are evaluating random neurons here, i.e. the neurons are selected randomly, so the displayed neuron may not be interpretable in the first place in many cases when the description does not match are because the neuron itself is not 'interpretable", i.c. there is no simple description that corresponds to the neurons functionality.
Even with the evaluation guidelines described above, these evaluations are subjective, and we found that two evaluators agreed 68.4% of the neurons with the vast majority of disagreements being between yes/maybe or maybe/no, with only 2.4% of neurons having yes from one evaluator and from another. For transparency, have included all 50 neurons, their descriptions and two
25

---

Published as a conference paper at ICLR 2023
The average description scores of the CLIP-Dissect descriptions for neurons of different layers. An average score of 1.0 indicates all descriptions match the neurons highly activating images, 0.0 means none do, and 0.5 could mean anything from all neurons 'maybe' match or half the neurons fully match and half don't at all.
Table 5:
Layer
convl 1
layer1
layer2
layer3
layer4
ResNet-18 18 (Places 365)
0.805
0.635
0.635
0,410
0.695
ResNet-50 (ImageNet)
0.815
0.670
0.550
0.640
0.695
evaluator's evaluations (E1, E2) of these descriptions for ResNet-50 layer convl in Figures I6 1 and and for ResNet-50 layer4 in Figures [9. and 21
26

---

Published as a conference paper at ICLR 2023
CLIMiisaett:
Elyye
EZiyex
Neurn
I9-lieect sumkeling
Flyess
#2ymeyhe
CLIP Clspedtseect:
Elimybb
yl:
EZimaybe
Nouron
CLiP-Oissed: orange
yls
E2iyes
Netron
CLiP-Oised:: yellow
Elmmayba #2:yes
y2:yes
E2:yes
4:39
Neuron 10:
CLIP Dissect: aa
El:ne
E2:no
Nouron 12:
CLIP-Oissect: underwater
Eliyes
E2:yes
workcheet
CLIP-ttseet:: yellow
Elayes
E2:yas
Neuron 13:
CLIP Dissect: red
ya:yes
U2/Yes
CLIP-Dissect:
El:yes
E2iyes
Nouron L8:
CLIP Dissect: pink
El:yes
E2ayes
Nauron Lt::
CLIP CisPeeesedt::
El:no
E2:no
wnvelength
Figure 16: Random neurons of 'convl' in ResNet-50 (ImageNet), and E1, E2's evaluation the description of CLiP-Dissect (PART 1).
27

---

Published as a conference paper at ICLR 2023
Neuron Zi:
CLIRkOiseet:: darkness
El:yes
E2.yes
4:33
Neuron 22:
CLIP Discect: aquariam
Elayes
E2
CLIP-Dissert:
CLIP-Otizect: snorkeling
El:yes
Neuron 30:
CLIP Dissect worksheet
El:yes
E2 yes
Neuron 21:
CLIP-Diszect: snorkeling
Ellyes
maybe
worichect
CLIP-Bissec:: snorkeling
El:yes
yes
Neuron 36:
CLIP Dissect: acommodzzlerst.mo
E2ino
Neuron 37:
CLIP-Diseett: workshect
Ellyes
E2imaybe
Neuron
CLI--Disset:: worksheet
Elomeybe
yellow
Neuron 41:
CLIP-Diseet: underwater
Cl:yes
[2:yes
Neuron 42:
CLI--iesec:: pink
Eltmayth mmyyyee
CLI--tisect:
Neuron 45:
CLIP-Dissect: underwater
Elmmyyb E2: yes
Figure 17: Random neurons of 'conv1' in ResNet-50 (ImageNet), and E1, E2's evaluation on the description of CLiP-Dissect (PART 28

---

Published as a conference paper at ICLR 2023
Neuron 46:
CLIMGissect: purchesed
Elayes
Z2:no
Niramnnd 47
INClower dertness
Flayes
Neuron dit
[PACinsect: worksheet
12ryen
equerium
12:yes
Elayes
12cyax
Neurrn $1:
CLIPCCiseei: turquaiie
EXomeyyo 12:yss
Netrrn 32:
CLIP-Cissed:: dar darkness
12mmybbe
12:yat
34:
CLIMGissed:: yellow
Elayes
Eztyes
Neuron bb:
CLIMEisset:: worksheet
El:yes
EZimaybe
Neuron 56:
CLI-Cissed:: magenta
Elsyes
E2yes
CLINDissed::
CLIP-Gisee::
ELlyas
E2:yas
9:33
Neuron S0:
CLIP-Gissed: statinnry
El:yes
E2:no
Neuron 62:
CLIP-Cisect: underwater
Ellyes
Elyes
CLIP-Gissect:
ELimaybe
Figure 18:
Random neurons of 'conv1' in ResNet-50 (ImageNet), and E1, E2's evaluation on the description of (PART) 3)
29

---

Published as a conference paper at ICLR 2023
Neuron III--isee::: intoois
Flona
F2
CLINiissed:::
CLiNNGisacd: groanhouse ouse
Elamoybi
Nouron 12:
CLIP-Gtssed:: baseneme
El:no
E2ino
Nzuron 58:
CLIP Discedt: pitchine
Elsmoviic Elymayoe
CLIMotssect: seimwear
Eltyes
Nzuron 159:
CLIP CLis-Oissect: tamel
Elimovbe mayoo
Nzuron 101:
CLIP-Oisset:: Infatablc
Ellyos
Elimayoe
CLIP-Oissec::
roofing
Neuron 330:
CLI Dissett: hound
Elimaybe maybe
Neuron 341
CLIPOissect: moxif
Elimaytie
Neuron CLIP-Oissect:
Elimaybe
mayoe
Neuron 534:
CLI--Otsect:: underweter
Eliyes
E2 yes
Neuron 542: CLIP-Oissec:: radiator
Ellyes
E2 (2iyes
Figure 19: Random neurons of 'layer 4" in ResNet-18 18 (Places 365), and E1, E2's evaluation the description of CLIP-Dissect (PART) 1)
30

---

Published as a conference paper at ICLR 2023
Neuron 845: CLI-- CUs--rosect reprints
Clayes
Camayybe
Neuron 853: CLI--Dosect ordered
Elemaybe 22:maytee
Neuron CLI-Diiseett
Clmmaybe
Elayes
E2ayes
Neuron 780:
CLIP Dispect kart
El:no
E2:maybe
Neuron 797:
CLIP Dissect safaris
Elayes
E2:yes
CLIP-Dissect
E2unaybe
pasture
Elayes
E2:yes
Neuron 013:
CLI--Diseact linens
Elayes
E2:naybe
Neuron 980:
CLIP-Dissact quitting
Elsyes
E2tmaybe
1061:
Elsmayba E2:maybe
CLI--iwwec::
Flayex
E2tyen
Neuron 12/0: CLI-Dissect squash
EJoyes
b:tyyes
Neumn 1>78:: IP-Diosect coller
Phono
Clayes
C2:yes
Neurnn 1335: CUP Die--Deseettto minolts
Elayes
EZryes
Neuron 1369: CUP- Dissett diconia
Elino
E2/no
Figure 20: Random neurons of 'layer ResNet-18 (Places 365), and E1, E2's evaluation on description of CLIP-Dissect (PART 2)
31

---

Published as a conference paper at ICLR 2023
1442: Ois-elssect:
EZamaybe
Neuron 1550: CLIP-Otsset:: brushes
Elimaybee E2mayye
Neuronn 1509: CLIP-Oisset:: diffs
EXemayte E2ayes
1645: CLP-Dissect:
12ryes
1662:
sbling
Flayes
12ryex
Neuron 1700: CL CLENGiveei::: paazle
Eltyas
E2ryss
Neturn 2710: CLPssissect; underng acuattes hlrmeybe
I.I-iiseet:::
Elayes
E2ayss
Neuron 1828: CLIP-Gissect: flgs
Elimaybe Ebmaybe
Neuron L835: CLIP Dissect: Bies
Elsyes
E2iyes
1912: CLI--Oiseet::
Ellyes
Elimayb
Neuron 2025: CLIP LIP-Olssect: gallon
Elimaybe E2imaybe
Figure 21: Random neurons of 'layer 4" in ResNet-18 (Places 365), and E1, E2's evaluation the description of CLIP-Dissect (PART 3)
32


Reviews:
[{"review_id": "hFtDXmRTiV", "reviewer": "Reviewer_xZV4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new method for describing the representations obtained in each neuron of neural networks. Unlike previous methods that require a dedicated dataset with annotations and thus can only detect concepts that appear in the dataset, the proposed method CLIP Dissect is able to describe the representation of neurons on the basis of the implicit knowledge stored in the pretrained large-scale vision-language model (CLIP). The experiment reveals that the proposed method can not only achieve the aforementioned goal, but also provide better descriptions than previous methods even on the predefined datasets.", "summary_of_the_review": "In my view, although the technical contribution of the paper is not that significant, the paper provides important extension in understanding the representations of neurons in neural networks. In addition, the paper clearly states its position and writing quality is good. Overall, I am leaning to positive on this paper, but I am open to the discussion with the authors and the other reviewers to make the value of this paper clearer.", "strength_and_weaknesses": "## Strength\n1. The paper provides sufficient background so that the position of the paper is clear. The difference with relevant works such as Network Dissection is well explained. Obtaining the capability of describing role of neurons without limiting the vocabulary to a predefined set is a good contribution to the community.\n1. The proposed approach is reasonable and technically sound. The method is so simple that readers can easily integrate the method into further studies in this domain.\n1. The paper is clearly written and easy to follow.\n\n## Weakness\n1. The technical novelty of the proposed method is limited since the method is rather straight-forward adaptation of CLIP for network dissection problems. The paper claims that it addresses the limitations stated in the last line of the 2nd paragraph in the introduction “Although these… new concept.”, but it is achieved mainly by the capability of CLIP pretrained model. In addition, the results presented in the experiment section is somewhat “as expected” given the CLIP’s capability. I acknowledge the paper’s empirical contribution, but the contribution in terms of the knowledge advancement in the community might be a little lower compared to standard ICLR papers. I would appreciate if the authors could discuss the novelty of the paper and new findings brought by the paper to the community.\n\nMinor points.\n1. The name “CLIP-Dissect” is emphasized with the bold font everywhere it appears, which I think is unnecessarily distracting.\n1. p3 “STOA” -> “SOTA”\n1. Please provide the citation for WPMI.\n1. p5 “among the three” -> “among the four”\n1. Section 4.3: “SoftPMI” -> “SoftWMPI”\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Ns-wGz4FqeR", "reviewer": "Reviewer_SGoY", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper addresses deep neural network explainability and specifically proposes a method for generating textual description of neurons in a pre-trained network. The method operates as follows: Given a pretrained network f, a \"probe set\" D (a set of images), a \"concept set\" S (a set of words/phrases) and a neuron in f, the method finds the concept in S that best describe the neuron, in terms of explaining what it \"does\"/responds to\" based on its activation map on the probe set D. The method works as follows:\n\n1. First, we use CLIP to compute image embeddings for D and text embeddings for S. \n2. Next, we compute the concept activation matrix P where the i,j entry in P is the the result of the inner product of the i'th image embedding with the j'th concept embedding. \n3.  For a given neuron k, we compute the mean of its activation on map on all the images in D and denote the resulting vector as q_k.\n4. Finally, we find the \"most similar\" concept t_l (in S), where the similarity function is computed using q_k and the matrix P. The paper explores several alternatives for a similarity function and provides detailed experiments comparing the different alternatives. \n\nThe paper provides quantitative as well as qualitative evaluations that demonstrate the improve performance, flexibility and runtime of the proposed method. ", "summary_of_the_review": "In my opinion, the paper presents a useful, general, flexible method fast method for addressing an important aspect of model explainability. The technical novelty of the paper is limited in the sense that it proposes a simple method which relies heavily on the strength of CLIP, but I do think the paper is good fit for ICLR.  ", "strength_and_weaknesses": "Strengths:\n1. The paper demonstrate a new and non trivial use case of strong pre-trained Vision and Language models which is, in my opinion, a contribution by itself to the field of joint multi-modal embedding. \n\n2. The proposed method is far more flexible than previous methods - leveraging CLIP's latent space, the method can work on practically any set of natural images and concepts without requiring any training or ground truth labels. The method is also dramatically faster than previous methods. \n\n3. The experiments demonstrate the methods flexibility and improved performance (compared to existing methods). The paper also demonstrates two non-trivial use cases of the method: detecting missing concepts from a given image set and investigating the connection between pairs different neurons in a pre-trained network.\n\n\nWeaknesses:\nEDIT: the rebuttal and the revised text (additional experiments and analysis) completely addressed the drawbacks I mentioned below. \n\n1. In my opinion, the quantitative analysis basically measures how well the method assigns the correct class labels for the neurons in the final layer, which is a very limited aspect of explainability. The authors discuss the limitations of different evaluation methods in the paper, but the fact is that there is no quantitative experiment evaluating the ability of the network to describe intermediate neurons in the network. I understand the limitations of having no ground truth, but I do believe designing a small user study could benefit the paper.\n\n2. minor comment: under \"Rank reorder\" similarity function, it says \"This is done by replacing the i-th largest element of P:,m by the i-th largest element of qk for all i.\". The resulting vector is a reordered version of q_k, wouldn't that mean that we replace the i'th largest element of q_k by the i'th largest element of P:,m? If I understand correctly, \"Rank reorder means re-ordering q_k using P:,m order\", I think the phrasing of this section can be made a bit more clear. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "DAoEncqovm", "reviewer": "Reviewer_XsCd", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a new way to label internal neurons of convolutional neural networks. The proposed method relies on the CLIP's vision and text embedding to generate the labels. This is an improvement over related work that relied on manually labeled datasets. For the evaluation, two networks are used: a ResNet-50 trained on ImageNet and a ResNet-18 for the Places-365 dataset. Their method is compared against two baseline methods on different datasets and tasks. ", "summary_of_the_review": "I think this paper makes interesting contributions and could be a good fit for ICLR. However, I doubt a few words can fully describe a neuron's functionality, and I think this mismatch must be analyzed or at least discussed. ", "strength_and_weaknesses": "**Strengths:**\n\n- The paper cleverly uses CLIP embeddings to generate labeled annotations for neurons.\n- Labeling internal neurons could be very useful for many downstream applications. For example, it could be used to monitor networks in production (e.g., does new data change the internal activations in unintended ways?). \n- The evaluation contains a good collection of qualitative and quantitative experiments.\n- Even though this new approach does not require any labels, it outperforms previous label-based approaches.\n\n**Weakness:**\n\n- My main concern is that this work did not measure the gap between the assigned meaning (as textual description) and actual meaning (as neuron activations). Previous works in that direction also did not analyze this gap in detail. For example, the following experiments could be used to measure this gap:\n\n  - How well can the final classes be predicted using the textual description of the internal neurons? This could also be used to evaluate different labeling strategies: the better the description, the higher the final score should be. \n  - Show that the most activating neurons correspond to the assigned concepts. Just providing the 5-most activating examples is not enough. I would like to see a uniform selection of most-activating samples from the top 0.1%,1%, and 5%-most activating examples per neuron. I would suspect that already the top 1% are quite diverse. \n\n- Similar to the previous comment: In Appendix A.3, it is stated: *\"Unintrepretable neurons: Some neurons seem to not be interpretable\"*. Could you provide a qualitative estimate on how many neurons are \"uninterpretable\" and how you measure this? \n\n- Limitations are only discussed in Appendix A.2. Please move them to the main paper. There is enough whitespace on page 9 to include them there. Furthermore, it should be discussed how well the meaning of a neuron matches the textual description. \n\n## ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "8qB_mkGw652", "reviewer": "Reviewer_Ujcx", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a network dissection method without using any labelled image data. This is achieved by utilizing a pretrained CLIP model to compute an image-concept similarity matrix. With the help of the matrix, the concept label for a neuron unit will be the one that maximizes the similarity between the neuron query and concept vector.", "summary_of_the_review": "Overall, this paper is an interesting application of CLIP model, but the main reason it works well is the pretrained CLIP's powerful representation ability.", "strength_and_weaknesses": "Strength\n- Since the introduction of CLIP model, its powerful representation ability has been proved by a variety of applications. This paper showcases an interesting application of CLIP model.\n- The experimental results are impressive and shows the effectiveness of the proposed method.\n\nWeaknesses\n- The contribution of the paper may be exaggerated. The superior results over existing methods may be largely attributed to the powerful representation ability of CLIP that is pretrained on hundreds of millions of image-text pairs.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%