CoSy: Evaluating Textual Explanations of Neurons
Authors: Laura Kopf
Philine Lou Bommer
Anna Hedström
Sebastian Lapuschkin
Marina MC Höhne
Kirill Bykov
Venue: NeurIPS 2024 poster (2024)
Keywords: Explainable AI
Evaluation of Explainability Methods
Mechanistic Interpretability
Primary area: interpretability_and_explainability
TL;DR: We propose CoSy, an automatic evaluation framework for textual explanations of neurons.

Abstract: A crucial aspect of understanding the complex nature of Deep Neural Networks (DNNs) is the ability to explain learned concepts within their latent representations. While methods exist to connect neurons to human-understandable textual descriptions, evaluating the quality of these explanations is challenging due to the lack of a unified quantitative approach. We introduce CoSy (Concept Synthesis), a novel, architecture-agnostic framework for evaluating textual explanations of latent neurons. Given textual explanations, our proposed framework uses a generative model conditioned on textual input to create data points representing the explanations. By comparing the neuron's response to these generated data points and control data points, we can estimate the quality of the explanation. We validate our framework through sanity checks and benchmark various neuron description methods for Computer Vision tasks, revealing significant differences in quality.

CoSy: Evaluating Textual Explanations of Neurons
Laura Kopfl
Philine Lou Bommer!
Anna Hedström 1,3,4 Sebastian Lapuschkin'
Marina M.-C. Höhne 3,5
Kirill Bykov!
'TU Berlin, Germany BIIOLD, Germany
UMM Lab, ATB Potsdam, Genmany
Fraunhofer Heinrich--Hert--nsiituee Germany
Uniiersitty of Potsdam, Germany
kopf@tu--belln..de
{phommer, awwdstrooe, xbyikov)aatb-ootedaa
sebastian www.scckihhhhhhhhh....cc. . orappuuccccciiiiii......oooo.... fraunhofer.de
Abstract
A crucial aspect of understanding the complex nature of Deep Neural Networks (DNNs) is the ability to explain learned concepts within their latent representations. While methods exist to connect neurons human- uumaan-unddrsaaddbblee textual descrip- tions, evaluating the quality of these explanations is challenging due to the lack of unified quantitative approach. We introduce CoSy (Concept Synthesis), a novel, srriitecturr-agnostte framework for evaluating textual explanations of latent neurons. Given textual explanations, our proposed framework uses generative model conditioned on textual input to create data points representing the expla- nations. By comparing the neuron's 's response to these generated data points and control data points, we can estimate the quality of the explanation. We validate our framework through sanity checks and benchmark various neuron description methods for Computer Vision tasks, revealing significant differences in quality. We provide an socr-source implementation on GitHut")
Introduction
One of the key obstacles the wider adoption of Machine Learning methods across various fields is the inherent opacity of modern Deep Neural Networks (DNNs)-in essence, we often lack an understanding of why these models make certain predictions. To address this problem, various explainability methods (2) have been developed to make the decision-maker processes of DNNs more understandable humans. Explainability methods have broadened their focus from interpreting the decision manking of DNNs locally- for instance, interpreting specific inputs using saliency to understanding the global behavior of models by analyzing individual model components and their functional purpose (7). Following the latter global explainability approach, often referred to as mechanistic interpretability X
maps 3
10, some methods aim describe the specific 16], enabling analysis of how these
concepts neurons have learned to detect [1 12
13
14
15
high-level concepts influence network predictions.
A popular approach for explaining the functionality of latent representations of a network is to textual concepts. A textual description is assigned to
describe neurons using human-understanme neuron based on the concepts that the neuron has learned to detect or is significantly activated by. Over time, these methods have evolved from providing label-specific descriptions [I]] more complex compositional [12] [66 and opea-vocabulary explanations [I3] [5]. However, significant challenge remains: the lack of universally accepted quantitative evaluation measure for open-vocabulary neuron descriptions. As consequence, different methods devised their own evaluation criteria, making it difficult to perform general-purpose, purpose, comprehensive coosscoompaissoss.
httpe://ggichub.coo//kkog/
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Problem: Evaluate Textual Explanations
Solution: Estimate Quality with CoSy Framework
Step 1:
What good explanation for the neuron?
Generate Synthetic Data
Step 3: Score Explanations
INVERT
Collect Neuron Activations
Sarteol delent
P. Dissect fruste
Jsn
0.4
MILAN
Neuton Activaaione
Neuron
(Reshiet1,, avgpool)
Figure I: A schematic illustration of the CoSy evaluation framework for Neuron 80 in ResNet 18's avgpool layer. The current challenge lies in the absence of general-purposs, quantitative evaluation measures to benchmark textual explanations of neurons. To address this, we propose CoSY, framework consisting of three steps: first, a generative model translates textual concepts into the visual domain, creating synthetic images for each explanation using tex--l--imae model. Then, inference is performed on these synthetic images alongside a control image dataset to collect neuron activations. Finally, by comparing activations from the synthetic images with those from the control dataset, we quantitatively assess the quality of the textual explanation and compare results across different explanation methods. The implementation details of this example can be found in Appendix A.2
In this work, we aim to bridge this gap by introducing Concept Synthesis (CoSY), the first automatic evaluation framework for textual explanations of neurons in Computer Vision (CV) models (illustrated in Figure []. Our approach builds on recent advancements in Generative AI, which enable the generation of synthetic images that align with provided neuron explanations. We use set of available text-to-image models to synthesize data points that are prototypical for specific target explanations. These data points allow to evaluate how neurons differentiate between concept-related images and non-concept-related images combined in a control dataset. We summarize our contributions as below:
(C1) We provide the first general-purpose, quantitative evaluation framework CoSy (Section 3) that enables the evaluation of individual or a set of textual explanation methods for CV models.
(C2) In series of sanity checks (Section we analyze the choice of generative models and prompts for synthetic image generation, demonstrating framework reliability.
(C3) We benchmark existing explanation methods (Section 5) and extract novel insights, revealing substantial variability in the quality of explanations. Generally, textual explanations for lower layers less accurate compared those for higher layers.
2 Related Works
Activation Maximization Activation Maximization is a commonly used methodology to under- stand what a neuron has learned detect [17]. Such methods work by identifying input signals that trigger the highest activation in a neuron. This can be achieved synthetically, where an optimization process is employed to create the optimal input that maximizes the neuron's 's activation [88 19 20),
or naturally, by finding such inputs within data corpus (21]. Activation Maximization has been employed for explaining latent representations of models [2 including probabilistie models (24)
detection of backdoor attacks (25 and spurious correlations 26 However, one of the key limitations
of this methodology lies its inability to scale, as it relies on users to manually audit maximization
signals.
Automatic Neuron Interpretation A more scalable alternative approach involves linking neurons with human-understanddee concepts through textual descriptions. Network Dissection [11] (NetDis- sect) a pioneering method this field, associating convolutional neurons with a concept based on the Intersection over Union (IoU) of neuron activation maps and ground truth segmentation masks.

---

Table I: Comparison of characteristics of neuron description methods. The columns (from left to right) represent the explanation method used, its textual output type fixed-label, compositional, or open-vocabulary) the type of neuron targeted for analysis (convolutional, scalar, or predetermined), the target metric the method optimizes (IoU, WPMI, AUC, etc.), whether the method relies on auxiliary black-box models for finding or generating explanations (img2txt model, CLIP), and whether the explanation method is architectue-aannosicc, meaning can be applied to any CV model.
For more detailed description of each method, refer to Appendix A.I
Method
Explanation
Neuron Type
Target
Bluck-Box Dependency
Archilectur-Agmostic
NetDissect
11
fixed-label compositional
IoU
CompExp
12
conv.
IoU
MILAN [3
upen-vocabulary
conv.
WPMI
img model
FALCON 14
open-vocubulary open vocabuuuyy compositional
predetermined scalar
avg. CLIP score
CLIP CLIP
CLIP- DisseDi
15
SoftWPMI
INVERT [16
scalar
AUC
Building on this, Compositional Explanations of Neurons (CompExp) enhanced explanation de- tail by enabling the use of compositional concepts-i.c. concepts constructed with logical operators. MILAN [13 further expanded this by allowing for apen-vocabulary explanations, permitting the generation of descriptions beyond predefined labels. INVERT (16] adopted a compositional concept approach, enabling explanations for general neuron types without the need for segmentation masks. assigns compositional labels based on neuron's 's ability to distinguish concepts, using the Area Under the Receiver Operating Characteristic Curve (AUC). FALCON [4 and CLIP-Dissect [15] compute image-text similarity with a CLIP model (27 for the most activating images and their corresponding captions or concept sets. sets. Each method defines its optimization criteria, lacking a unified consensus on what constitutes good explanation. For detailed descriptions of the methods and their optimization objectives, please refer to Appendix A.T An overview of the different techniques is illustrated in Table [
Prior Methods for Evaluation While significant effort has been made towards developing ap- there has been relatively limited focus on evaluating global methods, in particular neuron description methods. Currently, to the best of our knowledge, there is no unified approach that allows for benchmarking across models and explanation methods. In their respective papers, the INVERT and evaluated the accuracy of their explanations by comparing the generated neuron labels with ground truth descriptions provided for neurons in the output layer of a network. However, this evaluation limited to output neurons and fixed labels only. CLIP-Dissect additionally evaluates the quality of explanations by computing the Cosine Similarity in sentence embedding space between the ground truth class name for each neuron and the explanation generated by the method. FALCON employs a human study conducted on Amazon Mechanical Turk to evaluate the concepts generated by the method. Participants are tasked with selecting the best explanation for each target feature from selection of explanation methods, considering given set of highly and lowly activating images. MILAN evaluates the performance of neuron labeling methods relative to human annotations using BERTScores 31 While human studies are generally beneficial, the conventional setup can be misleading and may fail to fully capture the intended evaluation criteria, introducing potential biases. Typically, annotators describe the images that most strongly activate neuron, and these descriptions are then compared to an automatic explanation. However, this approach primarily evaluates the alignment with the most activating images rather than the accuracy of the explanation in describing the neuron's 's function. Moreover, these highly activating images may not accurately represent the neuron's 'ss overall behavior, as they only reflect the maximum tail of the distribution.
proaches and tools for evaluating local explanations (88 29 30
explanation methods
3 Method
In the following section, we introduce CoSy-a first automatic evaluation procedure for open- vocabulary textual explanations for neurons. We first define preliminary notations in Section 3.1 then
describe CoSy formally in Section 3.2
3

---

3.1 Preliminaries
Consider a Deep Neural Network (DNN) represented by the function 9 X - 2, where X' C Rhxuxx denotes the input image domain and Z C IR' represents the model's output domain. We can composition of two functions, Here y C. "dx xk" where d E N is the number of neurons in the layer, and w", hh the width and height of the feature map, respectively. The function F, which we refer to as the feature extractor, can be chosen based on the layer of the model aim to inspect. This could be an existing layer within the model or a concept bottleneck layer [2.. We refer to the i-th neuron within the layer as (1x)) (a(x)= : X + R=" x Within the scope of this paper, we refer to explanation method as an operator & that maps a neuron to the textual description s {ff) S, where S is set of potential textual explanations. The specific set of explanations depends on the implementation of the particular method (see Appendix A.II)
view the model
X y, and L y + Z, such that g = L
represent
3.2 CoSy: Evaluating Open- vven-Vocahulary Explanations
We assume that a good textual explanation for a neuron should provide a human- understandable description of input that strongly activates the neuron. However, modern methods for explaining the functional purpose of neurons often provide open-vocabulary textual explanations, complicating the quantitative collection of natural data that represents the explanation. To address this issue, CoSy utilizes recent advancements in generative models to synthesize data points that correspond to the synthetic images measured and compared
textual explanation. The response of neuron to the neuron's 's activation on a set of control natural images representing random concepts. This comparison allows for quantitative evaluation of the alignment between the explanation and the target neuron.
The parameters of the proposed method include a control dataset X.- {af..... x.) X, N. which consists of natural images representing the concepts on which the model was originally trained. Additionally, it incorporates generative model PM used for synthesizing images, along with specified number of generated images m E N. The control dataset typically includes a balanced selection of validation class images. Given neuron fa and explanation E S. CoSY evaluates the alignment between the explanation and neuron in three consecutive steps, which are illustrated in Figure [
1. Generate Synthetic Data. The first step involves generating synthetic images for given explanation s E S, which we use as prompt to generative model pM to create collection of synthetic images, denoted as X {=1 ... = pm (x s). This collection consists of m E N images, where m adjustable as parameter of the evaluation procedure.
2. Collect Neuron Activations. Given the control dataset Xn and the set of generated synthetic images X1, we collect activations as follows:
A (f(f((=))) ({((x))) E ER";
(1)
A =
e R""
where RR" xh* + R is an aggregation function for multi-dimennional neurons. Within the scope of our paper, we use Average Pooling as aggregation function
(y)
E y e DEYCR' CR"" kE(L, [EE[LA-
wh
3. Score Explanations. The final step of the proposed method relies on the evaluation of the difference between neuron activations on the control dataset Âg and neuron activations given the synthetic dataset Å₁. To quantify this difference, we utilize a scoring function V : Rd x R + R to measure the difference between the distributions of activations.
In the context of our paper, we employ the following scoring functions:
Area Under the Receiver Operating Characteristic (AUC)
AUC widely used pon-maramttric evaluation measure for assessing the performance of binary classification. In method, AUC measures the neuron's 's ability to distinguish

---

To ensure the reliability of our proposed evaluation measure, all steps within our framework need to be subject to sanity checks [33]. In this section, we analyze the following: (I) which generative models and prompts provide the best similarity to natural images, (2) whether the model's behavior between synthetic and control data points
V AAuc(Ao,AA)) EnEro EDEA, 1/a < b |Aol |Aa
Mean Activation Difference (MAD)
MAD is a parametric measure that quantifies the difference between the mean activation of the neuron on synthetic images and the mean activation on control data points
1 DEAL b En
V MAD A1)=
V n-1 EeEsg a)2
with mean control activation = "
These two chosen metrics complement each other. AUC, being non-parametric and stable to outliers, evaluates the classifier's ability to rank synthetic images higher than control images (with scores ranging from 0 to 1, where represents a perfect classifier and 0.5 is random). On the other hand, MAD allows us parametrically measure the extent which images corresponding explanations maximize neuron activation.
4
Sanity Checks
on synthetic and natural images differs for the same class, and (3) validating that CoSy provides appropriate evaluation scores for true and random explanations, given known ground truth class for the neuron.
4.1 Synthetic Image Reliability
One of the key features of CoSy is its reliance on generative models translate textual explanations of neurons into the visual domain. Thus, it is essential that the generated images reliably resemble textual concepts. In the following section, we present an experiment where we varied several parameters of the generation procedure and evaluated the visual similarity between generated images and natural ones, focusing on concepts for which we have collection of natural images.
For our analysis, we used only open-source and freely available text-to-image models, namely Stable Diffusion XL 1.0-base (SDXL) (34] and Stable Cascade (SC) [35). We also varied the prompts for image generation. To measure the similarity between synthetic images and natural images corresponding to the same concept, we used Cosine Similarity (CS) in the CLIP embedding space with the CLIP-ViT-BI the ImageNet validation dataset [36] For each [concept] we use five different prompts and employ them with SDXL and SC models, generating 50 images per class. We then measure the CS between image pairs of the same class.
model [27 We select set of 50 random concepts from the 1,000 classes in
Figure illustrates the comparison across all generative models and prompts in terms of CS of generated images to natural images of the same class. The results indicate that when using Prompt as input to SDXL, the synthetic images show the highest similarity to natural images. The performance generally best with the most detailed prompt (5) and closely aligns with prompts 1, 3, and Moreover, SDXL appears to be slightly more effectively realizing detailed prompts than Sc. As anticipated, the poorly constructed prompt (2) results in the lowest similarity natural images for both models. To address prompt bias and dataset dependency. we compare the object-focused ImageNet with the scene-focused Places365 (see Appendix A.3). We find that close-up prompts work well for object-centric datasets, while general prompts like "photo of" are better for bene-based datasets. not stated otherwise, for all following experiments, Prompt together with SDXL model was employed for image generation.
5

---

of batterrnn
1.a Pusrege
Siace CHIDS
SOOO
painting
photo
Gereepa
1.2WWC2022Z.C od Stande
photo Iassepe
S. realssic photo of diss up of
Coscepa
0.5
Coalre
Figure 2: An overview of the impact of varying the prompt on the similarity between natural and synthetic images, using two text-to-image models. Left: average Cosine Similarity (CS) across all natural and synthetic images over all classes are reported. Higher CS values are better, indicating greater similarity between the images, Right: an illustration of the visual differences produced by the SDXL and SC models response to diverse prompts for the concept "submarine". and natural images from the ImageNet validation dataset [36]. Our results show that both SDXL and SC generate similar images, with SDXL generally being more closely aligned with natural images than SC.
4.2 Do Models Respond Differently to Synthetic and Natural Images?
Given the visual similarity between natural and synthetic images of the same class, we investigate whether CV models respond differently to these groups and if the activation differences indicate adversarial behavior. To this end, we employed four different models pre-trained on ImageNet:
400. For each model, we randomly selected output classes and generated images per class using the class descriptions.
ResNet18 [37]. DenseNet161 [88, GoogleNet [39], and ViT-B/11 16
We pass both synthetic and natural images through the models, collecting the activations of the output
neuron corresponding to each class.
Figure (a) illustrates the distributions of the MAD between synthetic and natural images for the same class across the 50 classes. Across all models, we observe that the median activation of synthetic images is slightly higher than that of natural images of the same class. However, this difference small, given the 0 value lies within standard deviation. We also illustrate the activations of neuron 504 in the ResNet18 output layer for the "coffee mug" class in Figure (b). The results indicate a strong overlap the neural response to both synthetic and natural images. While synthetic images activate the neuron slightly more, this does not constitute an artifactual behavior or affect our framework, which we demonstrate in the following experiment.
MAD sytniat
Natural
12.0
75
0.08
Natural
2.5
3.0
0.04
-25
75
0.00
Synthetic
Deteebaetet
Gong alter
VIT-1/16
Activations
8) MAD between synthetic and natural images
b] ResNettä Neuron 504, "coffee mug"
Image examples for "coffee mug"
An overview of analyses performed to study the similarity between natural and synthetic images. From left to right: (a) an overview of MAD scores between synthetic and natural image activations of the output neuron's ground truth classes for each model studied in this work, (b) activa- tions collected for neuron 504 in ResNet18 for the class "coffee mug", showcasing the differrnnce between the natural and synthetic distributions and (c) examples of natural versus synthetic images. In both analyses, we observe a substantial overlap in the activations of synthetic and natural images, suggesting that the models respond similarly to both types of images.
Figure 3:

---

Table 2: Comparison of true and random explanations on output neurons with known ground truth labels. This table presents the average quality scores (with standard deviations) for true explanations, derived from target class labels, and random explanations, derived from randomly selected synthetic image classes (including the target class), across four models pre-trained on ImageNet. Higher values are better. Our results consistently show high scores for true explanations and low scores for random ones.
AUC (1)
MAD (t)
Model
True
Random
True
Random
ResNet18
0.98
0.47=0.21
6.46:t2.07
-0.11=0.79
DenseNet161
0.99:40.08
0.44#0.22
7.11#1882
-0.19=0073
GoogLeNet
0.99:+0.07
0.48
7.74#2.14
-0.06=0.788
ViT-B/16
0.99++0.05
0.49+0022
13.1243.19
0.09++1.10
4.3 Random Baseline
robust evaluation metric should reliably discern between random explanations resulting in low scores and non-random explanations resulting in high scores. To assess our evaluation framework regarding this requirement, we evaluated the results of the CoSy evaluation by comparing the scores of ground truth explanations with those of randomly selected explanations.
Following the experimental setup in Section 4.2 we selected
of 50 output neurons and compared the CoSY scores of the ground truth explanations, given by the neuron label, with those of randomly selected explanations. The results, presented in Table [ consistently demonstrate high scores for true explanations and low scores for random explanations. This experiment provides further evidence supporting the correctness of the proposed evaluation procedure. An additional experiment that excludes the target class from the control dataset is presented in Appendix A..4 along with an analysis of the robustness of the evaluation measure detailed in Appendix A.6
5 Evaluating Explanation Methods
Within the scope of this section, we produce a comprehensive comparisnn of various methods for the textual explanations of neurons. For this comparison, we employed models trained on different datasets, and we conducted our analysis on the latent layers of the models, where no ground truth is known.
5.1 Benchmarking Explanation Methods
In this section, we evaluated three recent textual explanation methods, namely MILAN, INVERT, and CLIP-Dissect. Our analysis involves six distinct models: four pre-trained on the ImageNet dataset (36) (ResNet18 (37] ResNet50 (37] ViT-B/16
40), DINO ViT-S/8
(41) and two pre-
trained on the Places365 dataset 42 (DenseNet161 38 ResNet50 37 The ImageNet dataset focuses on objects, whereas the Places365 dataset is designed for scene recognition. Consequently, customized our prompts accordingly: Prompt 5 performs best for object recognition, while for scene recognition, we found that Prompt 3 is more effective. Therefore, Prompt was utilized in the Places365 experiment.
For generating cxplanations with the explanation methods, we use a subset of 50,000 images from the training dataset on which the models were trained. For evaluation with CoSy, use the corresponding validation datasets the models were pre-trained on as the control dataset. Additionally, for
we define concept labels by combining the 20,000 most common English words with the corresponding dataset labels. For INVERT we set the compositional length of the explanation as L 1, where N. For more details on compute resources, refer to Appendix A.7 A.7
Results of the evaluation can be found in Table Overall, INVERT achieves the highest AUC scores across all models and datasets, except for DINO ViT-S/8 and ResNet 18 applied to ImageNet, where CL.P-Dissect achieves higher or similar score. Also across other models and datasets, CI.P--Dssset demonstrates consistently good results. Since INVERT optimizes AUC in explanation generation,
7

---

Table 3: Benchmarking of neuron description methods, for neurons in the second to last layer across different models. Explanations are generated for randomly selected set 50 neurons, with average scores for both AUC and MAD reported alongside standard deviations. Higher values indicate better performance; bold numbers represent the highest scores.
Dataset
Model
Layer
Method
AUC (t)
MAD (t)
MILAN
0.61+00223
0.69:11.35
ResNet18
Avgpool
CLIP-Dissect
0.93
3.85+1888
INVERT
0.9310.11
3,23+1.722
MILAN
0.44+-0.23
-0.08
ResNet50
Avgpool
CLIP-Disee
0,95
4.98+2.57
INVERT
0.9610.06
4.6242.26
ImageNet
MILAN
0.53+0.19
0.12+0..76
ViT-B/16
Features
CLIP-Disseas
0.78:10.19
1.29
INVERT
0.8910.17
1.67
MILAN
0.59#0.21
0.37
DIND ViT-S/8
Layer 11
CLIP-Dissect Dissect
0.95
4.59+22.2
INVERT
0.73+0027
2.70
MILAN
0.56100228
0.44+
DenseNet161
Features
CIIP--issect
0.82+0.21
2.33++
INVERT
0.85+0.166
2.21+1.95
Places365
MILAN
0.65+10.28
1.11
ResNet50
Avgpool
CLIP-Dissect Dissect
0.92+00.11
3.73+22.99
INVERT
0.94+0.08
3.54+1.99
may be biased towards AUC in our evaluation, leading to higher scores. MILAN generally performs poorly, with an average AUC below 0.65 across all tasks, indicating performance close to random guessing. MILAN tends to generate highly abstract explanations, such as "white areas", "nothing" or "similar patterns". These abstract concepts are particularly challenging for text-to-image model to generate accurately, likely contributing significantly to the low scores of MILAN. Contrary to the AUC scores, the MAD scores suggest that neural networks applied to both datasets. Nonetheless, in these cases, INVERT concepts also achieve consistently high scores. Otherwise, we find similar outcomes for both metries V, with MILAN achieving poor scores all experimental settings.
outperforms INVERT for convolutional
5.2 Explanation Methods Struggle to Explain Lower Layer Neurons
In addition the general benchmarking, we aimed study the quality of explanations for neurons in different layers of a model. Since it is well known that lower-layer neurons usually encode interesting
see whether explanation methods can capture the concepts these neurons detect. To investigate this, we examined the quality of explanations across layers 1 to 4 and the output layer of an ImageNet pre-trained ResNet18. addition three prior explanation methods, we included the FALCON method in our analysis. For more details on the implementation FALCON see Appendix A.1.44 for additional results of the original FALCON implementation see Appendix A.8 and for qualitative examples and discussion of lower-level concepts see App For each layer, we randomly selected 50 neurons for analysis.
lower-level concepts
In Figure 4 we present the AUC and MAD results for all explanation methods across layers 1 to 4 and the output layer of ResNet118 While less pronounced for the AUC metric, in general, find increasing scores for later layers across all methods and both metrics V which suggest higher concept quality later layers. Furthermore, we find that similar to the benchmarking experiments, MILAN achieves lower scores across metrics. Both MILAN and FALCON consistently show lower performance, with AUC scores of 0.5 indicating random guessing. Nonetheless, we point out that these methods typically output semantically high-level concepts. Potentially, this is related to the inherent difficulty in describing low-level abstractions natural language given their complexity (see Appendix A.10).

---

1.0
CLIP-Disset
FALCON
Random
MAD
INVERT
0.4
MILAN
FALCON
0.2
Random
Layer1
Layer2
Layerd
Output Layer
Layert
Laywr2
Layer3
Layerd
Output Layer
a) AUC across ResNet18 Layers
b) MAD across ResNet18 Layers
Figure 4: A comparison how different explanation methods vary in their quality, as measured by (a) AUC and (b) MAD, across different layers in ResNet18. INVERT and CLIP-Dissect maintain high AUC and MAD scores across all layers, while MILAN and FALCON have lower scores. Overall, performance declines in the lower layers for all methods.
Meuran 4g9 diseettt.. 2re-4)
INVERT
CUP- CUP-Dissect
MILAN
FALCON
Cafrican elephant
'indian @laphant
Living things
large snages
AUC: 0.8990, MMG:8.3878
MUC 0.0990, MADI 8.3225
AUC 0.397, MAG: 00351
AUG: AUG-0.7568, MA3-09221
Neuran 185 (ieenct Lopoe)
INVERT
CUP- Dissect
MILAN
FALCON
gers
''sabbrero"]
gong
roiroles"
red cross'
0,9040 MAC: 5,0052
Alc: 0.9972, 6.5068
AJD: 0.3122, MAD: 6.421
AUC: 0.2811, MAD: -0.599
Heur on 260
Shothes 3. Soperay
INVERT
CUI- Dissect
MILAN
FALCON
"cheeseburgge rger
hand-helld computer
"hands people'
broan hasr 30 uninl
0.8033, MAD
0.85953,
0.5357, MAD 0.848
Neur on 221 Horketts Jg0es)
INVERT
Dis-Dissect
MILAN
FALCON
"rapeseed
"rageseed
Green bells
green OT 898
0.0975, 53381
MAD: 0.027, MAD:
Figure 5: A qualitative example, of neuron explanations acToss four neurons. The first four panels include the textual explanation across INVERT, FALCON. CLIP- Disseett and MILAN alongside three corresponding generated images. The respective AUC and MAD scores are reported below each panel. The last panel shows the activation distributions across 50 generated images for each method and the distribution of the control data.
5.3 What are Good Explanations?
In our approach, we propose that testing visual representations of textual explanations on neurons can provide insights into what constitutes good explanations. Building on this premise, we observe consistently high results from CLIP-Dissect and INVERT. The qualitative examples in Figure 5 demonstrate that their explanations share visually similar concepts (neurons 155 and 459) or identical concepts (ncuron 221) while both achieving high AUC and MAD scores. It is important to note that although INVERT performs slightly better in several tasks, the explanations are constrained to the input data labels. contrast, CLIP-Dissect can generate descriptions from a broader selection of concepts, though its reliance black-box model reduces interpretability compared to INVERT.
There are instances, such as neuron 260 in Figure where all explanations vary significantly. In these cases, we find that the explanation activation distributions of FALCON and MILAN often overlap with or even match the control dataset, providing the user with nearly random explanations. This observation aligns with overall findings: both the AUC and MAD scores consistently reflect the

---

low performance of FALCON and MILAN explanations in the CoSY evaluation. Also, neurons 459 and 155 demonstrate the gap between consistently higher and lower-perforning explanation methods.
Conclusion
In this work, we propose the first automatic evaluation framework for textual explanations of neurons. Unlike existing ad-hoc evaluation methods, we can now quantitatively compare different neuron description methods against each other and test, whether the given explamation describes the neuron accurately, based on its activations. We can evaluate the quality of individual neuron explanations by examining how accurately they align with the generated concept data points, without requiring human involvement.
Our comprehensive sanity checks demonstrate that CoSY guarantees reliable explanation evaluation. In several experiments, we show that neuron description methods are most applicable for the last layers, where high-level concepts are learned. these layers, INVERT and CLIP-Dissed provide high-quality neuron concepts, whereas MILAN and FALCON explanations have lower quality and can present close to random concepts, which might lead to wrong conclusions about the network. Thus, the results highlight the importance of evaluation when using neuron description methods.
Limitations The use of generative models involves distinct set of limitations. For instance, text-to-image models may not include certain concepts in their training data, which can reduce their generative performance. This limitation, however, can often be addressed by analyzing the pre- training datasets and assessing model performance. Moreover, the model's capabilities of generating highly abstract concepts like "white objects" can be limited. However, the challenges with abstract concepts also reflect the descriptive quality of the provided explanations explanations should be inherently understandable to humans. In both cases, exploring more sophisticated, specialized, or constrained models may offer improvement.
Future Work Evaluation of non-local explanation methods is still a largely neglected research area, where CoSy plays important yet preliminary part. In the future, need additional, complementary definitions of explanation quality that extend our precise definition of AUC and MAD, e.g., that involve humans to assess plausibility 44 evaluate explanation quality via the success of a downstream task [45]. Furthermore, we plan to extend the application of our evaluation framework to additional domains including NLP and healthcare. In particular, it would be interesting to analyze the quality of more recent autointerpretable explanation methods given by highly opaque, large language models (LLMs) 46. 9 We believe that applying CoSy to healthcare datasets, where high-quality explanations are crucial, represents an impactful next step.
Acknowledgements
This work was partly funded by the German Ministry for Education and Research (BMBF) through the project Explaining 4.0 (ref. 01IS200551). Additionally, this work was supported by the European Union's 's Horizon Europe research and innovation programme (EU Horizon Europe) as grant TEMA (101093003); the European Union's 's Horizon 2020 research and innovation programme (EU Horizon 2020) as grant i'ToBoS (965221); and the state of Berlin within the innovation support programme ProFIT (IBB) as grant BerDiBa (10174498).
References
[1] Wojciech Samek and Klaus-Rober Müller. Towards explainable artificial intelligence. Explain- able Al: interpreting, explaining and visualizing deep learning, pages 5-22, 2019.
[2] Feiyu Xu, Hans Uszkoreit, Yangzhou Du, Wei Fan, Dongyan Zhao, and Jun Zhu. Explainable Al: A brief survey on history, research areas, approaches and challenges. In Natural Lun- guage Processing and Chinese Computing: 8th CCF International Conference, NLPCC 2019. Dunhuang, China, October 9-14, 2019, Proceedings, Part Il 8, pages 563-574, Springer, 2019.
10

---

[3] Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Rober- Müller, and Wojeiech Samek. On pixel-wise explanations for non non-linear classifier decisions by layer-wise relevance propagation. PloS one, 10(7):e0130140, 2015.
[4] K Simonyan, A Vedaldi, and A Zisserman. Deep inside convolutional networks: visualising image classification models and saliency maps. In Proceedings of the International Conference on Learning Representations (ICLR). ICLR, 2014.
[5] Ramprasaath Selvaraju. Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization, In Proceedings of the IEEE international conference on computer vision, pages 618-626, 2017.
[6] Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg. Smooth- grad: removing noise by adding noise. arXiv preprint
2017.
[7] Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction to circuits. Distill, 5():e00024-001, 2020.
[8] Kevin Ro Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the Wild: Circuit for Indirect Object Identification in GPT-2 Small. In The Eleventh International Conference on Learning Representations, 2022.
[9] Steven Bills, Nick Cammarata, Dan Mossing, Henk Tillman, Leo Gao, Gabriel Goh, Ilya Sutskever, Jan Leike, Jeff Wu, and William Saunders. Language models can ex- plain neurons in language models. https: bettss//opeene http///ppaapbbccc.coo blobbppb...........oooooooooobbb core. windows / weurron-xplaaine//pppee. indexiiddxxx... html| 2023.
[10] Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretability, In The Eleventh International Conference on Learning Representations, 2022.
[11] David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network Dissection: Quantifying Interpretability of Deep Visual Representations. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), July 2017.
[12] Jesse Mu and Jacob Andreas. Compositional Explanations of Neurons. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 17153-17163. Curran Associates, Inc., 2020.
[13] Evan Hernandez, Sarah Schwettmann, David Bau, Tcona Bagashvili, Antonio Torralba, and Jacob Andreas. Natural Language Descriptions of Deep Features. In International Conference on Learning Representations, 2022.
[14] Neha Kalibhat, Shweta Bhardwaj, C. Bayan Bruss, Hamed Firooz, Maziar Sanjabi, and Soheil Feizi. Identifying Interpretable Subspaces in Image Representations. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 15623-15638. PMLR, 23-29 Jul 2023.
[15] Tuomas Oikarinen and Tsui-Wei Weng. CLIP-Dissect: Automatic Description of Neuron Rep- resentations in Deep Vision Networks. International Conference on Learning Representations, 2023.
[16] Kirill Bykov, Laura Kopf, Shinichi Nakajima, Marius Kloft, and Marina MC Höhne. Labeling Neural Representations with Inverse Recognition. In Thirty-seventh Conference Neural Information Processing Systems, 2023.
[17] Dumitru Erhan, Yoshua Bengio, Aaron Courville, and Pascal Vincent. Visualizing higher-layer features of deep network. University of Montreal, 1341(3):1, 2009.
[18] Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Feature visualization. Distill, 2(11):e7, 2017.

---

[19] Anh Nguyen, Alexey Dosovitskiy, Jason Yosinski, Thomas Brox, and Jeff Clune. Synthesizing the preferred imputs for neurons in neural networks via deep generator networks, Advances in neural information processing systems, 29, 2016.
[20] Thomas Fel, Thibaut Boissin, Victor Boutin, Agustin Martin Picard, Paul Novello, Julien Colin, Drew Linsley, Tom ROUSSEAU, Remi Cadene, Lore Goetschalckx, Laurent Gardes, and Thomas Serre. Unlocking Feature Visualization for Deep Network with MAgnitude Constrained Optimization. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
[21] Judy Borowski, Roland Simon Zimmermann, Judith Schepers, Robert Geirhos, Thomas SA Wallis, Matthias Bethge, and Wieland Brendel. Natural images are more informative for interpreting enn activations than state-of-the-ar synthetic feature visualizations. In NeurIPS 2020 Workshop SVRHM, 2020.
[22] Gabriel Goh, Nick Cammarata, Chelsea Voss, Shan Carter, Michael Petrov, Ludwig Schubert, Alec Radford, and Chris Olah. Multimodal neurons in artificial neural networks. Distill, 6(3):e30, 2021.
[23] Naoya Yoshimura, Takuya Mackawa, and Takahiro Hara. Toward understanding acceleration- based activity recognition neural networks with activation maximization. In 2021 International Joint Conference on Neural Networks (ICNN), pages 1-8. IEEE, 2021.
[24] Dennis Grinwald, Kirill Bykov, Shinichi Nakajima, and Marina MC Höhne. Visualizing the Diversity of Representations Learned by Bayesian Neural Networks. Transactions on Machine Learning Research, 2023.
[25] Stephen Casper, Tong Bu, Yuxiao Li, Jiawei Li, Kevin Zhang, Kaivalya Hariharan, and Dylan Hednield-Menell. Red teaming deep neural networks with feature synthesis tools. In Thirty- seventh Conference on Neural Information Processing Systems, 2023.
[26] Kirill Bykov, Mayukh Deb, Dennis Grinwald, Klaus Robert Muller, and Marina MC Höhne. DORA: Exploring Outlier Representations in Deep Neural Networks. Transactions on Machine Learning Research, 2023.
[27] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748 8763. PMLR, 2021.
[28] Chirag Agarwal. Satyapriya Krishna. Eshika Saxena, Martin Pawelczyk, Nari Johnson, Isha Puri, Marinka Zitnik, and Himabindu Lakkaraju. OpenXAI: Towards a Transparent Evaluation of Model Explanations. In Thittyysixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
[29] Anna Hedström, Leander Weber, Daniel Krakowezyk, Dilyara Bareeva, Franz Motzkus, Woj- ciech Samek, Sebastian Lapuschkin, and Marina M.-C. Höhne. Quantus: An Explainable AI Toolkit for Responsible Evaluation of Neural Network Explanations and Beyond. Journal of Machine Learning Research, 24(34):1-11, 2023.
[30] Anna Hedström, Leander Weber, Sebastian Lapuschkin, and Marina Höhne. Sanity Checks Revisited: An Exploration to Repair the Model Parameter Randomisation Test. In XAI in Action: Past, Present, and Future Applications, 2023.
[31] Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi BERTScore: Evaluating Text Generation with BERT. In International Conference on Learning Representa- tions, 2019.
[32] Mert Yuksekgonul, Maggie Wang, and James Zou. Post-hoc Concept Bottleneck Models. In The Eleventh International Conference on Learning Representations, 2022.
[33] Anna Hedström, Philine Bommer, Kristoffer K. Wickstrøm, Wojeiech Samek, Sebastian La- puschkin, and Marina M. C. Höhne. The Meta-axaluation Problem in Explainable AI: Iden- tifying Reliable Estimators with MetaQuantus. Transactions on Machine Learning Research, 2023.
12

---

[34] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. SDXL: Improving Latent Diffusion Models for High- RishlResolution Image Synthesis. In The Twelfth International Conference Learning Representations, 2023.
[35] Pablo Pernias, Dominic Rampas, Mats Leon Richter, Christopher Pal, and Mare Aubreville. Würssthhen An Efficient Architecture for Large-Scale Text-to-lmage Diffusion Models. In The Twelfth International Conference on Learning Representations, 2023.
[36] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115(3):2112255 2015.
[37] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[38] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference computer vision and paltern recognition, pages 4700-4708, 2017.
[39] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1-9, 2015.
[40] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In International Conference Learning Representations, 2020.
[41] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in supervsud vision transformers. In Proceedings of the International Conference on Computer Vision (ICCV), 2021.
[42] Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million Image Database for Scene Recognition. IEEE Transactions Pattern Analysis and Machine Intelligence, 2017.
[43] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, 521(7553):4366 444, 2015.
[44] David Cheng-Han Chiang and Hung-yi Lee. A Closer Look into Using Large Language Models for Automatic Evaluation. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023, pages 9928-8942. Association for Computational Linguistics, 2023.
[45] Satyapriya Krishna, Jiaqi Ma, Dylan Slack, Asma Ghandeharioun, Sameer Singh, and Himabindu Lakkaraju. Post Hoc Explanations of Language Models Can Improve Language Models. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
[46] Nicholas Kroeger, Dan Ley, Satyapriya Krishna, Chirag Agarwal, and Himabindu Lakkaraju. Are Large Language Models Post Hoc Explainers? CoRR, abs/2310.05797 2023.
[47] Christoph Molnar. Interpretable Machine Learning. edition, 2022.
[48] Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In International conference on machine learning, pages 2048-2057. PMLR, 2015.
[49] Thomas H Cormen, Charles Leiserson, Ronald Rivest, and Clifford Stein. Introduction to algorithms. MIT press, 2022.
13

---

[50] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation. 983):1735-1780, 1997.
[51] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarezyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. LAION-400M: Open Dataset CLIP-Filtered 400 Million Image- Text Pairs, In NeurIPS Workshop Datacentric Al. online (online), 14 Dec 2021 14 Dec 2021, page p., Dec 2021.
[52] Tuomas Oikarinen and Tsui- Wei Weng. CLIP-Dissect: Automatic Description of Neuron Representations in Deep Vision Networks. In The Eleventh International Conference on Learning Representations, 2022.
[53] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreason- able effectiveness of deep features as perceptual metric. In CVPR, 2018.
[54] George A Miller. WordNet: lexical database for English. Communications of the ACM, 38(11):39 1995.
14

---

A Appendix
A.1 Neuron Description Methods
Neuron description methods aim to provide insights into human understandable concepts learned by DNNs, enabling deeper understanding of their rocision-making mechanisms. These methods provide textual descriptions for neurons in CV models. This creates a connection between the abstract representation of concept by the neural network and human interpretation. In general, concept can be any abstraction, such as a color, an object, or even an idea [47]. Textual explanations of a neuron can originate from various spaces depending their generation process.
As defined in Section 3.1 we refer to explanation method as an operator that maps a neuron to textual description (f+) S, where S is set of potential textual explanations. The specific set of explanations depends on the implementation of the particular method. We define the following subsets of textual descriptions S:
C represents the space of individual concepts,
L represents the space of logical combinations of concepts,
M represents the space of open-ended natural language concept descriptions.
These textual descriptions serve explanations for fi generated by explanation methods.
Examples for such explanation methods are MILAN [3), FALCON [4), CLIP-Dissect 15 and INVERT [16]. @foowr the general principle of how & works. In Table we outline the origin of textual descriptions and their corresponding set memberships for each E.
Concept 8
Explanation
"pomegranate"
Method &
MILAN, FALCON, INVERT, CLIP-Dissect etc.
Figure 6: Neuron Description Methods. A neuron fi is selected, and neuron description method E applied to generate textual description explaining fü-
Table 4: Set Membership and Origin of Descriptions. Generated textual descriptions have varying set membership and origin across all E. These descriptions can originate from distinct spaces: individual concepts C, logical combinations of concepts L, and open-ended natural language concept descriptions N.. A labeled dataset refers to collection of images paired with individual concept labels. Generated captions are produced by image-to- to-teetttttxtttttt models, such as Show-Attend- [48). image caption dataset consists of image-caption pairs. A concept set consists of textual concept labels.
Method
Set
Origin
NetDissect
C
labeled dataset
CompExp
L.
labeled dataset
MILAN
M
generated caption
FALCON
M
image caption dataset
-I.I-Dissect
C
concept set
INVERT
C
labeled dataset
A.1.1 NetDissect
Network Dissection (NetDissect) 11 is method designed to explain individual neurons of DNNs, particularly convolutional neural networks (CNNs) within the domain CV. This approach systemat- ically analyzes the network's 's learned concepts by aligning individual neurons with given semantic
15

---

concepts. To perform this analysis, annotated datasets with segmentation masks are required, where these masks label each pixel in an image with its corresponding object or attribute identity. The Broadly and Densely Labeled Dataset (Broden) [1] combines set of densely labeled image datasets that represent both low-level concepts, such as colors, and higher-level concepts, such as objects. provides a comprehensive set of ground truth examples for broad range of visual concepts such as objects, scenes, object parts, textures, and materials in a variety of contexts.
A concept 8 E C C S is defined as a visual concept in NetDissect and is provided by the pixel-level annotated Broden dataset. Given a CNN and the Broden dataset as input, NetDissect explains neuron fi by searching for the highest similarity between concept image segmentation masks and neuron activation masks. Concept image segmentation masks are provided by the Broden dataset (z) E {0, 1) 10 where value of signifies the pixel-level presence of s, and 0 denotes its absence. Neuron activation masks are obtained by thresholding the continuous neuron activations of fa into binary masks A(x) E (0, 1 | 'hen the similarity ōiu between image segmentation masks and binary neuron masks can be evaluated using the Intersection over Union score (IoU) for an individual neuron within layer:
(fs, s) = Exex 1 (B,())
(5)
Exex 1 (z) A(r))
The NetDissect method is optimized identify the concept that yields the highest IoU score between binary masks and image segmentation masks. This can be formalized as:
ENeiDissca (ft) arg max ooo s).
(6)
sECCS
NetDissect is constrained to segmentation datasets, relying on pixel-level annotated images with segmentation masks. Moreover, its labeling capabilities are confined to concepts provided within labeled dataset. Furthermore, only individual concepts can be associated with each neuron.
A.1.2 CompExp
To overcome the limitation of explaining neurons with only single concept. the Compositional Explanations of Neurons (CompExp) method was later introduced [12), enabling the labeling of neurons with compositional concepts. The method obtains its explanations by merging individual concepts into logical formulas using composition operators AND, OR, and NOT. The formula length L E N is defined beforehand. The initial stage of explanation generation is similar to NetDissect, set of images is taken as input, and convolutional neuron activations are converted into binary masks. The explanations are constructed through a beam search algorithm (49, beginning with individual concepts and gradually building them into more complex logical formulas. Throughout the beam search stages, the existing formulas the beam are combined with new concepts. These new formulas are measured by the IoU. The maximization of the IoU score is desired to get a high explanation quality.
The approach for obtaining nat is the same in Equation 5 In contrast to NetDissect, the explana- tions can be a combination of concepts, where s E L c S. The procedure of finding the best neuron description can be formalized as:
ECompExp (ff)
arg max &iou (fr, #ELCS
Similar to NetDissect, CompExp requires datasets containing segmentation masks and is primarily applicable to convolutional neurons.
A.1.3 MILAN
MILAN 13 is a method that aims to describe neurons within a DNN through open-ended natural language descriptions. First, a dataset fine-grained human deseriptions of image regions (Milan- notations) is collected. These descriptions can be defined as concepts that are open-ended natural language descriptions, where E A S. Given DNN and input images x X, neuron masks M (x) E Hx XC are collected of highly activated image regions for ft-
16

---

Two distributions are then derived: the probability p (x)) that a human would describe an image region with s, and the probability p(s) that human would use the description for any neuron. The
probability p(s|M (z)) is approximated with the Show-Attend-Te [48] image-to-text model trained on the Milannotations dataset. Additionally, p(s) approximated with a two-layer L.TTM language model (50) trained on the Milannotations dataset.
These distributions are then utilized to find a description that has high pointwise mutual information with M (x). A hyperparameter À E IR adjusts the significance of p(s) during the computation of pointwise mutual information (PMI) between descriptions s and M (x) sets, where the similarity SwrMu is weighted PMI (WPMI). The objective for WPMI is given by:
(s) log p (z)) Alog
8
MILAN aims to optimize high pointwise mutual information between and (x) to find the best description for f;:
EMILAN (f:) arg max SwPMI (fis 8)-
sENCS
The requirement of collecting the curated labeled dataset, Milannotations, limits MILAN's capabilities when applied to tasks beyond this specific dataset. Additionally, another drawback the requirement for model training.
A.1.4 FALCON
The FALCON [4] explainability method has similar approach to MILAN. Initially, it gathers the most highly activating images corresponding to a neuron. GradCam 5 is subsequently applied to identify highlighted features in these images, which are then cropped to focus these regions. These cropped images, along with large captioning dataset LAION-400m [51] with concepts s E C S, input to CLIP (Contrastive similarity between the text embeddings of captions and the input cropped images. The top 5 captions then extracted. Conversely, the least activating images are collected, and concepts are extracted and removed from the top-scoring concepts, ultimately yielding the explanation of the neuron.
Lannuugg-Imaage Pre-training) (27] which computes the image-text
The similarity OCLIPSSore is obtained by calculating the CLIP confidence matrix, which is essentially Cosine Similarity matrix. The aim is to find the maximum image-text similarity score between image embeddings and their closest text embeddings from large captioning dataset:
ÉFALCON (ft) = arg max SCLIPScore (fi, sENCS
(10)
This restriction significantly narrows down the range of models suitable for analysis, setting it apart considerably from other explanation methods.
FALCON Implementation In its original implementation, FALCON restricts the set of "explain- able neurons" based on specific parameters. These include the parameter E N, which determines the set of highly activating images for given feature by requiring a > 10. Additionally, it employs a threshold IR for CLIP Cosine Similarity, with a set value of 7 > 0.8.
These parameter settings significantly restrict the number of explainable neurons, resulting fewer than 50 explainable neurons. This constraint prevents the necessary randomization for comparison with other methods. To address this, we set a 0 and y = However, for the original FALCON implementation, we retain the original settings of a and and calculate y/ across all "explainable neurons". In our experiments on ResNet18, FALCON can only be applied to layers 2 to 4.
A.1.5 CLIP-Dissect
CLIP-Dissect [52 is an explanation method that describes neurons vision DNNs with open-ended concepts, eliminating the need for labeled data or human examples. This method integrates CLIP 27), which efficiently learns deep visual representations from natural language supervision. utilizes both the image encoder and text encoder components of CLIP model to compute the text embedding for
17

---

each concept E C C S from a concept dataset and the image embeddings for the probing images in the dataset, subsequently calculating a concept-activation matrix.
The activations of a target neuron Ji are then computed across all images in the probing dataset X. However, this process is designed for scalar neurons, these activations summarized by a function that calculates the mean of the activation map over spatial dimensions. The concept corresponding to the target neuron determined by identifying the most similar concept based on its activation vector. The most highly activated images are denoted as X, C
SPI is generalization of WPMI where the probability (æ x,) denotes the chance an image z belongs to the example set X, Standard WPMI corresponds to cases where p(z E x,,) is either 0 for all æ E X, while SoftWPMI relaxes this binary setting real values between 0 and The function can be formalized as:
0oo (8) = log Ip (s{xx)) - A log p(s).
(11)
The similarity function dSnnPMI aims identify the highest pointwise mutual information between the most highly activated images X, and a concept 8. This optimization search is expressed as:
ECLII-Dissca (f:) arg max dSANPMII (i,s))
(12)
sECEs
drawback of CLIP-Dissect lies in its interpretability; descriptions are generated by the CLIP model, which itself is challenging to interpret.
A.1.6 INVERT
Labeling Neural Representations with Inverse Recognition (INVERT) [16] shares the capability of constructing complex explanations like CompExp [12 but with the added advantage of not relying on segmentation masks and only needing labeled data. The method obtains its explanations by merging individual concepts into logical formulas using composition operators AND, OR, and NOT. It also exhibits greater versatility handling various neuron types and is computationally less demanding compared to previous methods such as NetDissect [1] and CompExp [22. Additionally, INVERT introduces a transparent metric for assessing the alignment between representations and their associated explanations. The non- parametric Area Under the Receiver Operating Characteristic (AUC) measure evaluates the relationship between representations and concepts based on the representation' ability to distinguish the presence from the absence of concept, with statistical significance. The probing dataset with the concept present is labeled as X,, while the dataset without the concept is labeled as Xo-
The goal of INVERT is to identify the concept E C S that maximizes ÊAUC with the neuron fi. Here, can be a combination of concepts. The optimization process resembles that of CompExp, employing beam search 49 find the optimal compositional concept. The top-performing concepts are iteratively selected until the predefined compositional length L E N is reached.
The similarity measure ÂUUC is defined as:
8 AUUC (f+) ExSEx, Exiex, I[f((z)) < f(
(13)
Xol [X1]
The objective of INVERT maximize the similarity SAUC between a concept s and the neuron fs which can be described as:
EINVERT (f.) =
arg max ÓAUC (fss s). .ECCS
(14)
INVERT constrained by the requirement of labeled dataset and is computationally more expensive compared to CLIP-Dissect.
A.2 Schematic Illustration of CoSy Implementation Details
In the example shown in Figure T we used the default settings of the explanation methods to generate explanations for neuron 80 in the avgpool layer of ResNet18. For CLIP-Dissed used the 20,000
18

---

most common English words as the concept dataset and the ImageNet validation dataset [36] as the probing dataset. We employed Stable Diffusion XL 1.0-base (SDXL) 34 as the -ox-tto-mmag model, using the prompt "realistic photo of close up of [concept]] to generate concept images, with [concept] being replaced by the textual explanation from the methods. We generated 50 images per concept for 50 randomly chosen neurons from the avgpool layer of ResNet18. For evaluation, we also used the ImageNet validation dataset as the control dataset.
A.3 Prompt Bias Analysis
To address prompt bias and dataset dependency, we extended our analysis by comparing results on the object-focused ImageNet with the scene-focused Places365 dataset. Our results show significant difference based on prompt selection and dataset: close-ups work well for object-centric datasets like ImageNet, while more general prompts, such "photo of [concept]", are more suitable for scene-based datasets like Places356 (see Figure 7
concept
painting
concept
photo of
conceep1
realistic photo
concept
realistic photo up
concept
0.40
0.58
0.50
0.15
3.90
0.23
Figure 7: Cosine Similarity between synthetic and natural concept images for 50 concepts in the scene-based dataset Places365 [42, using different input prompts for the text-to-image models Stable Diffusion XI. 1.0-base (SDXL. and Stable Cascade. The third prompt, "photo of [concept]", performs best for generating scene images.
Additionally, evaluated model performance using different similarity metrics across both datasets. Alongside Cosine Similarity (CS), we introduced two more distance measures: Learned Perceptual Image Patch Similarity (LPIPS) [53] which calculates perceptual similarity between two images and aligns well with human perception, using deep embeddings from a VGG model. Additionally, we included "Euclidean Distance" (ED) to capture the absolute differences in pixel values. The results of these evaluations are presented in Table [
A.4 Sanity Check Class Exclusion
In addition to the results presented in Table 2] we conducted the same experiment with the ground truth images excluded from the control dataset. The corresponding results, shown in Table 6 closely align with those obtained when the ground truth class was included. The findings are largely consistent with those obtained when the ground truth class was included.
A.5 Intraclass Image Similarity
In addition to comparing natural and synthetic images as Section 4.I distance to compare the similarity among synthetic images. Intraclass distance refers to the degree of diversity or dissimilarity observed within a set of images of the same class. It quantifies how much the individual images deviate from the average central tendency of the image In this context, intraclass distance is desirable, reflecting how visual concepts can appear in natural images. Higher similarity scores indicate greater divergence from natural occurrences of concepts.
we also analyze the intraclass
Cosine Similarity (CS), "Fuclidean distance" (ED), and Learned Perceptual Image Patch Similarity (LPIPS) commonly used metrics for measuring image similarity because they capture different
19

---

Table 5: Synthetic-to-Natie Image Similarity. This table illustrates the impact of varying parameters within our method pipeline. We evaluate five different prompts as input to two different text-to-image models, Stable Diffusion XL 1.0-base (SDXL) and Stable Cascade (SC). We randomly selected 50 from both ImageNet and Places365, using each class name as input to the prompt, denoted as [concept], generating 50 images per prompt for each model. We compute the average similarity metrics across all classes and report the standard deviation. Higher values of CS (Cosine Similarity) and lower values of ED ("Euclidean Distance") and LPIPS (Learned Perceptual Image Patch Similarity) indicate greater similarity between synthetic and natural images. High similarity desirable; bold values
represent the highest scores for each prompt.
Similarity Measure
Dataset
Prompt
Text-to-imace
CS (1)
ED (L)
LPIPS (L)
1. "a [concept]"
SDXL
0.70+#0.08
8.0+1.099
0.72:10.03
SC
0.69.10.08
8.1911.02
0.74.10.04
2. "a painting [concept]"
SDXL
0.64:10.07
8.7410.81
0.73-10.03
0.64::0.07
8.81+0.76
0.76:10.04
ImageNet
3. "photo of [concept]"
SDXL
0.71.00.08
1.94
0.72.10.03
8C
0.69110.08
8.2511001
0.75.10.04
4. "realistic photo of [concept:]"
SDXI.
0.70++0.08
8.07++1009
0.72+0003
SC
0.69.10.08
8.34.10.9
0.73.10.03
5. "realistic photo of a close of [concept]]"
SDXL.
0.72++0.08
7.97+1.11
0.72++0.03
0.69+0.08
8.31+1001
0.73+00.04
1. "a [concept]"
SDXL
0.62+
9.02+1.04
0.71++0.02
SC
0.63.10.09
8.9611104
0.72.10.33
2. "a painting [concept]"
SDIL
0.57:10.08
9.51+0083
0.72+00.2
SC
0.58.100.08
9.43
0.72.10.03
Places365
3. "photo of [concept]"
SDXL
0.64.1..09
8.85
0.71
SC
0.64+00009
8.89+1.08
0.72+00.03
4. "realistic photo of [concept]"
SDXL
0.62:10.09
9.08
0.71
SC
0.62.10.08
9.2110.98
0.72.10.03
"realistic photo of a close up [concept]"
SDXL
0.63++0.09
9.04+1005
0.71++0.03
SC
0.63++0.09
9.14+1.01
0.72++0.03
aspects of similarity and complement each other. We compute the average CS, ED, and LPIPS for each class and determine the overall class average. Table 7 provides detailed overview of the results quantifying the similarity within synthetic images using CS and ED. When evaluating these results, it is important note that high scores do not necessarily indicate optimal outcomes, they suggest nearly identical images, which may lack intraclass distance. Conversely, very low scores imply significant differences among images, which might not capture the essence of the concept adequately. Ideally, we aim for somewhat similar yet slightly varied images representing the same class. The results show that the Stable Cascade (SC) model consistently achieves higher scores across all prompts compared to the Stable Diffusion XL 1.0-base (SDXL) model. Notably, it obtains the highest score for the two most elaborate prompts (4, 5). This indicates that the SC model tends to offer less intraclass distance in visually representing concepts.
Table 6: Comparison of true and random explanations on output neurons with known ground truth labels. This table presents the average quality scores (with standard deviation) for true explanations, derived from target class labels, and random explanations, derived from randomly selected synthetic image classes (excluding the target class), across four models pre-trained on ImageNet. Higher values better. Our results consistently show high scores for true explanations and low scores for random ones.
AUC (1)
MAD (t)
Model
True
Random
True
Random
ResNet18
0.52+0.24 0.98-10.09
6.59:12.14
0.08-10.91
DenseNett61
0.99+0.08
0.52+0.24
7.33+1.91
0.04+0.84
GoogLeNet
0.99+0007
0.49:0.24
8,01+2.28
-0.01
ViT-B/16
0.99+0005
0.52+0.22
14.7=3.88
0.13
20

---

Table 7: Intraclass Image Similarity. This table illustrates the impact of varying parameters within our CoSY framework. We evaluate five different prompts as input to two different text-to-image models. A random selection of 50 ImageNet classes is made, with each class name used as input to the prompt, denoted as [concept], resulting in 50 images generated per prompt using a text-to- tox--to-image model. We compute the average intraclass similarity across all classes and report the standard deviation. Higher CS and lower ED values indicate greater similarity between the images. intraclass image similarity, neither excessively high nor excessively low scores are desirable.
Similarity Measure
Dataset
Prompt
Text-to-imace
CS (1)
(U)
LPIPS (J)
1. "a [concept]"
SDXL
0.85.10.07
5.6111.40
0.61.10.11
SC
0,93+00.33
3.76+0.92
0.49++0.10
painting [concept]"
0.87++0.05
1.94+11113
0.62++0.10
0,93+00033
3.67+0.87
0.54+0.10
ImageNet
3. "photo of [concept]]
SDXL
0.84+00.66
5.65+1.34
0.62+0.10
80
0.92
4.0311099
0.52+0.10
4. "realistic photo of [concept]"
SDXL
0.87+00066
5.15+1131
0.60=:0.10
SC
0,94.10.03
3.53
0.45.10.09
5. "realistic photo of a close up [concept]"
SDXL
0.8910004
4.8241.16
0.6010.100
0,94+00.03
3.58++0.6
0.50+00.09
"a [concept]"
SDXL
0.84+:0.06
5.75+1.35
0.61
0,92+0.03
4.02+00.55
0.54+00.09
2. "a painting [concept]"
SDXL
0.88
4.89
0.61
SC
0.93+-0.33
3.79+0.84
0.56+00.99
3. "photo of [concept]"
SDXL.
0.85+00066
5.61+1.27
0.61+0.11
Places365
80
0.93:+0003
3.9410.92
0.54
4. "realistic photo of [concept]"
SDXL
0.88+00.55
5.05+1.115
0.59+00.09
SC
0.94100.33
3.7010.88
0.5210008
5. "realistic photo of close up of [concept]"
SDIL
0.87:10.05
5,2511124
0.60110.09
SC
0.94.10.03
3.68
0.54.10.09
A.6 Model Stability
In this experiment, goal is to evaluate the stability of the image generation method employed, aiming to ensure consistent results within our CoSY framework. We achieve this by varying the seed of the image generator and observing the impact image generation. We anticipate consistent image representations across different model initializations, thus ensuring the stability of our framework.
For analysis, utilize 18 and focus its output neurons, as the ground-truth -tuth labels associated with these neurons are known. We randomly select six classes s from the ImageNet validation dataset [36] and examine the corresponding f, class output neurons using CoSy. Here, we exclude the s class from Ag and let A represent the s class. To ensure robustness, we initialize the text-to-image model across random set of 10 seeds. Our analysis involves calculating the first (mean) and second moment (STD) using V AUC. as well as evaluating the intraclass image similarity (refer Section A.5) within each synthetic ground truth class.
The results for our experiment, as shown Table g demonstrate remarkably high AUC scores, indi- cating near-perfect detection of synthetic ground truth classes across all image model initializations. Furthermore, the standard deviation is exceptionally low, suggesting consistent image generation regardless of the chosen seed. The intraclass similarity values indicate certain degree of distance in the generated images, indicating high similarity yet distinctiveness. This intraclass distance is desirable, ensuring that the images not identical but share common characteristics.
These findings underscore the reliability and consistency of our image generation pipeline within our CoSy framework. The high stability of text-to-imge to-image generation across different seeds and the diversity of image similarity contribute to the robustness of our approach.
A.7 Compute Resources
For running the task of image generation for CoSy we use distributed inference across multiple GPUs with PyTorch Distributed, enabling image generation with multiple prompts in parallel. We run our script on three Tesla for 3 prompts in parallel takes approximately 12 minutes.
GPUs an internal cluster. Generating 50 images
21

---

Table 8: Model Stability. A comparison of various model initializations across 10 random seeds using SDXL. The results represent the average scores with standard deviations for each class, calculated across 10 seeds.
Similarity Measure
Concept
AUC (1)
CS (1)
ED (L)
LPIPS (L)
bulbul
0.99961:0.0002
0.91
3,99-10.66
0.65+0.06
china cabinet
0.9999:10.0011
0.891:004
5.00+1.90
0.58+0.04
leatherback turtle
0.9994:10.0001
0.91
4.65:10.87
0.63+0.04
beer bottle
0.9919=000030
0.80
6.79#1141
0.69#0.08
half track
0.9998=+0.000 0.000
0.88+0.04
5.12::0911
0.61#0.04
hard disc
1.0000::0.0001
0.90
4.64#1.11
0.60=0.05
Overall Mean
0.998410.0007
0.881
5.03:10.26
0.6310.05
A.8 Additional Results for Method Comparison across ResNet18 Layers
Given that the original implementation FALCON only provides results for their defined "explain- able neurons" (see Appendix AA.144), we included additional results comparing all methods based on this subset of neurons. Specifically, there are 7 explainable neurons in layer 2, 5 in layer 3, and 15 in layer 4. Figure presents these results.
INVERT
CLIP-Dissect
0.9
FALCON
Random
0.7
INVERT
0.5
CLIP-Dissect
0.4
MILAN
Random
Layer2
Leyer3
Layard
Layer2
Layer3
Leyerd
a) AUC across ResNet18 Layers
b) MAD across ResNet18 Layers
Figure A comparison of explanation methods in ResNet 18 shows that INVERT and CLIP-Dissect maintain high MAD scores across all layers, while MILAN and FALCON have lower scores. Overall, performance declines in the lower layers for methods.
A.9 Qualitative Examples of Lower-Level Concepts
The performance of generative models may present challenges when dealing with abstract concepts. To assess this concern and provide a basis for further discussion, we have included an additional example of lower-layer concepts in Figure 9
A.10 Concept Broadness
While CoSy focuses on measuring the explanation quality, another open question is how broad or abstract are the concepts provided textual explanations. This question of how specific general an individual neuron is described by the explanation, might be relevant to different interpretability applications. For example, research fields where the user aims deploy the same network for multiple tasks with varying image domains. In this case, describing neuron's 's more general concept such "a round object" might be more informative than more (domain-)specific concept such as "a tennis ball" for the network assessment. To provide insight into the broadness of concepts, we assessed whether the similarity between images generated based on the same concept changes from more general to more specific concepts.
22

---

Feoture Vissaliatioon
INVERT
CLEP-SRRAARR
FALCON
Beuros
91ot
NUSAC
Thuson eye
Norrnn 54
Pasture Vissatization
INVERT
2eB elug
col DEUI Fonkey
"stripes
Visuaizzation
Nauren
'b01 peoper
vibrant
"the cOlor bLUe
Deeg unAu pramn grass
MAX
Heuran
site
Thand-helc computer
natura 126
C5E2E MAD 23106
AJC: 0.924 10730
ALC 04995 AND 08R2
AUC-C1755 MAD 0.1921
Figure 9: Examples of various explanations from four different methods applied to the low-layer neurons in layer 2 of the ResNet18 model are provided. From the illustrated Feature Visualizations, we can observe that these neurons detect low-level abstractions. However, the methods studied generally fail to provide low-level explanations, instead attributing more complex explanations.
In our experiment, define the broadness of concept based on the number of hypernyms in the WordNet hierarchy [54]. The more specific a concept the larger the number of hypernyms. We choose two ImageNet classes ("ladybug", "'uu")) and generate 50 images for each concept as well as each hypernym of both concepts (with the most general concept being "entity"). Then, we measure the Cosine Similarity (CS) of all images generated based on the same concept. The box plot of the CS across both concepts and all hypernyms, in Figure TO indicates that we do not find correlation. Thus, we hypothesize that the chosen temperature of the diffusion model has a stronger effect on image similarity than the broadness of the prompt used for image generation.
1.1
ladybug
pug
Cosine Similarity
0.9
0.8
0
2
4
7
8
9
10
11
12
14
15
Number of Hypernyms
Figure 10: The figure demonstrates the independence of the concept broadness measured by the number of hypernyms as defined in WordNet (54) to the inter-image similarity corresponding generated images.
A.11 Prompt and Text-to-Image Model Comparison
Figure showcases additional examples of synthetically generated images using both SDXL and SC across various prompts, highlighting the diversity and accuracy of concept representation.
23

---

1. a coffee nug
SDXL
Stable Cascade
2. painting of cofffee mug
3. photo of coffeee nug
SDXL
Stable Cascade
4. realistic photo of ceaffee
5. realistic photo of close coffee
SDXL
ILSVRC2012, val
coffee
Figure 11: Example images for "coffee mug" generated by the text-to-image models SDXL and sc across various prompts. (1) and (3) present examples of synthetic images with relatively low intraclass similarity and relatively high natural-to-synthete similarity scores. (2) shows examples of synthetic images with the lowest similarity to natural images. (4) illustrates examples of synthetic images
with the highest similarity to other synthetic images within the same class. (5) showcases examples of synthetic images with the highest similarity to natural images. (6) displays examples of natural images from the ImageNet validation dataset (36] belonging to the class "coffee mug".
24

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We state our contributions the end of in Section and and refer to the sections where each contribution is demonstrated.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We mention limitations throughout the text and discuss them further in Para-
graph 6
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiiatinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
25

---

Answer:
[NA]
Justification:
We do not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We describe algorithm, the datasets, models, methods, metrics, and parameters we use.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
26

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We provided our code and use publicly available models, datasets and methods.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.com
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips //ipss.cc/pill wttpcccpppppppppeeeeeeeeeeeeeeee httpa///giiieee/eeeeebiiisssssss /Cc////oddiiisssseeeeeee ssiiiiiiscyyyyyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: We specify and justify the hyperparameters, models, method restrictions, type and subsets of datasets.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Our box plots and line plots include error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi-
dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
27

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: In Appendix A.7 we give detailed information on the type computer resources, inference and running time.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
We mention biases in models and methods we use.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Our method provides more transparency of XAI methods and guides safer use.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
28

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
Safeguards
Question: Does the paper describe safeguards that have been put in place for the responsible release of data or models that have a high risk for misuse (e.g., pre-trained language models, image generators, or scraped datasets)?
Answer: [Yes]
Justification: We mention the bias in and limitations of generative models.
Guidelines: We do not provide new data new pre-trained model.
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All models, datasets, and methods used are cited.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, paperswithcone com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
29

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [Yes]
Question:
Justification: We provided our code and use publicly available models, datasets and methods and reference them.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsoureing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: No human studies were conducted.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: No crowdsourcing or research with human subjects was conducted.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere
NeurIPS Code of Ethics and the
guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
30


Reviews:
[{"review_id": "BFr2X0HXvY", "reviewer": "Reviewer_1Sdz", "summary": "This paper proposes an automatic evaluation for textual explanations of neurons in vision models. The evaluation works by using a text-to-image model to generate images based on the explanation of a neuron. Then, these images are passed through the vision model and that neuron’s activations are recorded. These activations are compared to neuron activations on control images that should have nothing to do with the explanation. A good explanation should yield generated images that produce high neuron activations, as compared to control images. Two metrics are used to measure the difference in neuron activations. The paper validates the evaluation framework by checking that (1) generated images are similar to natural images of class concepts, (2) vision model neuron activations are similar for generated and natural images, and (3) that “ground-truth” explanations (class concept labels) receive high scores while “random” explanations (random class labels) receive low scores. Finally, the authors apply their evaluation to several prominent neuron explanation methods and draw some conclusions about which methods work better and at which layers.", "questions": "- There has been a shift in the literature from treating neurons as the unit of analysis to feature directions, starting with TCAV (https://arxiv.org/abs/1711.11279) and extending through work like CRAFT (https://arxiv.org/pdf/2211.10154). One example of automatic textual explanation of feature directions is given for LLMs in Sec. 3 of https://arxiv.org/pdf/2309.08600 (as well as https://transformer-circuits.pub/2023/monosemantic-features#global-analysis-interp-auto-acts). I don’t know if any existing methods for vision models could be adapted to focus on latent feature directions rather than neurons, but this would be a ripe direction for future methods.\n- Based on the above, the related work section might also point to (1) https://arxiv.org/pdf/2309.08600, (2) https://transformer-circuits.pub/2023/monosemantic-features#global-analysis-interp-auto-acts, and (3) https://arxiv.org/pdf/2211.10154, and possibly point to automatic, model-based evaluations of local explanations like https://arxiv.org/pdf/2312.12747.\n- I’m curious what you would think of fitting a logistic classifier to the neuron activations and computing that classifier’s AUC. How would that differ from your non-parametric AUC?", "limitations": "I think the limitations discussion is sufficient.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Very important: This paper provides an automatic evaluation for textual explanations of neurons that should provide a reasonably reliable signal for explanation quality. Such an evaluation should serve to guide methods research in the area, and has been sorely needed.\n- Very important: The paper tackles an important problem, evaluation of explanation methods in computer vision.\n- Important: The paper is very clear, well-organized, and has good illustrations.\n- Important: Many small design choices in the paper are very reasonable, like the choice of metrics and prompt selection.\n- Important: The paper provides some empirical results with current explanation methods that highlight directions for future research in the area.", "weaknesses": "- Very important: The meta-evaluation in this paper suffers a bit on two fronts. First, it is difficult to say it is a meta-evaluation in the sense that we could plug another evaluation procedure into this framework and mesaure how different evaluation results correlate with some *third,* more ground-truth or utility-driven evaluation of explanation methods. Take for example the meta-evaluation of automatic machine-translation metrics. Metrics like BLEU and ROUGE are accepted on the basis of their correlation (or lack thereof) with human judgment of translation quality. What would make the meta-evaluation in this paper more of a meta-evaluation is if it compared with, for instance, evaluations from prior work described in “Prior Methods for Evaluation”, using some third ground-truth/utility-oriented evaluation as a target measure (see e.g. evaluations in https://arxiv.org/pdf/2211.10154 as possible targets). To clarify why I don’t think the results in Sec. 4.3 count as such a ground-truth evaluation, my second point here is that there is a major distribution shift in what we want to measure between typical uses of textual neuron explanations and the validations conducted in Sec. 4. Specifically, Sec. 4 focuses exclusively on evaluations of “output neurons”, i.e. class logits, whereas textual neuron explainers are applied almost exclusively to intermediate hidden neurons in models. This means that even a proper meta-evaluation comparing CoSY against competing explanation methods would be limited by a narrow focus on output neurons and not intermediate neurons. I wonder, could the paper inclue a meta-evaluation experiment utilizing a model that has been trained with intermediate representation supervision, like a kind of concept-bottleneck model, so there could be some ground-truth label for an intermediate neuron (at least as ground-truth as using output neurons, which are learned with direct supervision)?\n- Important: I have some other doubts about how the evaluation will work for neurons that represent low-level (perhaps early layer neurons) or abstract features (possibly middle/later layer neurons). For instance, how about low-level edge/shape features, or abstract features like “white objects”. It seems guaranteed that the control images will share many low-level features with the generated images. And it seems very possible that the control images include white objects. These cases seem problematic in that they could lead to low AUC or MAD scores even if the explanation is correct. Can the control images be generated in a conditional way that encourages them to lack the concepts in the proposed explanations?\n- Important: Concepts could overlap or be nested in semantic hierarchies, and I am not sure that the evaluation framework could clearly discriminate between good and bad explanations in these settings. For example, we might want to distinguish between “red apples”, “red fruits”, and “red objects” as explanations for a neuron. In order to do this, we need the generated images for each explanation to achieve a sufficient amount of diversity. It would be a problem if when the text-to-image model gets the phrase “red fruits”, it generates mostly red apples (a very salient example of red fruit). So, I would suggest that one meta-evaluation experiment focus on measuring diversity or coverage of the generated images.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate Reviewer 1 (R1) for their time taken and their great attention to detail shown in the review. We are honored by the positive feedback, that our work was found to be highly relevant and impactful. In the following, we address the reviewer's comments in detail.\n\n**[A1] Meta-Evaluation:**\n\n- **Choice of wording:**  We strongly agree with R1 in their assessment of the term Meta-Evaluation. We apologize that this terminology was misleading. Due to the lack of ground-truth, we should not use the term Meta-Evaluation and therefore replaced it with Sanity Checks.\n\n- **Comparison to prior evaluation:** \nGenerally, prior evaluation methods can be divided into two major groups: evaluations based on human studies and evaluations based on assumed ''ground-truth'' labels. Due to the variety of evaluation approaches, a comprehensive comparison in terms of a meta-evaluation is not established. However, we discuss a qualitative comparison to previous methods in response to R4 [A1].\n\n- **Concept-bottleneck for lower-level representations:**\nWhile we agree with the reviewer's concern regarding lower layer representations and share the interest in concept-bottleneck (CB) based meta-evaluation results, we initially refrained from such experiments. We believe the ground-truth nature of CB concepts and last-layer labels to be often similar. Specifically, mostly during training, intermediate concepts are learned in a supervised manner, often using a similar loss function as the output of the model [1]. Therefore we consider that there is little difference between CB neurons and output neurons.\n\n**[A2] Similarity of generated concepts to control data:**\nWe thank R1 for the detailed questions. We acknowledge that the control dataset may indeed contain images visually similar to those in the generated explanations. However, this is not a problem, since the control dataset primarily serves as a collection of random images that represent a diverse array of concepts. While the control dataset might include some images of “white objects”, the majority of images do not include “white objects”. Therefore, for an accurate explanation, the generated images all significantly activate certain neurons, whereas most control images will not. This distinction will be correctly reflected in the AUC and MAD metrics.\n\nRegarding the performance of our methods with low-level neurons, we attribute the low AUC/MAD to the fact that these methods typically output semantically high-level concepts (see Figure 1 in the PDF). This may also be related to the inherent difficulty in describing low-level abstractions in natural language due to their complexity. We will include a broader discussion of this point in the manuscript.\n\n**[A3] Diversity of generated images or concept broadness:**\nWe thank reviewer 1 for bringing up the important consideration of the diversity in generated images per concept and the problem of semantically similar concepts. We will include this topic in more detail in the final manuscript. To this end, we will reference our results in Section 5, and Figures 1 (red objects and pomegranate) and 5 (last line), which show that even different concepts with high perceivable image similarity result in separable evaluation results. Moreover, we will discuss that, semantically similar concepts can warrant similar evaluation scores given similar meanings. Thus, differentiation between justified and failed evaluation for semantically similar concepts would require a ground-truth label and even then can be hard to argue. Regarding the diversity of generated images for a single concept, Appendix A.8 and Figure 9 show that the diversity of the generated images does not depend on the chosen prompt and concept, but rather on the temperature value (value of entropy) in the diffusion model. Thus, we see a high similarity of images generated for the same concept.\n\n**[Q1] Feature Directions:** We appreciate the reviewer’s detailed question and thank them. Our evaluation procedure can be applied to extracted concepts or any scalar function within the model, such as linear or nonlinear combinations of neurons, for example, CRAFT. The choice for evaluating the neurons (i.e. canonical basis) and not concepts lies in the fact that the respective publications of most methods explain neurons specifically, with even more limited approaches such as FALCON [2] that can only be applied to certain ''explainable'' neurons. Thus we considered only the canonical basis, to establish a fair comparison. Nonetheless, we are very much interested in extending our work in this direction and plan to address latent feature directions in future work.\n\n**[Q2] Related Work:**  We thank the reviewer for pointing out these important works, which we will add to the related work section.\n\n**[Q3] Fitting a logistic classifier to the neuron activations:** We appreciate the reviewer’s suggestion and the opportunity to address this point. However, we kindly request further clarification regarding the proposal. Specifically, could the reviewer provide more details on what is meant by applying a logistic classifier prior to computing the AUC?\n\n[1] Koh, Pang Wei, et al. ''Concept bottleneck models.'' ICML, (2020).\n\n[2] Kalibhat, Neha, et al. ''Identifying interpretable subspaces in image representations.'' ICML, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wXUCCF2NCJ", "reviewer": "Reviewer_hkvS", "summary": "The paper presents a novel, architecture-agnostic framework called COSY for quantitatively evaluating textual explanations of neurons in deep neural networks. The framework utilizes generative models to create synthetic images based on textual explanations, allowing for a standardized comparison of neuron responses to these synthetic images versus control images. Through a series of meta-evaluation experiments, the authors demonstrate the reliability and practical value of COSY by benchmarking various concept-based textual explanation methods for computer vision tasks, revealing significant variability in their quality.", "questions": "see the Weaknesses part", "limitations": "Yes, the authors adequately addressed the limitations.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The problem of evaluating textual explanations of neurons is worth studying, which is critical for the advancement of explainable AI and the wider adoption of machine learning models. \n\n2. The COSY framework is designed to be architecture-agnostic, meaning it can be applied to any computer vision model regardless of its underlying architecture. \n\n3. COSY introduces a novel, quantitative evaluation framework for textual explanations of neurons, which addresses the lack of unified, general-purpose evaluation methods. \n\n4. Through the COSY framework, various existing concept-based textual explanation methods can be benchmarked and compared. This is demonstrated in the paper by benchmarking multiple explanation methods, revealing significant differences in their quality and providing insights into their performance.", "weaknesses": "1. The effectiveness of the COSY framework relies heavily on the quality of the generative models used to create synthetic images. If the generative models are not trained on a diverse set of concepts, they may fail to produce accurate synthetic images, thereby affecting the evaluation's reliability.\n\n2. As mentioned by the authors, the INVERT method optimizes for the AUC metric during explanation generation; it may be biased towards achieving higher AUC scores in the COSY evaluation, potentially leading to an overestimation of its performance compared to other methods. \n\n3. As noted in the paper, the COSY framework demonstrates a decline in explanation quality for neurons in the lower layers of the network, which typically encode lower-level features. Could the authors provide more insight into this phenomenon? Is it possible that this is due to limitations in the generative models' capabilities, given that lower layers generally encode more basic concepts which might not be well-represented by the generative model?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer 2 (R2) for the detailed comments and in-depth remarks. We are pleased that our work was found to be a valuable contribution. Below, we address all individual comments.\n\n**[A1] Text-to-image models:** We agree that the effectiveness of CoSy depends up on the performance of the generative model, which indeed is a limitation, which we point out in the limitations section of the original manuscript. We recommend exercising caution when selecting specific text-to-image models. It is important to understand the performance characteristics and limitations of each model during the evaluation process.\n\n**[A2] Bias towards INVERT:** We acknowledge the concern regarding the potential bias of the INVERT method toward achieving higher AUC scores in the CoSy evaluation. To give a fair and balanced assessment of explanation quality, we, therefore, introduced the MAD metric as an additional evaluation metric. \nHere, it is also important to note that CLIP-Dissect, another method included in our study, is biased toward CLIP models (also used in Stable Diffusion). By employing multiple evaluation metrics and methods, we strive to offer a comprehensive and unbiased evaluation of performance. For more details on the correlation of AUC and MAD, please refer to our answer [Q1] to R4.\n\n**[A3] Lower-level Concepts:** We also thank R2 for putting further emphasis on the important issue of lower-level concepts. We share the reviewer's interest and plan to extend existing discussions in Section 5.2. The performance of generative models might be a fundamental problem for abstract concepts, as discussed in our limitations. To assess this concern and provide a basis for further discussion, we have also added an additional example of lower-layer concepts in the same style as Figure 5 (see attached PDF, Figure 1). We agree that the generation model faces an increasingly harder task to generate images corresponding to more abstract concepts, which in part contributes to the larger error bars for lower layers. However, we argue that the limitations of text-to-image are still not the fundamental issue even for lower layers as our example shows that explanation methods often fail to provide abstract concepts and present widely different concepts, such that we can still distinguish individual performances. We also performed further experiments regarding low-level concepts and discussed this topic in our response [A3] to R1.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tRxFWMRatd", "reviewer": "Reviewer_yev1", "summary": "The authors present a new framework designed to evaluate the quality of textual explanations for neurons in deep neural networks (DNN). To this end, the paper introduces CoSY, which aims to provide a quantitative, architecture-agnostic evaluation method for these explanations, addressing the challenge of the lack of unified, general-purpose evaluation metrics in the field. Given a neuron $f$ of a DNN and explanation $s \\in S$, CoSY evaluates the alignment between the explanation and a neuron in three simple steps. It utilizes a generative model to create synthetic data points from textual explanations and compares the neuron's response to these data points with its response to control data points. This comparison provides a quality estimate of the textual explanations. Further, they perform a large set of empirical analyses to quantify the effectiveness of their proposed AUC and MAD metrics.", "questions": "1. Should we expect the AUC and MAD evaluation metrics to correlate? i.e., will an explanation method achieving high AUC also obtain a high MAD score? If yes, why don't we observe this consistently in the empirical analysis?\n\n2. Is quantifying the synthetic image reliability using only 10 random concepts from the 1,000 classes of ImageNet in Figure 2 justified? Do these results hold for a larger number of concepts?\n\n3. Is the aggregation of the activation values done using absolute sum or signed sum of the activations? This is important as negative activations may bias the aggregate value and impact the AUC and MAD metrics.\n\n4. The authors present a very interesting analysis in Fig 4, where they aim to study the quality of explanations for neurons in different layers of a model. However, the error in the values in Fig. 4 is very large, questioning the conclusions from the figure. Moreover, given that the first few layers of a model detect generic low-level features like corners, edges, angles, and colors, why don't we observe near-perfect cosine similarity?", "limitations": "Please refer to the weakness section for more details.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "1. The paper introduces a new quantitative framework that allows comparisons of different explanation methods, facilitating a more standardized approach to evaluating the quality of neuron explanations​​.\n\n2. The proposed framework is independent of specific neural network architectures, making it broadly applicable to various models in computer vision and potentially other domains.\n\n3. The paper includes extensive meta-evaluation experiments to validate the reliability of CoSY.", "weaknesses": "1. The authors indicate that \"the results using prompt 5 as input to SDXL leads to the highest similarity to natural images\" --- isn't there an implicit bias towards images with a single object in these results? Given that, most of the images in datasets like ImageNet consist of single objects, generating synthetic images using a concept and comparing them with natural images should only work for clean natural images, where the concept is explicitly observable.\n\n2. The authors argue that \"different methods devised their evaluation criteria, making it difficult to perform general-purpose, comprehensive cross-comparisons.\" While it is true that the community should work towards common benchmark metrics, it would be great if the authors could clarify what is the problem with the existing quantitative metrics. Further, why can't someone benchmark existing metrics?\n\n3. In Lines 159-161, the authors mention that they employ cosine similarity (CS) to measure the similarity between synthetic images and natural images corresponding to the same concept, but they never discuss the problems or motivations of using CS. For instance, cosine similarity is not a well-calibrated metric (also observed in Fig 2 where we observe very little variance in cosine values across different prompts) for models like CLIP, where the representations are densely packed using a large number of concepts.\n\n4. Can the authors comment how possible techniques to alleviate the dependency of a text-to-image generative model in their evaluation pipeline?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer 3 for the constructive and insightful remarks and appreciate the positive feedback regarding the presentation quality. Below, we address each of their points in detail.\n\n**[A1] Prompt Bias:** We appreciate and strongly agree with the reviewer's insight regarding the dataset dependency of the prompt. We have addressed this in our manuscript by comparing the ImageNet dataset (object-focused) with the Places365 (scene-focused), leading us to use a more general prompt for Places365 models.\nFor further investigations, we expanded our image similarity analysis from 10 to 50 classes, as suggested in [Q2]. We also compared results across both datasets on different similarity measures, as suggested in [W3]. For a detailed analysis, please refer to Figure 2 and Table 1 in our PDF. Our results show a significant difference based on prompt selection and dataset: close-ups work well for object-focused datasets, while more general prompts like ''photo of'' are better for scene datasets.\n\n**[A2] Prior Evaluation Methods:**\n\nBenchmarking existing metrics is challenging due to several reasons:\n\n- **Challenges in Human Evaluations:** Human evaluations mostly fail to give a holistic description of a neuron, since they rely on describing highly activating images only [1]. Furthermore, human-based evaluations suffer from a lack of standardized experimental setups, e.g. varying protocols, tasks, and participant groups, introducing biases, inconsistencies, and are overall not scalable.\n\n- **Limited Scope of Ground-Truth Labels:** Label-based evaluations are restricted to output neurons with predefined classes [2, 3]. Intermediate layers and open-vocabulary explanations are not covered, which limits the applicability of these metrics for a comprehensive evaluation across all neurons in a model. Furthermore, although output neurons are trained to detect specific concepts, their performance may not exactly match their intended function.\n\nWe will add these points to the paper. Thus, clarifying that our approach unifies the evaluation procedure for open-vocabulary explanations and future textual explanation methods for any neuron.\n\n**[A3] Limitations of Cosine Similarity as image similarity measure:**\nWe acknowledge the reviewer's concern regarding the limitations of using cosine similarity (CS). In response, we have incorporated two additional distance measures: Learned Perceptual Image Patch Similarity (LPIPS), which calculates perceptual similarity between two images and aligns well with human perception, using deep embeddings from a VGG model. Additionally, we included Euclidean Distance (ED) to capture the absolute differences in pixel values.\nThese measures provide further context, allowing a broader assessment of visual similarity beyond CS. For detailed results and comparisons, please refer to our PDF, particularly Table 1, which analyzes image similarity with more concepts, as suggested in [Q2]. Importantly, our overall conclusions remain consistent even with these new metrics.\n\n**[A4] Dependency of a text-to-image generative model in evaluation framework:** \nFor our proposed approach, txt2img models are crucial, because most explanation methods rely on open-vocabulary descriptions for which images are not always available.\n\nTheoretically, a way to mitigate reliance on txt2img models is to manually collect data points corresponding to the neuron label (i.e., images that, in the CoSy approach, are generated by the txt2img model), which however in practice would not be scalable. Furthermore, we ensure that the effects of the chosen txt2img model are limited by performing the sanity checks (see Section 4).\n\n**[Q1] Correlation of AUC and MAD:**\nCoSy measures the difference between activation distributions in the control dataset and the images corresponding to the given explanations. The AUC, a nonparametric test, assesses whether a neuron ranks data points corresponding to the explanation systematically higher than random images. In contrast, MAD is a parametric test similar to the Student's t-test, using exact activations and being more susceptible to outliers. While practically high MAD scores often accompany high AUC scores, these metrics can sometimes disagree, thus complementing each other and providing a broader evaluation of the explanation's quality.\n\n**[Q2] Small Concept Size:**\nWe agree and expanded our experiment to include 50 ImageNet concepts, as shown in Figure 2 of the PDF, and applied the same methodology to 50 scene concepts from the Places365 dataset. The results for ImageNet remain consistent, with increased standard deviation but maintaining the general trend.\n\n**[Q3] Aggregation operation:**\nThe aggregation operation is only performed when the output of a specific neuron is multidimensional (non-scalar) and follows Equation 2 in the paper, corresponding to the average pooling operation. Negative activations do not impact the AUC or MAD metrics because these metrics assess distribution differences.\n\n**[Q4] Large errorbars in Figure 4:**\nWe acknowledge the concern about the large errors in Figure 4. These errors stem from the evaluation methods' performance. Contrary to the reviewer's claim, Figure 4 does not measure cosine similarity (CS); it measures AUC and MAD. The significant errors suggest that no single method is definitively superior, but the results still offer valuable insights for future research and improvements.\n\n[1] Kalibhat, Neha, et al. ''Identifying interpretable subspaces in image representations.'' ICML, 2023.\n\n[2] Bykov, Kirill, et al. ''Labeling Neural Representations with Inverse\nRecognition.'' NeurIPS, 2024.\n\n[3] Oikarinen, Tuomas, et al. ''CLIP-Dissect: Automatic Description of Neuron Representations in Deep Vision Networks.'' ICML, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8iR9qmNw3W", "reviewer": "Reviewer_9Kbo", "summary": "The authors propose a framework for evaluating Neuron Annotation methods which label a neuron in a given vision model, via a textual description. This framework is based on generating a set of images using a text2image model, given the predicted textual description of the neuron which acts as the textual input prompt to the generative model. The idea is that if the textual description truly describes the neuron, then the generated image should activate that neuron more than that of a random baseline image from the training dataset the model was trained on (denoted as the control image). The authors use two scoring functions: the AUC and MAD for quantitative evaluation. The authors further evaluate the effectiveness of the evaluation framework, showing that it is a valid way. They then proceed to evaluating existing Neuron Annotation works, and drawing findings and conclusions, such as the fact that some methods are generating random textual descriptions of neurons.", "questions": "In general, i feel this paper is a clear accept to me, and it will have a good impact in the field. I would like that the moderate weaknesses are addressed. While I understand that Rebutal time is limited, and computation time of the proposed framework is also time-consuming, I encourage the authors to address as much as they can from the moderate weaknesses (W1-W2-W3-W4).  \n\nThe minor weaknesses are just comments that can be included in the final manuscript.", "limitations": "Limitations are discussed. No special issues with negative societal impact.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "I would like to congratulate the authors for their work. I really enjoyed reading the paper.\n- This work is the first to propose an unified, intuitive and valid way of evaluating neuron annotation works. Indeed, most of these works provide their own evaluation metrics which makes it hard to compare with previous works, and verify if one method is better than the other\n- The paper is very well written, understandable and straight-forward. The evaluation framework is also simple, something which all people and researchers will appreciate.\n- The papers provides a meta-evaluation section to show the validity of the proposed framework, which I find interesting and important \n- The finding that some Neuron Annotation works provide wrong predictions of textual descriptions (or let's say, statistical-based descriptions) is very interesting and important, which raises a concern in the interpretability field, especially in safety-critical applications.", "weaknesses": "I could not really find major weaknesses that are grounds for rejection. There are some moderate weaknesses:\n\n- [W1] How does the proposed evaluation framework relate to the other evaluation methods used in existing Neuron Annotation works? Do they align well with the new proposed measure? For example, does the proposed evaluation framework align well with the BERTScores for the human annotations of MILAN? In CLIP-Dissect, they labeled the neurons of the final classification layer and measured accuracy with the ground-truth class names. Does the proposed evaluation framework align well here also? \n- [W2] The pool of models is a bit small. More models (especially for for ImageNet) should be analysed. ImageNet ResNet50 is a common model in most Neuron Annotation works and should be reported. What about self-supervised models, such as DINO? How do they compare to classifiers? MILAN also performs analysis on these models. \n- [W3] There is another related Neuron Annotation work [R1], which is only applicable to ViTs. Here, attention heads and tokens are considered as Neurons and labeled. The authors may extend the analysis to this work to strengthen the paper, given that [R1] is a mathematically-valid decomposition of the ViT, and therefore should align best with the proposed evaluation framework compared to other Neuron Annotation works using the ViT.\n- [W4] The Area Under the Receiver Operating Characteristic (AUC) in section 3.2 seems to be a measure which does not involve generating a curve. But usually AUC means that we have a curve in the first place before taking the area under it. What is the curve in the author's case?\n\nOther minor weaknesses: \n- The text2image models are very time-consuming (the authors also mention that it takes 12 min using a parallel GPU setup, for generating 50 images). This limits the applicability to new users that want to evaluate their works. It would be nice to report the speed of different Stable Diffusion models along in Section 4.1, and include other fast-optimized and less-computationally expensive models -- i am not an expert in the text2image field so i do not know which models in the current literature are faster than the ones the authors use. Fortunately, given how fast the generative text2image field is moving, this problem will not be a problem in the very near future. \n- The fact that CLIP-dissect achieves good results in not surprising - because CLIP-Dissect interprets CLIP Neurons rather than CNN/ViT neurons. Their method is based on scaling CNN/ViT neuron activations using CLIP. Therefore, any image-text pair judged low by CLIP will also reduce the CNN/ViT neuron activation value for the corresponding image. In essence, one could think of CLIP-dissect as an amplified value of the CLIP image-text score (amplified by the CNN/ViT neuron activation value for the same image). And since CLIP itself is a very strong model and especially in scoring image-text inputs, it is not surprising that is performs best. \n- Line 156, \"synthetic\" should be replaced by \"natural images\" (as you already used \"generated images\" in line 155). Also, please be consistent with terminology. \n\n[R1] INTERPRETING CLIP’S IMAGE REPRESENTATION VIA TEXT-BASED DECOMPOSITION, ICLR 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate Reviewer 4 (R4) for their time taken and their great attention to detail shown in the review. We are honored by your positive feedback, and thankful that you consider our work as important.\n\n**[A1] Comparison to other evaluation scores:**\nWe thank the reviewer for asking these important questions. We extend the discussion of comparable methods in the final manuscript. While the results presented in the primary publication are comparable, with BERTScores around 0.5 for MILAN across architectures and good performance of CLIP-Dissect, we argue that due to the lack of standardized procedures across publications, we cannot rule out biases toward specific methods. Moreover, human-based evaluation methods often also include further biases. For example, many evaluations supply highly activating images [1], to make the neuron annotation by humans possible. These images may not accurately represent the neuron's overall behavior by only referencing the maximum tail of the distribution.\n\n**[A2] Broader range of explained models:**\nWe acknowledge the reviewer's suggestion to analyze a broader range of models. In response, we have now included ResNet50 pretrained on ImageNet in our analysis as suggested (see Table 2 in PDF). We can observe that INVERT achieves the highest AUC score, while CLIP-Dissect attains the highest MAD score, consistent with our previous benchmarking results on ResNet50 pretrained on Places365. Additionally, we plan to incorporate additional models, such as the suggested DINO, in our benchmarking table for a broader evaluation. \n\n**[A3] Additional textual explanation methods:**\nWe highly value the reviewer's comment and suggestion to include more explanation methods. Due to the limited rebuttal period, we were not able to complete the additional computations in time. Nonetheless, we aim to address this request going forward.\n\n**[A4] Clarification on AUC:**\nThank you for your remark on the AUC. We apologize for not describing it clearly enough in our initial submission. To clarify, we have included a plot in Figure 3 in the PDF that illustrates the AUC scores for a specific neuron (Neuron 358, ResNet18, Layer 4) across four different methods.\nThe AUC score is the Area Under the Receiver Operating Characteristic (ROC) Curve. Figure 3 in PDF illustrates these ROC curves. In this context, the target is our binary label (0 for the control dataset and 1 for concept images), and the ''predictions'' are the neuron activations for each image. The AUC demonstrates how well the activations of the concept images discriminate against the control image activations.\nIn our benchmarking table, we present the mean AUC scores across all selected neurons, providing a more comprehensive evaluation. We hope this explanation and the included visualizations help to clarify our use of AUC.\n\n**[A5] Computational Cost:**\nWe strongly agree with R4 that the computational cost can be a limiting factor. Thus, we tried to incorporate other open-source text-to-image models, which claim to improve upon computation time but found no significant improvements. In case of limited resources, the only option therefore is the reduction of inference steps to improve run-time. We note, however, that a smaller number of inference steps reduces image quality, which might affect the algorithmic performance. \n\n**[A6] CLIP Bias:** \nWe highly appreciate this question. The explanations from the CLIP-Dissect method are selected from a predefined list of concepts, therefore the potential bias towards CLIP-Dissect is minimal.\n\n**[A7] Consistent Terminology:**\nWe thank the reviewer for pointing out the inconsistency and will ensure consistent terminology in the final manuscript.\n\n[1] Kalibhat, Neha, et al. ''Identifying interpretable subspaces in image representations.'' ICML, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
First, we would like to deeply thank all the reviewers for the time they spent reviewing our manuscript. We express our gratitude for the valuable comments and advice. We are honored by their detailed feedback and are strongly encouraged by the generally positive feedback.

We are particularly grateful that all reviewers found our work to be sound and well-presented. We also appreciate reviewers R1, R2, and R4 for recognizing the significance of our contributions and emphasizing the importance of our paper in addressing the critical issue of automatic evaluation for textual explanations of neurons. As R1 highlighted, ''[s]uch an evaluation should serve to guide methods research in the area, and has been sorely needed.'' Additionally, R2 noted that this work is ''critical for the advancement of explainable AI,'' while R4 praised our work as ''the first to propose a unified, intuitive, and valid way of evaluating neuron annotation works.''

Furthermore, we appreciate that reviewers R3 and R2 acknowledged the CoSy framework's architecture-agnostic design, which makes it ''broadly applicable to various models'' (R3). This flexibility enhances the framework's relevance and applicability in the field.

Our empirical results not only present the current state of explanation methods but as R1 noted, it also ''highlight directions for future research,''. Moreover, R4 found our findings ''very interesting and important, which raises a concern in the interpretability field,'' and appreciated that our evaluation framework is ''simple, something which all people and researchers will appreciate.''

Inspired by their helpful comments, we have incorporated the following main changes into the revision:

- We discuss prior evaluation methods in more depth (R1-A1, R3-A2, R4-A1).
- Provide a more detailed explanation of the metrics used, such as AUC, MAD, and image similarity (R3-A3, R3-Q1, R4-A4).
- We have added a normalization term to the MAD metric, increasing the interpretability of the resulting score. Please note that all qualitative results remain unchanged, and there are no changes in the performance rankings of explanation methods. (Equation 1, PDF)
- Use more clear and coherent terminology, including renaming the meta-evaluation section for clarity (R1-A1, R4-A8).
- Include a greater number of concepts in the sanity checks (R3-Q2).
- Expand the benchmark comparison to include additional models (R4-A2).

Below each review, we will address the specific questions and concerns separately. The PDF file containing updated figures and results is attached. 

We thank all the reviewers again for their time, effort, and constructive feedback!

Meta-review: This paper proposes a framework to evaluate the text explanations of neurons, Concept Synthesis (CoSy). Given textual explanations, CoSy uses a generative model conditioned on textual input to create data points representing the textual explanation. The neuron’s response towards these data points are compared with the response to control data points to provide a quality estimate of the given explanation. This is the first general-purpose, quantitative evaluation framework that enables the evaluation of explanations for CV models. With extensive meta-evaluation experiments, this work analyze the choice of generative models, prompts, and explanation methods.

The reviewers agree on the positive ratings of this paper, and the rebuttals addressed most concerns. That said, the reviewers mentioned some aspects of improvement, e.g., about the empirical rigor of the evaluation procedure and the coverage of the concepts. There can be further inquiry into the mechanisms (and further, the developments) of the models. But this paper itself already marks a solid, significant and valuable contribution.
Decision: Accept (poster)