Evaluating Neuron Explanations: A Unified Framework with Sanity Checks
Authors: Tuomas Oikarinen
Ge Yan
Tsui-Wei Weng
Venue: ICML 2025 poster (2025)
Keywords: Interpretability
Mechanistic Interpretability
Interpretability Evaluation
Primary area: social_aspects->accountability_transparency_and_interpretability
TL;DR: We unify a diverse set of neuron explanation evaluations under one mathematical framework, and find many existing methods are significantly flawed and do not pass simple sanity checks.

Abstract: Understanding the function of individual units in a neural network is an important building block for mechanistic interpretability. This is often done by generating a simple text explanation of the behavior of individual neurons or units. For these explanations to be useful, we must understand how reliable and truthful they are. In this work we unify many existing explanation evaluation methods under one mathematical framework. This allows us to compare and contrast existing evaluation metrics, understand the evaluation pipeline with increased clarity and apply existing statistical concepts on the evaluation. In addition, we propose two simple sanity checks on the evaluation metrics and show that many commonly used metrics fail these tests and do not change their score after massive changes to the concept labels. Based on our experimental and theoretical results, we propose guidelines that future evaluations should follow and identify a set of reliable evaluation
metrics.

Evaluating Neuron Explanations: A Unified Framework with Sanity Checks
Tuomas Oikarinen
Ge Yan
Tsui- Wei Weng? ?
Abstract
explanations (Bills al., 2023; Lee et al., 2023) for large language models, as well explaining features of sparse autoencoders (Bricken et al., 2023; Templeton et al., 2024).
Understanding the function of individual units in neural network is an important building block for mechanistic interpretability. This is often done by generating a simple text explanation of the be- havior of individual neurons or units. For these explanations to be useful, we must understand how reliable and truthful they are. In this work we unify many existing explanation evaluation meth- ods under one mathematical framework. This allows us to compare existing evaluation metrics, understand the evaluation pipeline with increased clarity and apply existing statistical methods on the evaluation. In addition, we propose two sim- ple sanity checks the evaluation metrics and show that many commonly used metrics fail these tests and do not change their score after massive changes to the concept labels. Based on our ex- perimental and theoretical results, we propose guidelines that future evaluations should follow and identify a set of reliable evaluation metrics.
Despite the introduction of various approaches for gener- ating neuron explanations, cxisting papers often use very different metrics to evaluate how good their descriptions are, and it is not clear how they compare to each other. In addition, many evaluation metries have problems, as shown by (Huang et al., 2023). To ensure that unit explanations are reliable and trustworthy, it is crucial to establish standard- ized framework for evaluation.
Motivated by the need for a standardized approach, in this work we unify many existing evaluation methods under a sin- gle mathematical framework: NeuronEval, which provides much needed conceptual clarity to the topic of explanation evaluation. This framework allows to clearly compare and contrast current evaluation techniques and provides a more transparent understanding of the evaluation pipeline. Inspired by this, we introduce two sanity tests to validate the metrics, revealing that most commonly used evaluation metries fail least one of these basic tests and should not be relied alone. In summary, in this paper we:
1. Introduction
Formalize the task of evaluating individual unit ex- planations and unify 20 existing methods under the NeuronEval framework, which lets us apply standard statistical methods to evaluating neuron explanations.
Deep neural networks (DNNs) have achieved great success on a wide range of tasks, but they are very difficult to un- derstand and often perceived black-boxes (Rudin al., 2022). To address this challenge, the field of mechanistic interpretability has recently emerged, aiming provide clearer understanding of the internal mechanisms of DNNs.
Propose two sanity checks for evaluation metrics: Miss- ing Labels Test and Extra Labels Test. We then test dif- ferent metrics both theoretically and empirically across hidden layer and final layer neurons, as well as on linear probes on both vision and language models.
Providing natural language explanations for small compo- nents of a neural network is an important part of mechanistic interpretability. Classic work in this area includes Network Dissection (Bau et al., 2017) and other works explaining individual neurons in deep vision models (Mu & Andreas, 2020; Hernandez al., 2022; Oikarinen & Weng, 2023; Bai et al., 2025). Other examples include automated neuron
Out of the 18 metrics tested only the following pass the tests: Covrelation(Pearson)) Cosine Similarity, AUPRC, IoU and Fl-score. In contrast, many com- monly used evaluations fail these sanity tests, such as: using biased rop-and-randon sampling, only eval- uating on highly activating inputs(Recall) measuring AUC or mean activation difference MAD, and cannot be relied on by themselves.
'CSE, UC San Diego, CA, USA ²HDSI, UC San Diego CA. USA. Correspondence to: Tuomas Oikarinen <toikari- nen@ucsd udu Tsui- Wei Weng <lweng @ucs..edu>.
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
Our code and results are publicly available at Lat.://wwortyyLL.LLa///eeeee Eval.

---

Evaluating Neuron Explanations: A Unified Framework with Sanity Checks
NeuronEval (sec 3)
Meta-Evaluation (sec 4, 5)
Neuron activations 8x
Explanation Evaluation
#1: Sanity Checks (i) Missing Label Test,
Target Nouron 1
Explanation score
(II) Extra Label Test (sec 4)
Socul 0.5
Probing data D
Specision
#2: Performance on Neurons with Known concepts (sec 5)
Sy
S,1 = 0.71
"Hippo"
Explanation
Concept activations C,
Which score should we trust?
Figure 1. An overview of contributions. We first unify many existing explanation evaluation methods into single mathematical framework NeuroEEval containing different metrics 8M- Next, we perform mela-evaluation NeuronEval to answer question: which evaluation metries reliably measure how good explanation is?
2. Definitions
neural network? f and text description i, and returns scalar score, where a higher score indicates the description is better i.e. more reliable/faithful. See Figure for an illustration of the evaluation process.
2.1. What is an individual unit in a neural network?
In this paper we are focused on individual unit explanations. By a unit, we mean smaller part of a neural network that can have an independent meaning. The simplest form of such units is single neuron, or a single channel of Convolutional Neural Network (CNN), but unit can be any scalar function of network inputs. Other interesting units that fit in our framework are linear combinations of neurons (i.e. directions in activation space), which are considered to correspond to a specific interpretable concept. These are used in studies such as TCAV (Kim et al., 2018), Concept Bottleneck Models (Koh al., 2020; Yuksekgonul al., 2023; Oikarinen et al., 2023; Srivastava et al., 2024), Linear Probing (Gurnee et al., 2023) or steering vectors (Subramani et al., 2022). Finally, unit could be a feature of a Sparse Autoencoder (Cunningham et al., 2023; Bricken et al., 2023) trained to disentangle a layer's activations into interpretable individual components. In this paper, we focus on units with scalar activations, excluding larger components e.g. attention heads.
For explanations to be reliable, evaluating explanations is very important, but the community has not reached an agree- ment on which to use. Currently different papers often use different & without theoretical justifications. Thus, one focus of our paper is perform meta-evaluation and rigor- ously investigate different evaluation metrics E.
2.3. What are the goals of an explanation?
Neuron explanations are typically generated to improve understanding of the model, which can then help improve, for example, safety and reliability of the models. However, this goal is vague and hard to measure generally. We believe more precise definition of the goals of an explanation is essential for thinking clearly about how to evaluate them.
In this paper, focus Input-based neuron explanations, which make up the majority of existing explanations. We argue that the goal of an explanation is to provide human- interpretable approximation of the following function:
2.2. Problem Deseription
Input + Unit Activation: x
c0:1 (z).
Our focus in this paper is explanation' evaluation. In evalu- ation, text explanation for neuron k is given, and our goal is to evaluate how faithfully this explanation describes the neuron. Below we define evaluation as function E:
A good explanation should be able to describe which inputs cause a high unit activation, and which do not. Note that there are other type of explanations being proposed (e.g. see App. B.2 for discussion.). which is not the focus of this work.
Evaluation (of Explanation f) E(R. D,t) + IR
Here E is function that takes a probing dataset D,
WW can write neuron & in layer 1 of neural network f as a function fe((()) where f represents the i through j ''th layers of the neural network f(z). Here assume the units are neurons for notational simplicity.
Note: Throughout the paper we "explanation", "concept", and "description" interchangeably to refer to text description.
2

---

Evaluating Neuron Explanations: A Unified Framework with Sanity Checks
3. A Unified Evaluation Framework
don't t need to evaluate it all inputs. Here we also binarize the concept vector ea by rounding to 1 for elements 2 0.5 (indicating concept probability 2 0.5) and to 0 otherwise.
We start by studying some popular neuron explanation evalu- ations in Sec. 3.1 and showing they can be formalized under the same notation. We then take a further step in See 3.2 to show that almost all existing work can be unified under this evaluation framework, which we call NeuronEval.
If we consider the goal of an explanation to be predicting neuron activations given whether the concept is present, i.e. simulation, then we can formulate the neuron explanation as a binary classification problem. Specifically, as we show in Appendix A.2 that Equation (3) is actually equivalent to measuring "Recall":
explanationn
3.1. Formalizing Example Evaluations
Example 1: Crowdsourced Highly Activating Inputs Evaluation. Currently, perhaps the most common way to evaluate neuron explanations is to show (crowdsourced) human raters examples of highly activating inputs, and ask the raters whether these inputs can be described by the explanation (also known as 'concept') (Zhou et al., 2015; Bau et al., 2017; Oikarinen & Weng, 2023).
B(ax) B(aa) TP+FN 11B(ax]11
TP
&Recall
(4)
So crowdsourced evaluation of highly activating inputs is equivalent to only measuring "Recall", i.e. we have s = SM with metric M Recall.
To analyze this approach more rigorously, describe it through the following mathematical formulation: Given set of probing data D = } we first define the neuron activation vector dg for neuron as:
Example 2: IoU with Labeled Data. In addition to crowdsourcing, other types of evaluations have been used for validating neuron explanations. For example, (Bau et al., 2017) collected a curated labeled dataset to eval- uate neuron explanations by measuring the Intersection over Union (IoU, also known as Jaccard index) between neuron activations and concept labels on the annotated dataset. This measures [PConcept AND neuron is active) P(concept OR neuron is active).
[ax]: = (2.) ViE [D]. a% E R IP,
(1)
Here, the i-th entry of ag denotes the neuron activation on th input z4. To represent highly activating inputs, we define a binarization function B R= - {0, 1)",, that takes a real vector and turns its element into if it is a "high" activation and O otherwise. In particular, we use the top binarization function for neuron activations, where the activations in top percentile return 1, and the rest return O as this is how highly activating inputs are typically selected. See Appendix A.1 for more detailed description.
We can also describe this evaluation with the same formal- ism as in Example 1, but with two main changes. First, we use a different concept source, where the elements of cs come from the labeled data instead of crowdsourced ratings. Second, we set the metric M as IoU, giving the following evaluation score:
Next, we describe the human ratings for a presence of text concept t in the inputs by defining the concept activation vector c;:
B(ax) B(c,) SooU |B((a)] + [[B())]l B(ax) B(a,)
(5)
Example 3: Automated Simulation Evaluation with Lan- guage Models. More recently, (Bills et al., 2023) evaluate neuron explanations of large language models by measuring the correlation between actual neuron activations and activa- tions predicted by language model given the explanation.
4 E RIPI, [eal: P(t)a)) ViE [DI]
(2)
Here each element of c,, i.e. [et]. represents the fraction of raters that agreed on concept / being present on that input Iy Finally, given ag and ct. we denote s as an explanation axplanatio score to quantify whether neuron le is well explained by the concept I. In this evaluation, the fraction of highly activating inputs that have concept i present is used to represent the quality of explanations. To formalize this we can define in terms of as and e as follows:
While in different domains (Examples I and 2 are from vision models), this evaluation can also be formalized in the same manner as Example 1 and 2. First, the concept source for et is now model predicted neuron activations scaled 0-1, 1, as these are essentially lseudo-labels for the presence of the concept in the given input. Second, for the metric M, we now set it to be the Pearson's 's Correlation coefficient and obtain the evaluation score SCorelation as:
: P (cooccept neuron is active) B(ax) -B(r) || [(x))l
E, u(ax)) ((er) - (((e)) (6)
since the number of highly activated inputs can be computed as || ((xx)) |I and the number of highly activated inputs that
SCorrelation = D
a (ax)
have concept present is (ax)) ) B(c)). Note the value of o on not highly activating inputs doesn't 't affect the score so we
where maz) and (2) is the mean and standard deviation of the vector z.
3

---

Evaluating Neuron Explanations: A Unified Framework with Sanity Checks
NeuronEval: General Framework Evaluate Neuron Explanations
Metric M
Study
Concept Source
Granularity
Domain
Recall
(Zhou et al., 2015)
Crowdsourced
Whole Input
Vision
(Bau et ul., 2017; Oikarinen & Weng, 2023)
~Recall
Crowdsourced
Whole Inpur
Vision
(Oikarinen al., 2023; Bai et al., 2025)
Precision
(Srinivas al., 2025)
Generative
Whole Input
Vision
(Huang et al., 2023)
Generative + Model
Whole Input Per-token
Language Language
Fl-score
(Gurnee et al., 2023)
Labcled data
(Bau ct al., 2017; Mu & Andreas, 2020))
IoU
Labeled data
Per-pixel
Vision
(Laa Rosa et al., 2024)
Accuracy
(Koh et al., 2020)
Labeled data
Whole Input
Vision
~ACC
ZZimerraan et al., 2023)
Crowdsourced
Whole Input
Vision
(Bykov et al., 2023)
Labeled data
Whole Input
Vision
Inverse AUC
(Kopf et al., 2024)
Generative
Whole Input
Vision
Correlation RR)
(Bills et al., 2023)
Model
Per-token
Language
Correlation
(Oikarinen & Weng, 2024)
Model
Whole Input
Vision
Spearman
(Bricken al., 2023; Templeton et al., 2024)
Model
Per-token
Language
Correlation(T&&h
WPMI
(Oikarinen & Weng, 2023)
Model
Whole Input
Vision
MAD
(Kopf et al., 2024)
Generative
Whole Input
Vision
(Shaham al., 2024)
Generative
Whole Inpur Vision
MAD
(Singh et al., 2023)
Generative
Whole Inpur Language
Table 1. Summary of evaluations used in existing work. ~ indicates using metric with small differences from our definition, while T&R indicates use biased top-and ao-anddrandom sampling evaluate metric fewer samples. See Table B.1 an extended versione.
3.2. Unifying Neuron Explanation Evaluation: NeuronEval
P(neuron is active (ooncetconept).. defined as:
SPrecision = B(ax) B(c))
As we have shown in the previous three examples (Eq (4), (5) and (6)), evaluations that look extremely different on the surface,
(8)
actually be written simple functions of the neuron activation vector dę and concept activation vector Cz. This means the explanation score SM can be written function (ak, Cx) of and that is, EDD ()) = sM (M,, Ct), where M is the name the specific metric chosen. For convenience, we call this the NeuronEval Framework, showcased Figure 1. Going fur- show in Table 1, almost all existing
Formalizing metrics under the same framework also high- lights gaps in existing work, leading us to test standard statistical metrics that were not explored by previous work, such as AUPRC (defined in Appendix A.3).
Overall we have aimed include all evaluation metrics used by previous works, as well as standard statistical metrics not previously explored, for a total of 18 different metrics and evaluations from 19 studies within the NeuronEval frame- work, which we compare in our experiments (Section and 5), and the framework can easily include more in the future. In Appendix A.3 define all the other metrics evaluated, such Fi-score, Cosine similarity, AUC and AUPRC, as well additional details on these metrics.
ther, find that, methods in the literature for evaluation & can be formalized under this framework.
Notably, NewronEval is a general framework as shown in Table includes not only the standard statistical metrics (e.g. Recall, FI-score, Correlation), but also other metrics such as Mean Activation Difference (MAD), which com- pares the average neuron activation inputs where the concept is present vs those where concept is missing:
Existing Work Special cases of NeuronEval. We summarize how most existing evaluations fit into our Neu- ronEval framework in Table (see Appendix B.2 for discus- sion on few exceptions). This includes not only evaluations of individual neuron explanations, but also explanations of SAE features (Bricken et al., 2023), CBM neurons (Koh et al., 2020), or linear probes (Gurnee et al., 2023). Writing
B(c)) -a [B(())
(1 (-B(c))) ax
SMAD
|| - B(c)ll
Other standard metrics include Precision (Srinivas et al., 2025), which can be intuitively understood as measuring

---

Evaluating Neuron Explanations: A Unified Framework with Sanity Checks
diverse evaluations as special cases of the same framework allows us more clearly think about each component of evaluation in isolation, and can help develop more effective evaluations by combining the best parts of previous stud- ies. We classify differences in existing evaluations into four main components:
ages of animals 2t ordered as {dog, cat, dog, bear, monkey, flamingo). Suppose we have "pets" neuron that only acti- vates on pets (i.e. dogs cats), then we can compute the neuron activation vector aj as B((x)= = [1. 1, O, O, O]" We can also compute the concept activation vector et for different concepts t based on whether the input image 2; contain the concepts, giving us: Gdog = [1, 1, 0, 0,
Evaluation metrie M: This is the main focus of our paper, to analyze which evaluation metrics are good choices.
Ceat = [0, 1, 0, O, 0, 0] Gpet = [1, 1, 0, 0. 0] Canimal 1, 1, 1] We can then calculate the neuron explana-
tion score 8. of this "pets" neuron with different metrics M in the following table:
(ii) Source of Concept Vector c2: There are many choices for the concept vector ct- These include, but are not limited to: labels from a labeled dataset, using a model to create pseudo-labels, using a human evaluator, or generating new inputs and using the prompts as labels.
Explanation t dog
&Recall
Sprecision
$loU
0.67
1
0.67
cat
0.33
1
0.33
pet (ground truth) animal
1
1
0.5
0.5
(iii) Granularity of activation vectors: The simplest case is full input level activations, i.e. [D] |ox| lc,1 = n. These can also be more specific, for example pixel- level activations as is the case in Network Dissection (Bau et al., 2020), or token level as is often the case for language model explanations.
Table 2. A "pet" neuron: different metries M gives different eval- uation scores 8M on explanations t. For the metrics M=(Recall, Precision, IoU}, is the perfect score, and 0 is the worst score.
When comparing different evaluation metrics, we can see that Recall cannot distinguish between the correct concept (pet) and concept that is too generic (animal), Sreall gives perfect score for both explanations. On the other hand, Precision favors concepts that are too specific (dog, cat), as Sprecision = for 'dog', 'cat' and 'pet'. In contrast, IoU can unambiguously determine the correct concept 'pet', as Sto = 1 only for 'pet'. See Figure B. for illustration.
(iv) Probing dataset D: Which imputs are used for the eval- uation. Typical choices include the training/validation data of the model, special labeled dataset designed for probing, or different datasets for different concepts. Importantly, the dataset used for evaluation should be disjoint from the dataset used for explanation genera- tion to avoid overfitting.
4.2. Sanity Test Definitions
Importantly, regardless of the answer to these questions, or the application domain (e.g. vision or language), each evaluation fits under the same mathematical formalism of NeuronEval.
Inspired by this example, we propose two general tests to measure whether a certain metric is too biased towards spe- cific or generic concepts. These tests are motivated by the following idea: For reliable metric, correct description for neuron should get the highest score, while its subset (too specific) or superset (too general) should score lower.
4. Meta- EeatuaEvauttiion #1: Sanity Checks for Evaluation Metries
(I): Missing Labels Test
Now that we have organized previous evaluations under the NeuronEval framework, we can isolate and perform meta- evaluation on one particular component of the evaluation: the choice of metric M. As seen in Table 1, existing work uses a wide variety of metrics, often without justification. In order to understand which evaluation metrics reliable, we propose two sanity checks: Missing Labels Test and Extra Labels Test which we describe in detail in this section. These tests are inspired by (Adebayo et al., 2018), and passing them is intended to be a necessary (not sufficient) condition for a reliable evaluation metric.
The missing labels test generalized test to measure whether a metric can differentiate between the correct con- cept G and a concept c, that is too specific (a random subset of the correct concept). In particular o by randomly replacing half of the elements of e, with O. That is we remove half of the concept labels for concept t. El] c7 [la] = ||calla/2:
generate
{ [co]s with probability 0.5
[e]
(9)
0 with probability 0.5
We then measure the score difference As, i.e. how much does the evaluation score change when replacing the original concept vector Ct2, with concept that is too specific, cT:
4.1. A Motivating Example
We start by analyzing a simple failure case of the precision and recall metrics. Let the probing dataset contain 6 im-
As(k) = E., [sM(ak: ctt. - sM ct.) (10)
5

---

Evaluating Neuron Explanations: A Unified Framework with Sanity Checks
Ideal Neuron f*
Perfect Explanation
Extra Labels
Missing Labels
1
1
Neuron activations a*
Concept activations C
1
C,
C;
0
0
0
0
0
0
0
SRecal , c,²=
SRecau(a", G,7=
SRecau Fac c,)= 0.5
Extra Labels Test: Sm(a", c,") - Sm(a", c,) =0
Missing Labels Test: Sm(a", c,) SM""", c,) -0.5
X
Figure Overview of our theoretical Missing and Extra Labels Tests. We can see the Recall metric fails the extra labels test cannot differentiate between the perfect explanation and an explanation with extra labels. On the other hand Recall clearly passes the missing labels test as 0.5 << - -0.001
For the sanity test, it is not important how much the modified labels decrease the score. We simply want it decrease score by non-zero amount on all neurons, which leads to define our main metric: Decrease Acc.
where n is the length of vector ct. Similar to the Missing la- bels test, to pass the Extra Labels test, metric should have high Decrease Acc as defined in Eq. (11). If concept t is good description for neuron k. giving additional positive concept labels to random inputs should decrease its simi- larity score. For simplicity, we only apply these tests with ground truth labels as concept source where er is binary.
Decrease Acc
1 K
Σ 1[As(k)< KEK
(11)
Here K is the set of neurons looked at and l is the best/correct concept for neuron k. Note that we normal- ized the scores such that maximum of M is I and minimum value is 0 to allow for equal comparison between metrics. We set 0.001 in our experiments to require small but noticeable change in scores. We study the sensitivity of these tests to the choice of further in Appendix E.4. The ussumption behind this test is that if concept t is a good description for neuron k. a random subset of it should be a worse description.
4.3. Test Setup
We perform two versions of these tests, Experimental on real neurons across diverse settings, and Theoretical on ideal neurons described below.
Experimental: We experimentally evaluated these metries neurons from 8 different settings across vision and lan- guage models, covering final layer neurons, hidden layer neurons, CBM neurons and linear probe outputs. We evalu- ated vision models across 3 datasets: Imagenet, Places365 and CUB200, while language models were evaluated on a subset of Open pendix F for detailed description of the evaluation setting and results on individual datasets. We report the Averaged results in Table 3.
(II): Extra Labels Test
al., 2019). See Ap-
The extra labels test is essentially the opposite of missing labels test, where we test whether a metric can differen- tiate between a correct concept and concept that is too generic i.e. a random superset of the concept. In particu- lar, we create ci by randomly doubling the size of ct, i.c. [[[[eet []=]] 2||c/||1- That is:
Theoretical: In addition, we perform theoretical analysis on the Missing/Extra labels test for hypothetical neurons whose activations perfectly match a concept, which we discuss in detail in Appendix C. See Figure 2 for overview. this setting, the missing labels test measures whether metric can differentiate concept that perfectly matches neuron
[ot] - (
if (c); 1, else with probability otherwise
Iellll n-
(12)
6

---

Evaluating Neuron Explanations: A Unified Framework with Sanity Checks
Meta- Meta-Evaluation&
(I) Missing Labels Test
(II) Extra Labels Test
Pass
Experimental
Theoretical
Experimental
Theoretical
Recall
98.66%
100.00%
0.00%
0.00%
X
Precision
45.73%
0.00%
99.81% %
100.00%
X
Fl-score
93.68%
100.00%
99.82%
100.00%
IoU
93.62%
100.00%
99.81%
100.00%
V
Accuracy
23, 23.79%
60.00%
70.37%
69.68%
X
Balanced Accuracy
98.65%
100.00%
53.67%
60.00%
Inverse Balanced Accuracy
64.18%
60.00%
99.50%
100.00%
X
AUC
94.96%
100.00%
59.18%
60.00%
X
Inverse AUC
52.81%
60.00%
99.99%
100.00%
Correlation
99.41%
100.00%
99.92%
100.00%
Correlation (T&R)
87.83%
100.00%
60.26%
43.64
Spearman Correlation
64.05%
67.20%
49.21% %
44.08%
Spearman Correlation (T&R)
80.04%
100.00%
59.81% 1%
19.68%
X
Cosine
99.45%
100.00%
99.26%
100.00%
WPMI
95.89%
100.00%
58.84%
100.00%
MAD
59.81%
60.00%
99.34%
100.00%
AUPRC
95.61% 1%
100.00%
99.46%
100.00%
Inverse AUPRC
99.15 %
100.00%
95.58%
89.54%
Table Averaged experimental and theoretical results of our missing labels and extra labels test. Experimental results averaged across settings like vision and language domain, while theoretical results are averaged over different neuron activation frequencies. (T&R) indicates the use of top-and-random sampling. We most evaluation metries fail least of the tests.
activations e (Recall=1, Precision=1) from a concept G (with Recall=0.5 and Precision=1). Similarly, the Extra Labels test measures differentiation between the perfect concept and ct (with Recall=1 and Precision=0.5). We simulate neurons with many different activation frequencies, and averaged results are reported in Table 3.
AUC and Correlation(T&R) are biased towards con- cepts that too generic.
Our results are inconclusive for the fol- lowing two metrics: WPMI which passes the tests on certain hyperparameter choices but not on others, and Inverse AUPRC fails the theoretical test only on per- fectly balanced data but passes the empirical tests.
Borderline:
Finally, we analytically studied the expected change in scores for different binary metrics under Missing and Ex- tra labels test in Appendix D. Our analytical results are consistent with the empirical findings in Table 3 and Fig. 3.
Passing:
Fl-score, IoU, Correlation, Cosine similarity and AUPRC perform well and pass the tests.
4.4. Results and Discussion
Importantly, we think any metric failing these tests should not be relied on by itself, and encourage the use of pass- ing metrics. We also perform E. showing that our findings are not sensitive to specific parameter choices of the test.
Results: In Table 3 report the averaged results of this test across these two sets of neurons for all different eval- uation metrics. For simplicity, we say metric passes if its Decrease Ace > 90% for all tests. Failing methods are marked in red color. Overall, our theoretical results closely match our empirical observations. Based on test results, we can group the metrics as follows:
Ablation in Appendix
Concept Imbalance Explains Test Failures: In our theo- retical experiments, we discover that the main reason for metric failing one of these tests is poor handling of concept imbalance. Figure shows an example of this phenomenon. On balanced neurons/concepts that activate on more than 10% of the inputs, all metrics except for Recall pass the Extra Labels Test (large |As)). However, on neurons that activate less frequently, the failing metries such Accu- racy and AUC can no longer distinguish between the perfect concept and an overly generic concept, and the score dif- ference approaches 0. Therefore, failing the Missing/Extra Labels test can in most cases be attributed to the metrics being unable to handle imbalanced data, In this light, our
Accuracy and Spearman Correlation per- form poorly in both tests as their score is largely de- termined by the majority of inputs that neither activate the neuron nor have the concept.
Fail Both:
Fail Missing Labels Test: Precision, Inverse Balanced Accuracy, Inverse AUC and MAD are biased towards too specific concepts.
Fail Extra Labels Test: Recall, Balanced Accuracy,

---

Evaluating Neuron Explanations: A Unified Framework with Sanity Checks
Extra Labels Test As
5. Meta Evaluation #2: Performance on Neurons with Known Concepts
0.0
-0.1
In this section we perform additional comparison be- tween evaluation metrics by empirically comparing how well they perform on neurons where we know their ground truth function, such as neurons in final (classification) layer of a model.
-0.2
-0.3
Recall
-0.4
Precision
-0.3
Correlation
-0.6
Correlation (T6R)
5.1. Method: AUPRC between scores of correct and incorrect explanations
AUC
-0.7
MAD
Accuracy
The idea of this test is to directly measure how good specific evaluation metrics are. To do this, we have a concept set C that includes the correct explanation for each neuron l: E K, well as some incorrect explanations. K is the set of neurons evaluated such as all neurons in layer. We then score all K [[]| (neuron, explanation) pairs and measure whether the metric consistently assigns higher scores to pairs with correct explanations over incorrect ones.
10
10-1
10-?
10-1
Neuran/Cancept Activation Frecuency
Figure 3. The effect of Concept/Neuron Activation frequency the score difference Theoretical Extra Labels test. We can see that all the metrics that fail the Extra Labels test approach 0 score difference on neurons/concepts that rarely activate, í.e. where the concept labels are imbalanced. The passing metrics maintain non-zero score difference regardless of how rare the concept is.
In particular, we measure the AUPRC (area under precision- recall curve) since this is a very imbalanced task with most (neuron, explanation) pairs being incorrect. Math- ematically, we can write the predictions YA, as: YN { (ax: Ct) A k E K,t C}, while the labels Y are Y = {ilt = tgl) A k K,t E C). where ty. is the correct explanation for that neuron. Our meta-AUPRC eval- uation can then be defined as:
results align with previous statistical knowledge, and met- rics such as Accuracy and AUC which are not great for imbalanced datasets fail the tests, while metrics designed for imbalanced data such as Fl-score and AUPRC pass the tests. Since real units in neural network often activate wide range of frequencies, and are sometimes encouraged to activate very rarely (i.e. Sparse Autoencoders), it is essential to use evaluation metrics that can handle imbalanced data.
AUPR-AU AUPRC(Ý (YM, Y)
(13)
Top-and-rande random sampling: Following (Bills et al., 2023; Bricken al., 2023), we also tested a version of Correlation and Spearman Correlation that uses top-and-random sam- pling (T&R), where 50% of the evaluated samples are highly activating inputs only and the remaining 50% sampled uni- formly at random. For our T&R experiments we sampled 25 inputs from the top 0.2% highest activating inputs and 25 random inputs. While Correlation passes both tests with uniform sampling, hop-and-randon sampling makes it fail the extra labels test. This is not surprising, as greatly over- sampling highly activating inputs makes the metric more similar to Recall that only evaluates on highly activating inputs. This also explains why (Huang et al., 2023) found explanations from (Bills et al., 2023) with very high Corre- lation(T&R) scores to have relatively low Fi-scores.
A metric reaches perfect meta-AUPRC of if all correct (neuron, explanation) pairs get higher score than all incor- rect pairs, while random guess performance is 0.
Experimental Setup. Similar to section 4, we ran these test on 10 different setups, consisting of 5 separate models, 4 datasets, vision and language domains and both ground truth labels and pseudo-labels as concept source for G, See Appendix for detailed description of experimental setup and details on individual models.
For all experiments into validation set. For metrics that require hyperparameters such as we use the hyperparameters that performed the best in terms of Meta-AUPRC on the validation split for each setting. We then report performance on the remaining 95% of neurons. In Table 4, we report the average scores and average ranks (i.e. the best metric for each setup gets 1, the worst gets 17) of the metrics across the four setups for both tasks. See Appendix F for detailed breakdown of results.
split random 5% of the neurons
We discover that evaluations based on generating new inputs with generative models are limited to running evaluations that fail the missing labels test and are biased towards more specific concepts. This is because based on generative model prompts is inherently incomplete or missing labels, and as such should not be solely relied on. See Appendix B.1 for further discussion on this situation and potential remedies.
C from Generative models:
We can see that in general the metrics that passed our tests in Section 4 perform better than those that didn't 't, with the top performance going to Correlation, followed by Cosine simi-
8

---

Evaluating Neuron Explanations: A Unified Framework with Sanity Checks
Avg.
Avg.
Since our idea of "concept" is very general, it can include any text-based description. This means single concept could be highly specific (e.g. "flying bird"), tion of simpler concepts (e.g. "water OR river"). Below we show how common explanation failures relate to our tests.
Method
AUPRC
Rank
Recall
0.6722
11.30
composi-
Precision
0.8039
7.90
F1-score/IoU
0.8140
6.70
Accuracy
0.7215
10.80
Balanced Accuracy
0.7979
7.30
Polysemanticity: A popular model of polysemanticity is to model neuron activations as an OR of different concepts. If the explanation only captures one of these concepts, this means the explanation is too specific (FM-B).
Inverse Balanced Acc.
0.8087
7.10
AUC
0.7652
11.00
Inverse AUC
0.7569
10.90
Correlation
0.8765
1.60
Correlation(T&&
0,6606
10.70
Spearman Correlation
0.0853
16,20
Context specific activations: A context specific neu- ron activation such as flying bird means the neuron's 's "true" concept is subset of the non-context specific concept i.c. bird. If the explanation is not context- specific, the explanation is too generic (FM-A).
Spearman Correlation(T&I)
0.3418
15,40
Cosine
0,8666
2.30
WPMI
0.7999
7.30
MAD
0.6952
8,80
AUPRC
0.8406
3,90
Inverse AUPRC
0.6904
9.30
Non activating inputs still contain the concept: This is a common explanation failure case also known as spurious correlations, where all the highly activating inputs share concept, but not all inputs with this concept make the neuron activate. This means that the explanation concept is too generic i.e. superset of the "truc" concept (FM-A).
Table 4. Comparison of different evaluation metrics neurons with known concepts, averaged across 8 settings. Lower rank means better performance. Best performing metric in bold, and second best underlined.
larity then AUPRC and F1-score/loU. Overall we notice continuous metrics perform somewhat better than binary ones. This is likely because binarizing neuron activations loses valuable information, and is hard to find one bina- rization threshold a that works well for all neurons. Despite good performance here, the Cosine metric very sensitive show in E.2,
Finally, in Appendix E.3 we conducted a test using real semantic superclasses of the concept cf instead of our random superset, and found that the results are essentially identical, highlighting the relevance of our tests to real world settings.
to the average activation of neuron which raises some concerns regarding relying on it.
6.2. Conclusions
6. Discussion
In this paper, we have created the NeuronEval framework for unifying different evaluations under the same mathematical formalism, clarifying the definitions of around evaluating unit explanations. We have also proposed new samity tests which lead to our main finding: Neuron Explanation Eval- uations should use metries that pass the Missing and Extra Labels Tests We identified Correlation, Cosine, AUPRC, Fl-score and IoU as solid passing metrics in our study of 18 different metrics. Popular evaluation metrics used in the literature such as Recall and AUC as well as using top-and-randon random sampling do not pass these tests, and should not be relied on by themselves. With evaluation on neurons with ground truth explanation available (e.g. final layer neuron),
6.1. Relevance of our tests to Realistic Failure modes
While our sanity test definition Section is relatively abstract, we think that most real world explanation failures are closely connected to one of two failure modes, and that these failure modes are directly captured by our tests:
FM-A) Explanation too generie: This means the ex- planation concept is superset of the "true" neuron concept, e.g. describing neuron as "animals" when it only activates on dogs. Our Extra Labels Test captures whether metric can detect this failure mode.
FM-B) Explanation too specific: Explanation concept is a subset of real neuron activations, i.e. describing a neuron as "black cat" when it activates on all cats. Our Missing Labels Test captures whether metric can detect this failure mode.
identified that the top-performing met- rics overall are Correlation, Cosine and AUPRC. The final choice of metric may be further affected by considerations such as labeling cost discussed in B. 2 or other failure modes such as Cosine's ' sensitivity to average activation showcased in E.2.
NNte we combined Fi-score and loU into same element in this table they always return the same ordering between pairs leading to identical AUPRC, see Appendix A.4 for proof.

---

Evaluating Neuron Explanations: A Unified Framework with Sanity Checks
Impact Statement
neuron-explan neerron-eryllaanerrppeeer///eeee html, 2023.
Our paper is focused on rigorous evaluation of neural net- work interpretability results, and as such we believe it will have a largely positive societal impact. Being able to un- derstand/see inside models is likely to be very useful in deploying models in reliable and safe manner. In addition, rigorously evaluating their interpretations is very important so we can avoid relying on unfaithful explanations or inter- pretability illusions and can place appropriate trust in these models.
Bolukbasi, T., Pearce, A., Yuan, A., Coenen, A., Reif, E., Viégas, F,, and Wattenberg, M. An interpretabbliy illusion for bert. arXiv preprint arXiv:2104.00143, 2021.
Boyd- Graber, J., Carton, S., Feng, S., Liao, Q. V., Lom- brozo, T., Smith-Renner, A., and Tan, C. Human-centered evaluation of explanations. In Proceedings of the 2022 Conference of the North American Chapter of the Associ- ation for Computational Linguistics: Human Language Technologies: Tutorial Abstracts, pp. 26-32, 2022.
Acknowledgements
The authors are partially supported by National Science Foundation under Grant No. 2107189, 2313105, 2430539, Hellman Fellowship, Intel Rising Star Faculty Award, Nvidia Academic Grant Program as well as the U.S. Na- Learning-enabled enabled
Bricken, T., Templeton, A., Batson, J., Chen, B., Jermyn, A., Conerly, T., Turner, N., Anil, C.,, Denison, C.,, Askell, A., Lasenby, R., Wu, Y., Kravec, S., Schiefer, N., Maxwell, T., Joseph, N., Hatfield-Dodee K., McLean, B., Burke, J. E., Hume, T., Carter, S., Henighan, T., and Olah, C. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2023, https://trassfommer- html.
Z., Tamkin, A., Nguyen,
tional Science Foundation Institute for Learning Optimization at Scale (TILOS;; NSF CCF-2112663 The authors would like thank anonymous reviewers for valu- able feedback to improve the manuscript.
References
Bykov, K., Kopf, L., Nakajima, S., Kloft, M., and Höhne, M. M. Labeling neural representations with inverse recog- nition. In NeurlPS, 2023.
Adebayo, J., Gilmer, J., Muelly, M., Goodfellow, I., Hardt, M.. and Kim, B. Sanity checks for saliency maps. Ad- vances in neural information processing systems, 31, 2018.
Cunningham, H., Ewart, A., Riggs, L., Huben, R., and Sharkey, L. Sparse autoencoders find highly interpretable features in language models, 2023.
Bai, N., Iyer, R. A., Oikarinen, T., Kulkarni, A., and Weng, T.-W. Interpreting neurons in deep vision networks with language models. TMLR, 2025.
Deng, J., Dong, W., Socher, R., Li, L.-J,, Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vi- sion and Pattern Recognition, pp. 248-255, 2009. doi: 10. 10.1 10911VPR..20009..00 52000449..
Bau, D., Zhou, B., Khosla, A., Oliva, A., and Torralba, A. Network dissection: Quantifying interpretability of deep visual representations. In CVPR, 2017.
Bau, D., Zhu, J.-Y., Strobelt, H., Lapedriza, A., Zhou, B., and Torralba, A. Understanding the role of individual units in deep neural network. PNAS, 2020.
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
Bhatt, U., Weller, A., and Moura, J. M. F. Evaluating and aggregating feature-based model explanations. In Bessiere, C. (ed.), Proceedings of the Twenty-innh In- ternational Joint Conference on Artificial Intelligence, IJCAI-20, pp. 3016-3022. International Joint Confer- ences on Artificial Intelligence Organization, 7 2020. doi: 10.249633/jcca.220004417 URL https: http///o.oo////////oo org/ 10. 20..2493/iiiee 2020 417. Main track.
Fel, T., Cadène, R., Chalvidal, M., Cord, M., Vigouroux, D., and Serre, T. Look the variance! efficient black- box explanations with sobol-based sensitivity analysis. Advances in neural information processing systems, 34: 26005-26014, 2021.
Bills, S., Cammarata, N., Mossing, D., Tillman, H., Gao, L.. Goh, G., Sutskever, I. Leike, J., Wu, J., and Saunders, W. Language models can explain neurons in language models. https: / //openai publ ic. blob. core. inndwd..ne..
Fel, T., Picard, A., Bethune, L.,, Boissin, T., Vigouroux, D., Colin, J., Cadène, R., and Serre, T. Craft: Concept recursive activation factorization for explainability. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2012 2711-2721, 2023.
10

---

Evaluating Neuron Explanations: A Unified Framework with Sanity Checks
Fel, T., Boutin, V., Béthune, L.., Cadène, R., Moayeri, M., Andéol, L., Chalvidal, M., and Serre, T. A holistic ap- proach to unifying automatic concept extraction and con- cept importance estimation. Advances in Neural Informa- tion Processing Systems, 36, 2024.
(teav). In International conference on machine learning, pp. 2662226667777 PMLR, 2018.
Kim, J., Maathuis, H., and Sent, D. Human-centered evalu- ation of explainable ai applications: systematic review. Frontiers in Artificial Intelligence, 7:1456486, 2024.
Gandelsman, Y., Efros, A. A., and Steinhardt, J. Interpreting the second-order effects of neurons in clip. arXiv preprint arXiv:2406.04331, 2024.
Koh, P. W., Nguyen, T., Tang, Y. S., Mussmann, S., Pierson, E., Kim, B., and Liang, P. Concept bottleneck models. In International conference on machine learning, pp. 5338- 5348. PMLR, 2020.
Ghorbani, A., Wexler, J., Zou, J. Y., and Kim, B. Towards automatic concept-based explanations. Advances in neu- ral information processing systems, 32, 2019.
Kopf, L., Bommer, P. L., Hedström, A., Lapuschkin, S., Höhne, M., and Bykov, K. Cosy: Evaluating textual explanations of neurons. Advances in Neural Information Processing Systems, 37:34656 37:34656-34665, 2024.
Gokaslan, A., Cohen, V., Pawlick, E., and Tellex, S. Open- webtext corpus. http://Stt / /Skr://Skklll wo..io..0 1o/ Lt//ppenNebTexxtcorppt 2019.
La Rosa, B., Gilpin, I.., and Capobianco, R. Towards fuller understanding of neurons with clustered compositional explanations. Advances in Neural Information Processing Systems, 36, 2024.
Gur-Arieh, Y., Mayan, R., Agassy, C.,, Geiger, A., and Geva, M. Enhancing automated interpretability with output-centric feature descriptions. arXiv preprint arXiv:2501.08319, 2025.
Lee, J., Oikarinen, T., Chatha, A., Chang, K.-C,, Chen, Y., and Weng, T.-W. The importance of prompt tuning for automated neuron explanations. In NewrIPS ATTRIB Workshop, 2023.
Gurnee, W., Nanda, N., Pauly, M., Harvey, K., Troitskii, D., and Bertsimas, D. Finding neurons in haystack: Case studies with sparse probing. Transactions on Machine Learning Research, 2023.
Moakhar, A. S., Iofinova, E., Frantar, E., and Alistarh, D. Spade: sparsity-guider debugging for deep neural net- works. In Forty-first International Conference on Ma- chine Learning. 2024.
Hernandez, E., Schwettmann, S., Bau, D., Bagashvili, T., Torralba, A., and Andreas, J. Natural language descrip- tions of deep visual features. In ICLR, 2022.
Mu, J. and Andreas, J. Compositional explanations of neu- rons. In NeurlPS, 2020.
Hooker, S., Erhan, D., Kindermans, P.-J., and Kim, B. A benchmark for interpretability methods in deep neural networks. In Wallach, H., Larochelle, H., Beygelzimer, A., d'Alché-Buc, F., Fox, E., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https: neurips oc happppprr.aee hts//aper_ffleeeseeeeeeeeeesesss www.es/papee//21///ii.. 1ebb8550000f0cee94acccc56660 pdf.
Oikarinen, T. and Weng, T.-W. Clip-dissect: Automatic description of neuron representations in deep vision net- works. In ICLR, 2023.
Oikarinen, T. and Weng. T.-W. Linear explanations for indi- vidual neurons. In International Conference on Machine Learning, 2024.
Oikarinen, T., Das, S., Nguyen, L. M., and Weng. T.-W. Label-free concept bottleneck models. In International Conference on Learning Representations, 2023.
Huang, J., Geiger, A., D'Oosterlinck, K., Wu, Z., and Potts, C. Rigorously assessing natural language explanations of neurons. In Proceedings of the 6th BlackboxNLP Work- shop: Analyzing and Interpreting Neural Networks for NLP, 317-331, 2023.
Petsiuk, V., Das, A., and Saenko, K. Rise: Randomized sampling for explanation of black-box models. In BMVC, pp. 151. BMVA Press, 2018. URL http:// pdf.
bmve2018 orgg mmw.222....cc/coonntt//eaeeeee
Kazmierczak, R., Azzolin, S., Berthier, E., Hedström, A., Delhomme, P., Bousquet, N., Frehse, G., Mancini, M., Caramiaux, B., Passerini, A., et al. Benchmarking xai explanations with human-aligned evaluations. arXiv preprint arXiv:2411.024777 2024.
Radford, A.. Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. Language models are unsupervised multitask learners. 2019.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I. Learning transferable visual models from natural language supervision, 2021.
Kim, B., Wattenberg, M., Gilmer, J., Cai, C., Wexler, J., Viegas, F., et al. Interpretability beyond feature attribu- tion: Quantitative testing with concept activation vectors

---

Evaluating Neuron Explanations: A Unified Framework with Sanity Checks
Rudin, C., Chen, C., Chen, Z.., Huang, H., Semenova, L., and Zhong, C. Interpretable machine learning: Funda- mental principles and 10 grand challenges. Statistic Sur- veys, 16, 2022.
Zhang, R., Madumal, P., Miller, T., Ehinger, K. A., and Rubinstein, B. I. Invertible concept-basee based explanations for can models with non-negative concept activation vec- tors. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 11682-1 1682-11690, 2021.
Schwettmann, S., Shaham, T., Materzynska, J., Chowdhury, N., Li, S., Andreas, J., Bau, D., and Torralba, A. Find: A function description benchmark for evaluating inter- pretability methods. Advances in Neural Information Processing Systems, 16775688-75715, 2023.
Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., and Torralba, A. Object detectors emerge in deep scene cnns. In ICLR, 2015.
Zhou, B., Lapedriza, A., Khosla, A., Oliva, A., and Tor- ralba, A. Places: A 10 million image database for scene recognition. IEEE Transactions Pattern Analysis and Machine Intelligence, 2017.
Shaham, T. R., Schwettmann, S., Wang, F,, Rajaram, A., Hernandez, E., Andreas, J., and Torralba, A. A mul- timodal automated interpretability agent. In Forty-first International Conference on Machine Learning, 2024.
Zimmermann, R. S., Klein, T., and Brendel, W. Scale alone does not improve mechanistic interpretability in vision models. In NeurlPS, 2023.
Singh, C., Hsu, A. R., Antonello, R., Jain, S., Huth, A. G., Yu, B., and Gao, J. Explaining black box text modules in natural language with language models, 2023.
Srinivas, A. A., Oikarinen, T., Srivastava, D., Weng, W.-H., and Weng, T.-W. Sand: Enhancing open-set neuron de- scriptions through spatial awareness. In 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 2993-3002. IEEE, 2025.
Srivastava, D., Yan, G., and Weng, T.-W. Vlg-cbm: Train- ing concept bottleneck models with vision-lane laiiuae guid- ance. NeurlPS, 2024.
Subramani, N., Suresh, N., and Peters, M. E. Extracting latent steering vectors from pretrained language models. In Findings of the Association for Computational Linguis- tics: ACL 2022, pp. 566-581, 2022.
Templeton, A., Conerly, T., Marcus, J., Lindsey, J., Bricken, T., Chen, B., Pearce, A., Citro, C., Ameisen, E., Jones, A., Cunningham, H., Turner, N. L., McDougall, C., MacDiarmid, M., Freeman, C. D., Sumers, T. R., Rees, E., Batson, J., Jermyn, A., Carter, S., Olah, C., and Henighan, T. Scaling monosemanticity: Ex- tracting interpretable features from claude 3 sonnet. Transformer Circuits Thread, 2024. URL https:
//traansforme--irrciiie circuits...rr........ pub/2024/ salling-nonoeematiiittyy/// html.
Wah, C., Branson, S., Welinder, P., Perona, P., and Belongie, S. The caltech-ucsd birds-200-2011 dataset. Technical Report CNS-TR-201 1-001, California Institute of Tech- nology, 2011.
Yuksekgonul, M., Wang, M., and Zou, J. Post-hoc concept bottleneck models. International Conference on Learning Representations, 2023.
Zhai, X., Mustafa, B., Kolesnikov, A., and Beyer, L. Sig- moid loss for language image pre-training. ICCV, 2023.
12

---

Evaluating Neuron Explanations: A Unified Framework with Sanity Checks
Appendix
Metric Definition & Details
14
A.1 Binarization
14
A.2 Simulation vs Classification Framing.
14
A.3 Additional Metric Definitions
15
A.4 Equivalences
16
Discussion
18
B.1 Generative Model based Evaluation and missing labels test
18
B.2 Limitations
18
B.3 Motivating Example for Sanity Checks
20
B.4 Additional Related Works
21
Theoretical Missing/Extra Labels Test and Concept Imbalance
23
C.1 Results.
23
D Analytical solution for Missing/Extra Labels Test
27
D.1 Special case Theoretical Test
29
E Ablations and Extra checks
30
E.1 Increase/decrease Fraction in Missing/Extra labels test
30
Failure case of cosine: Adding a constant activation
E.3
Random vs Semantic subsets
32
E.4 Epsilon ablation
33
E.5 Number of samples for c+
34
E,6
Combination Metrics
35
Detailed results
37
F.1 Missing and Extra Labels Test:
37
F.2 Meta-AUPRC evaluation:
40
13

---

Evaluating Neuron Explanations: A Unified Framework with Sanity Checks
A. Metric Definition & Details
A.1. Binarization
Many popular metrics in the literature require inputs to be binary such as (Bau et al., 2017; Huang et al., 2023). Since neuron activations and concept activations from some sources are continuous, we need binarize these vectors. We denote this with the binarization function B R {0, 13
Typically for neuron activation ok, following previous work we use B top.. where we take top fraction of activations to be I, and others to be O. We formalize this topa
{
if 2 2 bai otherwise
top (=)];
(A.1)
where b, satisfies Ei-1 in top-5%, and for others. Note a is a hyperparameter needed for all binary similarity functions. We typically select a independently for each metric by finding the value that performs the best on small validation split of neurons. For concept vectors or, we usually binarize simply by rounding, denoted B r, where:
1[,22h.] = a, and z E RR. For example, if a = 0.05, then topa. has I's for inputs with activations
1
if z; 2 0.5
[r(2)]
(0
(A.2)
if 2, 0.5
A.2. Simulation vs Classification Framing
Simulation: In our definitions in section 3 we use the neuron activation the ground truth, and concept presence as our prediction. This corresponds to framing the evaluation as simulation, i.e. trying to predict neuron activation based on concept value. Given this, we can express the Binary Classification Confusion Matrix elements (True Positive (TP), False Positive (FP), False Negative (FN) and True Negative (TN)) in terms of the vectors ag and G as:
TP B(ax) B(c),, FP B(ax) B(cs), FN B(a)) B(a)). TN = B(ax) B(a),
where B() represents element-wise NOT operation on the binary vector (equivalent to 1 - B(-) where 1 is vector of all l's.)
This corresponds to seeing the explanation from a simulation point of view, i.e. our goal is predict how the neuron activates, based the neuron's is explanation and current inputs. This gives binary classification metric definitions that aligned with those of (Huang et al., 2023).
Classification: However this is an arbitrary choice, and we could just well define o the ground truth and the prediction. This corresponds to classification view, where our goal is to use neuron k as classifier for concept t. In terms of metrics, this doesn't i change the definitions for True Positive (TP) or True Negative (TN), but it switches the places of False Positive and False Negative, i.e.
TPds B(ax): B(G,) TP sim
FPats = B(ax): B(G)) = FN sim
FN B(ax) B(G,) FP sim
TNd. = B(az): B(c,) = TN Timi
This change in framing also affects for metrics which are not symmetric in terms of False Positives and False Negatives. For example, Recal(siimulaton) Precision(classfication)) and Precision(simulation) Recallclaassfiiation)
TP sin
TP
Recallssimulatio))
PrecisionCclasiffcatin))
TP TPsim + FN sim
TP
FPale
14

---

Evaluating Neuron Explanations: A Unified Framework with Sanity Checks
The other binary metric that is sensitive this framing is balanced accuracy. The metrie we call Balanced Accuracy in Appendix A.3 corresponds to the simulation version of Balanced Accuracy, for completeness sake we also include Inverse Balanced Accuracy (Appendix A.3), which is the classification version.
Of the non-binary metrics, AUC and AUPRC are sensitive to this framing, and the standard versions AUC and AUPRC we define in Appendix A.3 are AUC(simulaiion) and AUPRC(simulation) while we also include classification versions of these metries labeled as Inverse AUC/AUPRC.
We note that some related works use the simulation framing, while others such as (Zhou et al., 2015) use the classification framing, which leads conflicting definitions of Precision and other metrics.
A.3. Additional Metric Definitions
1. Accuracy: Standard binary accuracy. Percentage of time the neuron activation matches concept activation.
TP+TN SAc: TP+FP4+ FN +TN
B(ax). B(e)) (1 - B(ax)) B(G))
(A.3)
2. Fi-score: Fi-score is the harmonic mean of precision and recall, and can be expressed as:
2. B(as) B(e,)
(A.4)
[B(a))]l + B(a((l
3. Balanced Accuracy: A version of accuracy designed for imbalanced datasets that averages the accuracy on positive and negative inputs.
B(ax). B(e)). + (1 B(ax)) (1- B(c,))
SBA B(x)ll
(A.5)
2 B((x))ll
4. Inverse Balanced Accuracy: Balanced accuracy but consider aş to be the prediction and to be the ground truth.
B(ax) B(e)) (1- B()B(
SIBA
(A.6)
- B(c))l
5. AUC: Area under ROC curve also known AUROC, can be efficiently calculated as:
EiIB(ar) Eml(ar) 1cu < ctp + 0.3 1[cu = ctyl
SAUC
[B(ax)] B(ax)ll
(A.7)
6. Inverse AUC: Area under
curve, where we consider dg to be the predic-
tion and e be the ground truth(classification framing).
SIAUC Eapteo) Egjmars 1aam akgl + 0.5 [[ax: okz]
(A.8)
[B(c) B(c))ll1
7. Spearman Correlation: The Spearman correlation is equivalent to the Pearson Correlation between the ranks of elements.
(R(ax) - ((R(ax))) (R(c)-
8 Spear
(A.9)
(R(())))(((())
Here a function that elementwise returns the ranks of each element.
8. Cosine similarity: The standard cosine similarity between two vectors.
8.os
|ax [ax||a||cH] 2
(A.10)
15

---

Evaluating Neuron Explanations: A Unified Framework with Sanity Checks
9. WPMI: Weighted pointwise-mutual information. A version of this objective is used by (Hernandez al., 2022) and (Oikarinen & Weng, 2023) generate explanations, and by (Oikarinen & Weng, 2023) to evaluate said explanations.
SWPMI
Σ BBaa))
[og(ce) log(u(()))
(A.11)
In the above equations is the length of dr and Ct. Jr calculates the mean of the vector and its standard deviation. À is hyperparameter and R is the rank operator, which transforms each element to its rank, with smallest element becoming and largest ni.
10. AUPRC: Area Under Precision-Recall Curve(AUPRC) a popular metric for measuring classification performance, in particular for imbalanced data. While we are not aware of a closed form solution, can be calculated as:
(a) Calculate precision and recall at each threshold Ts. where threshold contains distinct values of c.. Recall
[[(aa]] ((,2)) precision P. = B(a, 2e: B(ay)
R
(b) Calculate area under precision-recal curve using numerical integral:
8 PUP E(R, ((R--R;11)) R;
AUPRC outputs values in [0, 1] range.
11. Inverse AUPRC: Same as AUPRC, but with differenct framing we flip cz and ag in the calculations.
See Tables and A.2 for additional details on our metrics.
A.4. Equivalences
During our analysis also notice that certain separate metrics are equivalent or very similar each other.
Correlation and Cosine similarity: Calculating correlation between two vectors is equal to normalizing each vector to have mean 0 and then taking their cosine similarity, shown below:
Let å (xx) for any vector
Then
(ax - m(ax)) (a- x(c))
Cosine 61)
||axs12/|@@11
(ax (an)
(ax - p(a))) ))- (ct - p(c+)) (c())
Correlation(as, ct)
(A.12)
This explains why the two perform very similarly in our evaluations.
IoU and F1-score: Below we show that IoU and Fl-score are very closely related. In fact, Fl-score can be written as monotonously increasing function of loU. This means that for any vectors 21, V1, U2, IoU(21, y1) < IoU V2) + [1 1/1 < Fl (12, V2), so for the purposes of comparing similarites they behave identically, and the choice of which one to use doesn't 't matter. As their performance was exactly the same in all tasks, we report them in the same row in Table 4.
Intersection over Union (IoU) also known as Jaccard index is defined as
TP
IoU
(A.13)
TP+FP+FN FN
while F1-score also known as Dice-score is defined as:
2TP 2TP + FP + FN
(A.14)
16

---

Evaluating Neuron Explanations: A Unified Framework with Sanity Checks
Now
2TP F1 2TP + FN
2TP
TP +P+FP+R
(A.15)
TP+ FN
2TP + PPP+FP+ FN
21oU 1TP FP FN 2. loU +1
(A.16)
T
Which is monotonously increasing for 0 S IoU < 1. So using either metric gives the same comparative results.
Metric
Definition
Range
Recall
TPP (TP + FN)
[0, 1]
Precision
TP/ + FP)
[0, 1]
Fi-score
2TP (2TP + + FN)
[0, 1]
IoU
TP/ (TP + +FPFFN)
[0, 1]
Accuracy
(TP+ (TP + FP + TN FN)
[0,1]
Balanced accuracy
(TP TN / (TN FP)] 2
[0,
Inverse balanced accuracy (classification version of balanced accuracy (see App. A.2))
(TP/++++)))) + TN (TN + FN))
[0,
Table A.I. Definition of commonly-usee binary classification metrics. Here, TP, FP, TN, FN refer true positive, false positive, true negative and false negative, respectively.
Metric
Definition
Range
En =1 Eg,= =0 [1(x; 0o > x2} + 0.5 * 1(,, = x2
AUC (swap and to get inverse AUC)
[0, 1]
yy= 1| = 0
E(,, (1
Correlation
[-1, 1]
(i; - 22 (1) 17)
Spearman correlation
Replace y corresponding rank R(x), R(y) in correlation
[-1,1]
Σ rzli
Cosine
[-1,1]
| 2 lellz
WPMI
logp(z: y) Alog(r(x))
(-)0,
E D;
Σ no
Li
MAD
(-00,
1i 1
1/: 0
Table A.2. Definition of other commonly-use metrics. Here. v, R are real vectors, y refer to the mean of
17

---

Evaluating Neuron Explanations: A Unified Framework with Sanity Checks
B. Discussion
B.1. Generative Model based Evaluation and missing labels test
C: from Generative Models: Evaluation methods that use generative models to generate new data (Kopf et al., 2024; Shaham et al., 2024; Singh et al., 2023) effectively define the concept vector [ce]: if z; was generated with prompt t, and 0 otherwise. However, this concept vector is incomplete, as some of "negative" examples that were not generated with that prompt may still contain the concept. In effect, the concept vector e2 is actually naturally missing labels similar to of from our missing labels test. This is because the generated inputs serve as positive labels for et. but the negative inputs are often taken to just be all inputs from a dataset, even though some of actually do have the concept t. So the we get from generative models is actually randomly missing a potentially large portion of the labels. Because of this, most evaluation methods using generative C₁ in Table 1 use methods that fail the missing labels test such as Inverse AUC and MAD. In fact, this is desirable with this concept source, as the missing labels in ot do not affect the explanation score with these metrics. However, these metries alone should not be relied for evaluation.
Potential Solutions and Future Directions: The best way to use generative models in evaluation might be combine them with another evaluation that doesn't ' fail the missing labels test, as is done by (Huang et al., 2023) who measure Precision using generated data, and measure Recall on existing data with model based pseudo-labels and combine these results into F1-score. A useful evaluation in this vein could also be using a crowdsourced evaluation to measure Recall, and combining that with generative model based evaluation of Precision. Another option would be to not use the c derived from prompt directly, but instead use another model to estimate et on all inputs, not just generated ones which would avoid the missing labels problem. Finally an interesting direction would be to look into combinations of other metrics, for example could AUC and Inverse AUC be combined similarly to how Recall and Precision combine into Fi-score. We explore these combination metrics further in Appendix E.6.
B.2. Limitations
Framework Limitations:
Not every evaluation of neuron descriptions can fit into our framework. Below we split these into few separate cases and discuss whether each case represents limitation of the framework or not:
Evaluating Multiple Inputs at once: Our evaluation framework assumes that the presence of a concept is estimated separately for each input. Many human study based evaluations (e.g. (Bau et al., 2017), (Oikarinen & Weng, 2023)) instead evaluate group of inputs at once, asking questions like "How well does concept match this group of images?". However we believe this is simply less preciseʃless objective way of asking whether the concept matches each input separately and does not in general represent a significant limitation for the framework.
Comparing similarity to "correct" explanation: Another approach to evaluate neuron descriptions is to compare how close they are to "correct" description, typically in a text-embedding space. For example, this is the main evaluation used by MILAN (Hermandez et al., 2022), where they generate "correct" explanation by asking Mechanical Turk workers to describe neurons based on their most highly activating inputs. We do not think this a very reliable way to evaluate explanations, because relies on the assumption that there exists a single "correct" text-based explanation for each neuron (and that we have some way of finding it), and we do not think this is the case for many real neurons because of issues like polysemanticity and non-verbal concepts like specific graphical patterns. For these hard-to-interpre neurons it is better to just measure how well our explanation matches the neuron like the metrics in our framework do.
Non-text based explanations: While we focus on text based concepts in our paper, the framework works on non-textual concepts just as well, as long as we have some way of generating a concept vector e, for that concept. For example, the evaluation of (Zimmermann al., 2023) uses a group of highly activating inputs as the concept, and then asks workers whether new input is similar to those inputs not. Despite this difference, it can be described neatly within our framework.
Output Based Neuron Explanations:
For clarity, it is useful to divide the neural network f(x into two parts, fou and gl+1: where 0 corresponds to the input layer, fid represents the through j layers of the neural network f(x), is the layer of the neuron we are interested in and I. is the total number of layers in the network. Then:
18

---

Evaluating Neuron Explanations: A Unified Framework with Sanity Checks
f(x) f
(B.1)
where 12 is the activation of neuron of interest in layer 1, while (2((()) is the activations of all the other neurons in layer L.
Alternatiely to Input-based neuron explanations we define in section 2.3, a neuron explanation could instead be Output-based, that is an interpretable approximation of the following function:
-niit Activation + Output:
fl 1: fxx
Where = is a real number (e.g. intervened value). This function describes how changes in the unit activation change the final network output, as has been the focus of a few recent papers on neuron explanations (Gandelsman et al., 2024; Gur-Arieh et al., 2025).
As we can see from the above notations, output and input-based explanations are different problems, and may require different methods to solve and evaluate. our paper, we focus on evaluating explanations of Input-based, i.e. the function x + fir (z). as this is more common existing evaluations and can be applied more generally, for example to explain linear combinations of neurons that don't have a direct effect on the output such as TCAV (Kim et al., 2018) vectors. While evaluating Output-based based explanations is also important, this may require different methods and is outside the scope of our current work.
This may be the most significant limitations of our framework, as we believe measuring Output based explanations is equally important, and an ideal neuron explanation should describe both functions(potennially with different explanations). However, improving the evaluation of Input-based explanations is already a significant contribution towards better evaluation practices. In addition, while currently our framework is meant for Input based evaluation only, we believe many of the ideas and metrics we discussed could be useful in evaluating output based explanations, For example, in generative models we could use the same metries measure similarity between unit activation and the presence of specific concept in the output. However in output- evaluations there additional considerations such as measuring difference in outputs when changing the unit activation that may be more important. We believe extending this framework or creating a similar one for output-based evaluations is an important direction for future work.
Experimental Result Limitations:
Overall we are quite confident in the generality of our results on the missing/extra labels test (Table 3) as they are consistent across final/hidden layer neurons and different datasets, and we showed theoretically that they are caused by poor metric performance on imbalanced data. Importantly, this theoretical result is independent of the data domain, type of concepts or the type of unit in question.
However, it is good to note that passing these sanity checks does not guarantee that the metric is good metric, but failing them does indicate a metric should not be relied on. This is similar to sanity checks proposed by (Adebayo et al., 2018), which have been quite influential in the field of saliency maps/imput importance estimation.
On the other hand, our comparison results in Table are mostly focused on final layer neurons or other units where we have ground truth available such concept neurons inside a CBM. While they consistently prefer certain metrics, it is possible that these final layer neurons are systematically different from other units are interested in such as hidden layer neurons, and these results should not be relied on too strongly.
Labeling Cost:
One of the main reasons existing crowd-sourced evaluations only measure Recall is it's lower evaluation cost, as we only need to only evaluate e, on highly activating inputs, i.e. where B(ax) = 1, because the values of e on other inputs do not affect the score. In contrast, evaluating most other metrics requires knowledge of the full c. Similar cost issues are also faced when using an expensive LLM based simulation pipeline predict e done by (Bills et al., 2023), causing evaluation metrics to often be evaluated on a small subset instead.
Crowd-sourced evaluation of the entire will likely give noisy results as most inputs do not contain the concept. To avoid this, another approach is to oversample highly activating inputs similarly to Top-and- aap-ddo-rando sampling, though more effective sampling strategies likely exist. This can be done without failing the extra labels test if the proper sampling correction applied to correct for the bias introduced by the non-uniform sampling. As our paper is focused on the correctness of
19

---

Evaluating Neuron Explanations: A Unified Framework with Sanity Checks
different metrics, the ease of measuring them, and the effective sampling/user study design for non-recall metrics is rather large and complex topic that does not fit within the scope of the current paper, but it is promising direction for future work.
An alternative way to reduce labeling costs is to use combination of metrics, for example evaluating FI Fl-score by combining a crowd-sourced evaluation of Recall with a generative model based evaluation of Precision.
B.3. Motivating Example for Sanity Checks
Here, we illustrate the motivating example discussed in Section 4.1 in the following Figure. Figure B.1 displays example neuron that only activates on images containing pets to highlight how similarity scores are calculated. We can see that only measuring Recall gives a perfect score to explanations that are too generic (animal), while only measuring precision favors explanations that are too specific(dog. cat).
Example neuron activating pets only
Probing data Dtest
B(a²)
0
Coog
Gcat
1
0
0
0
0
Opet
0
0
0
Canimal
1
Results of different evaluation metrics:
Concept
SRecall
Preciiion
StoU
Correct concept gets perfect
score with all metrics
Dog
0.67
2/2
2/3 0.67
Recall incorrectly gives perfect score to superclass Anima!
Cat
1/3 0.33
1/1 #1
1/3 0.33
Pet
3/3=1
3/3
3/3
Precision incorrectly gives perfect
score to subclasses Dog and Cat
Animal
3/3
3/6
3/6
Figure B.I. A hypothetical neuron that only activates on pets (dogs or cats). When comparing different evaluation metries, we can see recall cannot distinguish between the correct concept (Pet) and concept that is generic (Animal), while precision favors concepts that are too specific (Dog, Cat). IoU can unambiguously determine the correct concept.
20

---

Evaluating Neuron Explanations: A Unified Framework with Sanity Checks
B.4. Additional Related Works
Evaluation of individual neuron explanations. Table B.1 shows an expanded version of our comparison table (Table I in the main text section 4) of existing evaluation methods. It can be seen in Table B.1 that prior work (Bau et al., 2017; Oikarinen & Weng, 2023; Oikarinen et al., 2023; Bai et al., 2025; Huang al., 2023; Gurnec et al., 2023; Mu & Andreas, 2020; La Rosa et al., 2024; Koh et al., 2020; Zimmermann et al., 2023; Bykov et al., 2023; Kopf et al., 2024; Bills et al.,
2023; Oikarinen & Weng, 2024; Bricken et al.. 2023; Templeton et al., 2024; Oikarinen et al., 2023; Oikarinen & Weng, 2023; Shaham et al., 2024; Singh et al., 2023) only use 1-2 metrics for evaluation and typically did not discuss or justify why the metric should be used. Among all the prior work in Table B.I, we believe that the most similar work to ours is (Huang et al., 2023), which has focused on evaluating individual neuron explanations in language models. In particular, they discover discrepancy between the evaluation metrics, i.e. neurons with very high still have relatively low Fl-scores. However, different from our work their scope is much more specific and they do not provide analysis comparing different evaluation metrics or justification on why they use Fl-score specifically. Their findings are in line with ours, as we found that Correlation(top-ante and-random) fails the extra labels test, while Fl Fl-score passes our sanity checks.
score can
Overall we find many evaluations in previous works to be lacking, either due to using poor metrics that fail our sanity checks, or using very small sample sizes. In addition, some popular methods like TCAV (Kim al., 2018) completely lack evaluation of whether the concept directions they learn are good. To our best knowledge, no human-study has been conducted using metrics that pass our sanity checks, instead most existing human-studies only measure Recall or a similar metric, and running such study would be valuable for better understanding of unit interpreability and/or explanation methods.
Known Concept Evaluation: Similar to our experiments in Section 5, many previous works such as (Oikarinen & Weng, 2023; Schwettmann et al., 2023; Moakhar et al., 2024; Shaham et al., 2024) have utilized neurons or units with known concepts to evaluate explanation quality. However, these papers typically focus on evaluating explanation methods or individual explanations, while our focus is on meta-evaluattin understand which metrics are best to use for evaluation, meaning our methods and motivations for the evaluation are noticeably different from these previous works.
Interpretability Illusions: (Bolukbasi et al., 2021) discovered an interpretability illusion for BERT-models, which shows that neurons that look interpretable when only looking their most highly activating inputs on one probing dataset D, seem to be doing something completely different when analyzed on another dataset. This highlights the importance of good choice of probing dataset, or confirming explanations on multiple datasets. However, we believe there is another reason for these illusions, namely that the authors only look at highly activating inputs (equivalent to Recall in our framework), which is not a reliable metric. Overall, our goal is to reduce such interpretability illusions, i.e. the appearance of interpretability when something is really not interpretable, by encouraging more rigorous evaluation.
Sanity Checks. The sanity checks proposed in this paper (Missing and Extra Labels test) are inspired by the sanity checks (Adebayo et al., 2018) proposed for saliency maps, which has had a large impact in guiding that field towards more faithful explanations. However the topic is very different, since (Adebayo et al., 2018) focus on local imputtimportance instead of global neuron explanations in our paper. Similarly, the specific tests proposed by (Adebayo et al., 2018) are also very different from ours.
Concept extraction. Extracting interpretable concepts from a learned representation is a common challenge and relevant for finding individual units to evaluate in our framework. This can either be supervised as proposed by (Kim et al., 2018) where the concepts are specified by human and labels are provided. This approach is also used by linear probing based work such as (Gurnee al., 2023). Later, series of WhhksaGi al., 2019; Zhang et al., 2021; Fel al., 2023) was proposed to automatically extract concepts from model activations, without human supervision, i.e. unsupervised. (Fel et al., 2024) claimed that concept extraction could be regarded as dictionary learning problem. Recently Sparse Autoencoders (Cunningham et al., 2023; Templeton et al., 2024) have also gained popularity as an unsupervised concept extraction method. Note that most unsupervised concept extraction methods are discovering "units" defined in our work that are not directly understandable to humans, and require an explanation method to provide human uumaan-nndertandaalle explanations. Our work focuses on the evaluation of the explanations of those concepts.
21

---

Evaluating Neuron Explanations: A Unified Framework with Sanity Checks
Metric M
Study
Concept Source &
Granularity
Probing Datasef
Domain
Target
Specific eval dataset Specific eval dataset
Recall
Human Eval (Zhou et al., 2015)
Crowdsourced
Whole Input
Vision
Neurons
Reeaal
Human Eval (Bau al., 2017)
Crowdsourced
Whole Input
Vision
Neurons
Human Eval (Oikarinen & Weng, 2023)
Recall
(Bai et al., 2025; Srinivas et al., 2025)
2025)
Crowdsourced
Whole Input
Validation data
Vision
Neurons
Recall
Human Eval (Oikarinen al., 2023)
Crowdsourced
Whole Input
Validation Data
Vision
CBM neurons
Precision
Automated Fval (Srinivas et al., 2025)
Generative
Whole Input
Generated+ Validation data
Vision
Neurons
Fl-score
Observation based (Huang al., 2023)
Generative model
Whole Input
Generated Training data
Language
Neurons
Fl-score
Sparse probes (Gurnce ct al., 2023)
Labcled data
Per-token
dataset
Specific eval
Language
Linear comb. of neurons
Fi-score
CBM Concept Error (Fig. 2)
(Koh et al., 2020)
Labeled data
Whole Input
Validation Data
Vision
CBM neurons
Broden ToU (Bau et al., 2017)
Specific datasct
IoU
(Mu & Andreas, 2020; La Rosa et al., 2024)
Labeled data
Per-pixel
Vision
Neurons
Accuracy
CBM Concept Error (Koh et al., 2020)
Labeled data
Whole Input
Validation
Vision
CBM neurons
~ACC
Comparative Human Study
Crowdsourced
Whole Input
Training data
Vision
Neurons
(Zimmermann et al., 2023)
Inverse AUC
INVERT (Bykov al., 2023)
Labeled data
Whole Input
Validation data Generated+ Validation data
Vision
Neurons
Inverse AUC
CoSy AUC (Kopf et al., 2024)
Generative
Whole Input
Vision
Neurons
Correlation(T&R)
Simulation Correlation Score (Bills et al., 2023)
Model
Per-token
Training data
Language
Neurons
Simulation Correlation Score
Corrllatinn
(Oikarinen & Weng. 2024)
Model
Whole Input
Validation data
Vision
Neurons
Correlation
CBM - Concept Eiror (Fig. 2)
(Koh al., 2020)
Labeled data
Whole Input
Validation Data
Vision
CBM neurons
Conrelation
Spearman
(Bricken al., 2023; Templeton et 2024)
SAE Auto Interp
Model
Per-token
Training data
Language
SAE features
CLIP-Disseet- Similarity (Oikarinen & Weng, 2023)
WPMM
Model
Whole Input
Validation data
Vision
Neurons
MAD
CoSy MAD (Kopf al., 2024)
Generative
Whole Input
Generated+ Validation data Generated
Vision
Neurons
MAD
MAIA (Shahum et al., 2024)
Generative
Whole Input
Vision
Neurons
Transformer
MAD
Explanation Score (Singh et al., 2023)
Generative
Whole Input
Generated
Language
factors
Table B.I. Extended Table (of Table 1 in the main text section 4) comparing related work. Note (Zhou et al., 2015) call their evaluation Precision, as they use the classification framing. This is equivalent to Recall our framing as shown in Section A.2.
Human-centere evaluation on concept-based models. Human-centered evaluation of model explanation Kim et al., 2024; Boyd-Graber et al., 2022) has drawn attention from the XAI community. Recently, (Kazmierczak et al., 2024) collected human evaluation of XAI explanations as a benchmark for explanation methods. These works provide important techniques and inspiration for evaluating explainability, but almost all existing work is focus on local feature importance explanations, which is very different from work global neuron-level explanations.
Evaluation Metrics for Input- Improtamme Explanations The field of input-importance explanations has evolution in the evaluation metries used, with initial focus on finding the features that humans think are important. Later metrics such as deletion and insertion proposed by (Petsiuk et al., 2018) allow for more principled evaluation of the explanation fidelity, ie. whether it actually matches what the model in vision models, and (Fel et al., 2021) extends the deletion metric to natural language settings. (Hooker et al., 2019) proposes the Remove And Retrain (ROAR) framework as an alternative method for evaluating the quality of input-importance explanations by evaluating whether a model retrained on data without the most important pixels can still solve the task. (Bhatt et al., 2020) propose additional checks, such as measuring whether an explanation method has high sensitivity, with the intuition that similar inputs should have similar explanations. While this line of work focuses on different topic (local input importance explanations) instead of global neuron-level explanations, and mostly proposes new metrics without significant meta-evaluation, it highlights the importance of good evaluation mnetrics in explainable AI.
22

---

Evaluating Neuron Explanations: A Unified Framework with Sanity Checks
C. Theoretical Missing/Extra Labels Test and Concept Imbalance
In this section we analyze the effects of missing/extra labels test on a simpler toy setting, where neuron k's activations as perfectly match the concept labels of concept fx. i.e. O₃ = Ctyy In this setting, we assume binary neuron and concept activations, i.c. the neuron's 's activation is 1 if concept is present on the input, and 0 if its not. Consequently, we do not need to perform additional binarization of concept activations with top like we did in previous sections.
In this simplified setting, our missing and extra labels tests correspond to being able to differentiate between three concepts as defined in section 4:
1. Ct. The perfect predictor for neuron ag., with precision=1 and recall=1.
2. OT. : (Missing labels) This concept has Precision=1 since whenever a concept present, the neuron is also active, and Recall of 0.5 since only half to inputs where the neuron activate now have the concept.
3. c (Extra labels) Inverse from above, this concept has Precision=0.5 and Recall=1.
We then measure the average score difference As across neurons:
Missing labels test: (k) Beg [SAs (aks(ak; (T,) sM (ake )tc)
Extra labels test: As(k)= = B (ak, ctt) sar ctu)]
A good metric should be able to reliably differentiate between these concepts. Interestingly we find that the ability of most metrics to differentiate greatly depends on whether the data is balanced or not.
Since the neuron activations perfectly align with concept and are binary, the only parameter that can effect the results of our missing and extra labels test is the activation frequency of concept t, i.c. what fraction of inputs æ E D contain concept t. Following the notation in section D, we denote this fraction as y. Note technically should be y + 17. but 1) = in this case with perfect match between concept and neuron. We then test whether metric passes the test on different values of y, using simulated data on tables C.3 and C.4. Each number is the average result from 1000 evaluations with 500,000 datapoints each. In addition, in Section D, we derived closed form solution to the binary metrics under missing or extra labels as a function of y and other parameters. This simplifies nicely when we consider an ideal neuron with ag Ct, and we can derive the expected result of missing/extra labels test as a simple function of ? alone in Table D.2. These theoretical results perfectly agree with our simulated results.
C.1. Results
In Tables C.1 and C.2 we report the Decrease Acc defined in Equation 11 on different activation frequencies 7 in our simulated neuron. We say a metric as passing the Theoretical test if it has Decrease Acc > 90% on every frequency. In tables C.3 and C.4 (visualized in Fig. 3) we report the average As on the same set different activation sparsities. We have highlighted metrics with As > -0.01 in red to highlight metries approaching 0 difference.
We can most metrics (expect from recall and precision) perform well on balanced data (7 0.499 and ي 0.1). However, their performance often starts to drop with score difference As approaching 0 as the data becomes more and more unbalanced. We can see that practically all the metrics that failed our experimental missing/extra labels tests cannot differentiate between perfect and modified concept specifically on imbalanced data, highlighting that likely the root cause of the failure on these test is that the metrics performs poorly on imbalanced data. This is also aligned with conventional knowledge that metrics such as accuracy and AUC are a poor choice to rely on when your data is heavily imbalanced. On the other hand, metries that passed the tests are insensitive to activation frequency 7 and converge to nonzero constant as ? decreases.
The results in terms of passing are almost identical to our experimental results in section 4. The only differences were WPMI which passes the theoretical extra labels test but fails the experimental one. We believe this has to do with hyperparameter(a, A) choices and that WPMI can in principle pass the test but with poor hyperparameters(small X) it will not, leading us to overall recommend against using it in practice as hyperparameter choice is challenging in the real world. An interesting case is Inverse AUPRC. This metric is designed for imbalanced data and works well in that domain,
23

---

Evaluating Neuron Explanations: A Unified Framework with Sanity Checks
but actually performs worse when the data is balanced. In particular Inverse AUPRC fails the test when data is perfectly balanced. This indicates caution should be used if relying on it, in case you have some neurons with extremely common concepts.
We argue that being able to pass these tests regardless of activation frequency is important for any evaluation metric to be used, as we typically do not know what frequency each neuron will have in advance, in many cases the interesting neurons/concept might activate very sparsely, for example in Sparse Autoencoders.
Theoretical Missing Labels Test, Decrease Acc
Activation Frequency 7:
0.499
0.1
0.01
0.001
0.0001
Pass
Recall
100.00%
100.00%
100.00%
100.00%
100.00%
Precision
0.00%
0.00%
0.00%
0.00%
0.00%
X
F1-score
100.00%
100.00%
100.00%
100.00%
100.00%
IoU
100.00%
100.00%
100.00%
100.00%
100.00%
Accuracy
100.00%
100.00%
100.00%
0.00%
0.00%
X
Balanced Acc.
100.00%
100.00%
100.00%
100.00%
100.00%
V
Inverse Balanced Acc.
100.00%
100.00%
100.00%
0.00%
0.00%
X
AUC
100.00%
100.00%
100.00%
100.00%
100.00%
Inverse AUC
100.00%
100.00%
100.00%
0.00%
0.00%
Correlation
100.00%
100.00%
100.00%
100.00%
100.00%
Correlation(T&&))
100.00%
100.00%
100.00%
100.00%
100.00%
Spearman Correlation
100.00%
100.00%
100.00%
22.70%
13.30%
Spearman Correlation
100.00%
100.00%
100.00%
100.00%
100.00%
Cosine
100.00%
100.00%
100.00%
100.00%
100.00%
WPMI
100.00%
100.00%
100.00%
100.00%
100.00%
MAD
100.00%
100.00%
100.00%
0.00%
0.00%
X
AUPRC
100.00%
100.00%
100.00%
100.00%
100.00%
Inverse AUPRC
100.00%
100.00%
100.00%
100.00%
100.00%
V
Table C.I. Average Decrease Accuracy on Theoretical Missing labels Test. metric fails if has low decrease accuracy any concept
frequency 7-
24

---

Evaluating Neuron Explanations: A Unified Framework with Sanity Checks
Theoretical Extra Labels Test, Decrease Acc
Activation Frequency 7::
0.499
0.1
0.01
0.001
0.0001
Pass
Recall
0.00%
0.00%
0.00%
0.00%
0.00%
X
Precision
100.00%
100.00%
100.00%
100.00%
100.00%
F1-score
100.00%
100.00%
100.00%
100.00%
100.00%
IoU
100.00%
100.00%
100.00%
100.00%
100.00%
V
Accuracy
100.00%
100.00%
100.00%
48.40%
0.00%
x
Balanced Acc.
100.00%
100.00%
100.00%
0.00%
0.00%
X
Inverse Balanced Acc.
100.00%
100.00%
100.00%
100.00%
100.00%
AUC
100.00%
100.00%
100.00%
0.00%
0.00%
X
Inverse AUC
100.00%
100.00%
100.00%
100.00%
100.00%
Correlation
100.00%
100.00%
100.00%
100.00%
100.00%
J
Correlation(T&RR
100.00%
92.80%
22.60%
2.70%
0.10%
Spearman Correlation
100.00%
100.00%
5.60%
3.20%
11.60%
Spearman Correlation(T&RR
0.20%
8.60%
20.60%
63.70%
5.30%
Cosine
100.00%
100.00%
100.00%
100.00%
100.00%
WPMI
100.00%
100.00%
100.00%
100.00%
100.00%
MAD
100.00%
100.00%
100.00%
100.00%
100.00%
AUPRC
100.00%
100.00%
100.00%
100.00%
100.00%
V
Inverse AUPRC
47.70%
100.00%
100.00%
100.00%
100.00%
X
Table C.2. Average Decrease Accuracy on Theoretical Extra labels Test. A metric fails if it has low decrease accuracy on any concept
frequency 7.
Theoretical Missing Labels Test, Average As
Activation Frequency 7:
0.499
0.1
0.01
0.001
0.0001
limy-vo
Recall
-0.5000
-0.4999
-0.5002
-0.5007
-0.5025
-0.5000
Precision
0.0000
0.0000
0.0000
0.0000
0.0000
0.0000
F1-score
-0.3334
-0.3333
-0.3335
-0.3341
-0.3352
-0.3333
IoU
-0,5000
-0.5002
-0.4998
-0,5005
-0.5032
-0,5000
Accuracy
-0.4495
-0.0500
-0.0050
-0.0005
0.0000
0.0000
Balanced Acc.
-0.5500
-0.2500
-0.2501
-0.2500
-0.2500
-0.2500
Inverse Balanced Acc.
-.1662
-0.0263
-0.0225
-0.0002
0.0000
0.0000
AUC
-0,5000
-0.2500
-0,2500
-0,2493
-0.2508
Inverse AUC
-0.1662
-0.0263
-0.0025
-0.0003
0.0000
Correlation
-0.2111
-0.1559
-0.1474
-0.1466
-0.1479
Correlation(T&R)
-0.4965
-0.4894
-0.4649
-0.3928
-0.2449
Spearman Correlation
-020000
-0.0686
-0.0079
-0.0006
-0.0018
Spearman Correlation(T&R)
-0.1005
-0.1009
-0.1066
-0.0902
-0.2006
Cosine
-0.1464
-0.1465
-0.1465
-0.1464
-0.1474
WPMI
-0.5500
-0.3591
-0.3589
-0.3587
-0.3602
MAD
-0.2210
-0.0350
-0.0033
-0.0003
0.0000
AUPRC
-0.2505
-0.4499
-0.4953
-0.5003
-0.4964
Inverse AUPRC
-0.5000
-0.4999
-0.5002
-0.4988
-0.4996
Table C.3. Simulation results missing lahels test on idealized neuron with perfect correspondence to a concept activation. We can see most metrics pass when data relatively balanced, but several start struggle on inbalanced data (low Y). contrast, passing metries maintain a clearly ponzero As regardless of activation frequency. The lim,- column is analytical solution from D.2.
25

---

Evaluating Neuron Explanations: A Unified Framework with Sanity Checks
Theoretical Extra Labels Test, Average As
Activation Frequency 7:
0.499
0.1
0.01
0.001
0.0001
limy-o
Recall
0.0000
0.0000
0.0000
0.0000
0.0000
0.0000
Precision
-0,5000
-0.5001
-0.4999
-0.4993
-0.4963
-0,5000
Fi-score
-0.3333
-0.3334
-0.3334
-0.3336
-0.3333
-0.3333
IoU
-0.5000
-0.5000
-0.5001
-0.4997
-0.4970
-0.5000
Accuracy
-0.4990
-0.1000
-0,0100
-0.0010
-0.0001
0.0000
Balanced Acc.
-0.4980
-0.0556
-0.0051
-0.0005
0.0000
0.0000
Inverse Balanced Acc.
-025000
-0.2500
-0.2500
-0,2500
-0.2483
-0,2500
AUC
-0.7562
-0.0844
-0,0077
-0.0008
-0.0001
Inverse AUC
-0.2500
-0.2500
-0.2501
-0,2500
-0.2491
Correlation
-0.4777
-0.1667
-0.1483
-0.1465
-0.1461
Correlation(T&R)
-0.4929
-0.0293
-0.0227
-0.0002
0.0000
Spearman Correlation
-0.5672
-0.0238
-0.0007
-0.0002
-0.0018
Spearman Co)Telation(T&RR)
0,0002
0.0000
-0.0036
0.0307
-0.0064
Cosine
-0.1464
-0.1464
-0.1463
-0.1464
0.1456
WPMI
-0.0292
-0.0292
-0.0292
-0.0293
-0.0292
MAD
-0.3324
-0.3324
-0.3324
-0.3321
-0.3298
AUPRC
-0.5000
-0.5000
-0.4999
-0.4998
-0.4974
Inverse AUPRC
-0.0010
-0.4000
-0.4899
-0.4984
-0.4957
Table C.4. Simulation results extra labels test on idealized neuron with perfect correspondence to a concept activation. We can see most metrics pass when the data is relatively balanced, but several struggle inbalanced data (low 7). In contrast, passing metries maintain a clearly nonzero As regardless of activation frequency. The lim column is analytical solution from D.2.
26

---

Evaluating Neuron Explanations: A Unified Framework with Sanity Checks
D. A nnllytica solution for Missing/Extra Labels Test
In this section, we provide a theoratical analysis for missing label and extra label test binary classification metrics. For simplicity of symbols, in this section we analyze the population statistics. Suppose we have following confusion matrix:
c=
7
7 d
Here, we use to denote neuron activation and to denote concept activation. In missing labels test, consider general case where we randomly flip c into with probability p. Thus, the resulting confusion matrix is:
c=1
c=0
a=l
(1 - p)y
b b+
a=0
p)v
+ pr
In extra labels test, similarly, we turn into c == with probability y=
p(a+v) 64d
the resulting confusion matrix is
c=1
c=0
a=l
y + qb
(1 9
a=0
17 + qd
q)d
With these, we could plug in corresponding TP/FP/NN/NN into metrics to calculate metric value in these two tests.
1. Recall:
SM (ax, cz) B(ax) B(c,)
(D.1)
BB((x)]ll b+7
In extra label test:
7 qb b+7
sM(ak,
> sM(
(D.2)
In missing label test:
7 7-P7 PY
(ak, 67
< SM (ax: Ct).
(D.3)
b b+9
From the derivation above, could see that increasing labels only raises recall metric while reducing labels always leads to drop in recall as we found in our experiments.
2. Precision:
(an, B(ax) B(a))
7
(D.4)
[ 7 +1
In extra label test:
7 qb
sx)(a,,))) 7 qb + qd
(D.5)
Precision will increase if b 7 In missing label test:
(1 p)y
SM (ak, C ==
(D.6)
p)y (1 p)n
Thus, the precision does not change in the missing labels test.
27

---

Evaluating Neuron Explanations: A Unified Framework with Sanity Checks
3. Fl-score:
24 2 + 17 +
SM (ak,
(D.7)
In extra label test:
27 + 2qb
8M (kk
(t) (ak,
(D.8)
20 qb + qd +
Fl-score increases if 26 b4d
2x
27-1)+6"
In missing label test:
2(1 - p)y
27 - 2pry
(ak p17 + (1 p)n + + py
(D.9)
2y + (1 p)1 Py
Fi-score decreases in missing label test.
4. loU:
B(as) B(a))
(ak, B 11 B|[[(c - B(ax) B(c))
(D.10)
7+ 7+1++ +
In extra label test:
7 qb
(t)
(D.11)
7 17 qd +
IoU increases if 3 > 7+ In missing label test:
(1- SM (T) (1-p)7 (1 p)1 +b
7- pr
(D.12)
7+ (1 p)n +
Thus, IoU decreases in missing label test.
5. Accuracy:
SM (ak, = B(ax) BB(c) + (1- ((((x)))) (1- B(a))
= 7 + d.
(D.13)
n
In extra label test:
8M ((),,(t)=)) = qb + - qd.
(D.14)
accuracy increases b > d. In missing label test:
(7) = py pd-
(D.15)
Accuracy increases if 17 >
6. Balanced Accuracy:
s (a c.) B(as). B(e)).
(1 -1 ((x)) (1 B(a))
7
d
+
(D.16)
2) B(ax)llı
- B((x))ll
2n + 26
2n + 2d
In extra label test:
= 7 qb
+ qd
SM (an,
(D.17)
27 + 26
2n + 2d
balanced accuracy increases if In missing label test:
d
2y426
2d
- py
d pq
SM Car, 07
(D.18)
2n + 2b
2n + 2d
balanced accuracy increases if
7
25
2n 2d
28

---

Evaluating Neuron Explanations: A Unified Framework with Sanity Checks
7. Inverse Balanced Accuracy:
B(ax)- B(e))
(1 - B(ax)) (1-10011 B(a))
?
+ d
sar ce)
(D.19)
Bl)B(l
2|[(1 BB()))l
27 + 2n
2b + 2d
In extra label test:
7 7 qb
d qd
SM (ap,
(D.20)
2y + 21) + 2qb + 2qd
2b + 2d 2qd - 2gb
In missing label test:
7 Py
+ pd
Sy (ak cr)
(D.21)
24 + 2n 2p7 2p7
26 + 2d + 2pr + 2pm
Metric
Missing label: sMr c.
Extra label: s (ak: of
Recall
p
Precision
1
1
Fl-score
2 2p
2-p
2+p
IoU
1 -
14p
Accuracy
My
1 YP
Balanced Accuracy
1 2
1
21
Inverse Balanced Accuracy
(1-7)
2+P
2(1-7)+2pp
2(1+p)
Table D.J. The evaluation scores for different metrics under missing and extra label tests on ideal neuron whose activations perfectly match the presence our concept.
D.1. Special case Theoretical Test
In this section, we consider special case where the neuron activations perfectly match the concept, i.e. c In this case, we have = b = 0, d= = - 7. Plugging in those variables, we get the equations for metric scores under sanity test settings in Table D.I. Further simplifying this by plugging in the values we typically use we get the simple form for score diff in Table D.2.
Missing labels test(p 0.5):
Extra labels test(p 1):
Metric
SMd (ag, G sM (ak, c.)
S (ag, (ak,ct) SM (ak, Ca)
Recall
0.5
0
Precision
0
0.5
F1-score
IoU
0.5
-0.5
Accuracy
-7
Balanced Accuracy
0.25
2(1-7)
Inverse Balanced Accuracy
-0.25
2(2
Table D.2. Further simplifying from Table D.1 by plugging in p values we typically use in our tests, and ar (ag, 0t) = 1, we can calculate the theoretical score diff after running tests for different binary metrics on ideal neurons.
29

---

Evaluating Neuron Explanations: A Unified Framework with Sanity Checks
E. Ablations and Extra checks
E.1. Incrcase/decreass Fraction Missing/Extra labels test
In our standard missing labels we reduce the fraction of positive labels by half, i.c.
r_ l 1l
0.5
||elll
||
7 =
||ca/11
=
In this section we run ablation the importance of these specific values by running the test other combinations of values: 0.75 and r+ 1.33 (smaller change) and 0.33 and r+ 3 (larger change). We report the results on final layer neurons (including superclasses) of ViT-B-16 (trained on ImageNet) in Table E. We used ground turth Overall we can see that these parameters do not impact our qualitative observations, i.e. which metric passes vs which doesn't. 't. While larger changes to r typically increases Decrease Acc, it doesn" 't affect which metrics approach zero score diff' vs which do not, which is what we' re mostly interested in. Overall this shows our sanity test is not sensitive to the specific parameter choice but instead reflects overall trends of the metric.
Missing Labels Test
Extra Labels Test
7- 0.33
r- 0.5
r- 0.75
F+ 1.33
r+ 2.0
r+ 3.0
Recall
99.55%
95.94%
75.34%
0.00%
0.00%
0.00%
Precision
48.20%
46.09%
49.62%
99.32%
99.62%
99.62%
F1-score
97.07%
95.64%
92.78%
99.62%
99.70%
99.70%
IoU
96.84%
95.34%
92.41%
99.32%
99.62%
99.62%
Accuracy
0.00%
0.00%
0.00%
16.24%
63.08%
99.92%
Balanced Acc.
99.47%
95.86%
75.19%
9.17%
23.98%
63.61%
Inverse Balanced Acc.
48.05%
45.56%
48.35%
99.17%
99.62%
99.92%
AUC
98.65%
95.34%
76.32%
16.99%
30.23%
59.32%
Inverse AUC
24.51% %
21.35%
16.47%
99.92%
99.92%
99.92%
Correlation
99.92%
99.92%
99.77%
99.92%
99.92%
99.92%
Correlation(T&&
99.40%
98.12%
90.68%
43.76%
44.96%
48.12%
Spearman Correlation
57.29%
53.91%
46.92%
37.07%
38.05%
37.67%
Spearman Correlation(T&&R
97.14%
93.61%
82.63%
49.92%
50.75%
52.26%
Cosine
100.00%
100.00%
99.85%
99.85%
99.85%
99.85%
WPMI
99.55%
95.94%
75.34%
99.77%
99.70%
99.77%
MAD
50.83%
49.47%
49.02%
99.70%
99.70%
99.70%
AUPRC
99.47%
99.25%
98.27%
99.32%
99.62%
99.62%
Inverse AUPRC
100.00%
100.00%
99.85%
99.70%
99.62%
99.32%
Table E. Ablation study on sanity checks varying the magnitude of change in labels. We can see overall our results are consistent and not heavily dependent the choice of
30

---

Evaluating Neuron Explanations: A Unified Framework with Sanity Checks
E.2. Failure case of cosine: Adding constant activation
In this section we discuss failure case of cosine similarity an evaluation metric. The main idea is that cosine similarity outputs are not independent of the mean of the neuron's 's activation, while all other evaluation metrics are. This causes it to associate neurons with large average activation values with very generic concepts that almost always active. As an example, we added a constant (1) to all activations of concept neurons in a Concept Bottleneck Model (Koh et al., 2020) trained on CUB200, and ran our meta- AUPRC test(Sec 5). All other evaluation metrics are invariant to this change, but the AUPRC of Cosine drops from 99.07% to 0.93% with this small change, as shown in Table E.2. We believe this a flaw that points against using cosine similarity, as the average activation a hidden layer neuron not functionally important, and could be absorbed into biases of the next layer. Instead we recommend using Pearson correlation which is identical cosine similarity after normalizing to mean 0, show in Section A.4.
CUB200 CBM concept neurons
Original Ox d/. ax +
Metric
AUPRC
AUPRC
AUPRC
Recall
0.3958
0.3958
0.0000
Precision
0.6803
0.6803
0.0000
F1-score/loU
0,6386
0.6386
0,0000
Accuracy
0.5853
0.5853
0.0000
Balanced Acc.
0.7165
0.7165
0.0000
Inverse Balanced Acc.
0.6628
0.6628
0.0000
AUC
0.7045
0.7045
0.0000
Inverse AUC
0.8956
0.8956
0.0000
Correlation
0.8883
0,8883
0.0000
Correlation(T&R)
0.7170
0.7170
0.0000
Spearman Correlation
0.4233
0,4233
0.0000
Spearman Correlation(T&RR
0,4243
0.4243
0,0000
Cosine
0.8985
0.0443
-0.8542
WPMI
0.7323
0.7323
0.0000
MAD
0.7760
0.7760
0.0000
AUPRC
0,6712
0.6712
0.0000
Inverse AUPRC
0.3975
0.3975
0.0000
Table E.2. Adding constant to activation values of all neurons causes cosine similarity to perform very poorly, while other metrics are unchanged.
31

---

Evaluating Neuron Explanations: A Unified Framework with Sanity Checks
E.3. Random Semantic subsets
Our missing/extra labels tests are mathematical model of semantic sub/superclassee that can be run without knowledge of the semanties relationships between concepts. To test this mathematical model, we conducted version of extra labels test on the ImageNet dataset where instead of randomly sampling ct, used the smallest superclass of the concept according to WordNet hierarchy as ct,, and version of missing labels test where we used the largest subclass of the concept cm. As shown in new Table G.5, the results are essentially identical to using random sub/supersets, showcasing our random sub/superset is a good model of real semantic relationships for our purposes.
ViT-B-16 16 superclass neurons
ViT-B-16 single class neurons
Missing Labels Test
Extra Labels Test
Metric
Random Subsets
Semantic Subsets
Random Supersets
Semantic Supersets
Recall
96.32%
92.89%
0.00%
0.00%
Precision
44.21
43.68%
100.00%
99.79%
F1-score
97.37%
91.84%
100.00%
99.79%
IoU
96.84%
91.58%
100.00%
99.79%
Accuracy
44.74%
31.32%
46. 11%
71.37%
Balanced Ace.
99.47%
97.37%
0.00%
64.53%
Inverse Balanced Acc.
28.95%
37.37%
100.00%
99.79 99.79%
AUC
99.21%
97.1 1 %
17.47%
52. %
Inverse AUC
51.58%
54.47%
100.00%
96.00%
Correlation
100.00%
100.00%
100.00%
99.58%
Correlation(T&ll
100.00%
99.21%
47, 47.47%
70. %
Spearman Correlation
70.53%
75.26%
37.37%
20.63%
Spearman Correlation(T&RR
98.42%
97.63%
51.05%
35, 16%
Cosine
100.00%
100.00%
100.00%
99.58%
WPMI
100.00%
100.00%
0.00%
0.00%
MAD
58. 38.42%
51.58%
100.00%
100,00%
AUPRC
99.21 %
96.84%
100.00%
99.37%
Inverse AUPRC
100.00%
100.00%
100.00%
99.37%
Table E.3. Comparison of 6. derived as random subelass cs real semantic subelass c, derived from WordNet hierarchy. The missing labels test evaluated on all 400 superclass neurons created in the final layer, while the extra labels test was conducted on all 1000 individual class neurons. can see that from the point view of passing our tests the random subclass behaves almost exactly real semantic subclass.
32

---

Evaluating Neuron Explanations: A Unified Framework with Sanity Checks
E.4. Epsilon ablation
In this section we study the effect of the choice of in Equation 10 on experimental missing/extra labels test. Overall we see in Table E.4 that while changing by changes results for some metrics, for the most part it does not affect which metrics pass do not pass the test despite orders of magnitude change in showing our tests are not very sensitive to this choice.
Prehaps more importantly, theoretically we do not think the e choice important. An alternative and perhaps more fundamental definition for our theoretical tests could as follows: A metric passes the test if
3e s.t. Vy 0, 0. <E
(E.1)
where As is calculated with the theoretical missing labels test defined in Appendix C and is the concept/neuron activation frequency. This definition is purely mathematical property of the metric without any hyperparameters or ties to any setting, and as far as we know matches our theoretical test results of Table C. and Table C.2 that use a fixed = 0.001.
The issue with this alternative definition is that, we can only prove a metric passes this version the test if we have an analytical solution, which we have for binary classification metrics in Table D.2. This makes this definition hard to use in practice. We
for example accuracy fails this test lim.,-0 As As -c On the other hand, for F1-score will pass the theoretical tests for any 1/3 regardless of of From this see that if decrease in the current theoretical test, more metrics would pass, but the same metrics would fail again if we expand Table C.1/C.2 to the right by including smaller values, while current passing metrics would not fail regardless of how many additional ? values we test.
matter what choose, Fy s.t.
Setting 1: ViT-B-16 Final Layer neurons
Missing Labels Test Decrease Ace
Extra Labels Test Decrease Acc
0.0001
0.001
0.01
0.0001
0.001
0.01
Recall
97.07%
97.07%
97.07%
0.00%
0.00%
0.00%
Precision
48.65%
47.14%
37.74%
99.70%
99.20%
97.89%
F1-score
96.48%
95.79%
93.38%
99.77%
99.70%
98.72%
IoU
96.69%
95.41%
92.48%
99.70%
99.62%
97.89%
Accuracy
66.09%
0.00%
0.00%
99.92%
61.50%
6.32%
Balanced Acc.
97.07%
97.07%
97.07%
99.77%
23.08%
3.76%
Inverse Balanced Acc.
51.95%
46.54%
31.35%
99.92%
99.62%
96.77%
AUC
96.17%
95.86%
95.19%
84.81 %
29.25%
11.58%
Inverse AUC
78.35%
21.43%
3.68%
99.92%
99.92%
99.92%
Correlation
100.00%
100.00%
99.92%
99.92%
99.92%
99 99.92%
Correlation(T&R)
97.52%
97.29%
96.47%
48.42%
46.02%
37.07%
Spearman Correlation
63.68%
53.91%
5.94%
48.87%
37.07%
1.28%
Spearman Correlation(T&&R)
93.91%
93.76%
92.56%
51.73%
50.90%
44.36%
Cosine
100.00%
100.00%
100.00%
99.92%
99.85%
99.70%
WPMI
97.07%
97.07%
97.07%
99.77%
99.70%
99.55%
MAD
52.78%
50.75%
26.32%
99.70%
99.70%
99.70%
AUPRC
99.77%
99.40%
97.59%
99.70%
99.62%
97.89%
Inverse AUPRC
100.00%
100.00%
100.00%
99.62%
99.62%
99.62%
Table E.4. An Ablation study the effect of choice results. While the exact percentages vary, that changing by orders of magnitude mostly does not change which metrics pass the tests, highlighting robustness to this hyperparameter choice.
33

---

Evaluating Neuron Explanations: A Unified Framework with Sanity Checks
E.5. Number of samples for of
Most of results only use one sample of of to reduce computational cost, even though our equations are defined in terms of its expectation. In Table E.5 test whether using samples of of affects results. As see, the number of samples has very little effect results likely because they already averages many neurons and large datasets.
Setting 1: ViT-B-16 Final Layer neurons
Missing Labels Test
Extra Labels Test
samples per neuron:
3
10
100
10
100
Recall
97.07%
100.00%
100.00%
100.00%
0.00%
0.00%
0.00%
0.00%
Precision
47.14%
45.04%
44.29%
38.80%
99.20%
99.62%
99.62%
99.62%
F1-score
95.79%
96.77%
97.29%
97.74%
99.70%
99.70%
99.70%
99.70%
IoU
95.41%
96.02%
96.62%
97.22%
99.62%
99.62%
99.62%
99.62%
Accuracy
0.00%
0.00%
0.00%
0.00%
61.50%
61.88%
64.66%
66.54%
Balanced Acc.
97.07%
99.92%
99.77%
99.85%
23.08%
22.48%
23.31%
22.18%
Inverse Balanced Acc.
46.54%
43.83%
42.41%
34.44%
99.62%
99.62%
99.62%
99.62%
AUC
95.86%
99.25%
99.70%
99.77%
29.25%
26.84%
27.59%
28.35%
Inverse AUC
21.43%
19.17%
16.32%
14.21%
99.92%
99.92%
99.92%
99.92%
Correlation
100.00%
100.00%
99.92%
99.92%
99.92%
99.92%
99.92%
99.92%
Correlation
97.29%
99.32%
99.85%
99.77%
46.02%
52.48%
54.51
55.19%
Spearman Correlation
53.91% %
54.14%
54.59%
55.64%
37.07%
34.96%
35.86%
35.04%
Spearman Correlation(T&RR)
93.76%
96.39%
97.67%
98.65%
50.90%
57.97%
51.95%
53.83%
Cosine
100.00%
100.00%
100.00%
100.00%
99.85%
99.85%
99.85%
99.85%
WPMI
97.07%
100.00%
100.00%
100.00%
99.70%
99.77%
99.77%
99.85%
MAD
50.75%
48.95%
49.40%
39.17%
99.70%
99.70%
99.70%
99.70%
AUPRC
99.40%
99.32%
99.32%
99.32%
99.62%
99.62%
99.62%
99.62%
Inverse AUPRC
100.00%
100.00%
100.00%
100.00%
99.62%
99.62%
99.62%
99.62%
Table E.5. A study measuring whether our test results are sensitive to sampling o only once per neuron. We see this randomness has no effect results in terms of which metrics pass and do not pass the likely because the results already averaged over hundreds or thousands of neurons.
34

---

Evaluating Neuron Explanations: A Unified Framework with Sanity Checks
E.6. Combination Metrics
In addition individual metrics we studied in the main paper, we also explore whether combinations of different metries could serve good evaluation metrics. This investigation is inspired by the success of Fl-score, which is the harmonic mean of the Recall and Precision metrics.
R8ec Precissoo
(E.2)
1/ /Rccall + 1/ Precision
Secall
+ SProcision
In this section we investigated using the harmonic mean of other metrics as our metric. The missing and extra labels test results are reported in Tables E.6, E.7 and E.8. We can see multiple combinations perform well and now pass the theoretical missing/extra labels tests. In general, our initial results indicate that combining metric that passes the extra labels test with metric that passes the missing labels test will pass both tests the same test, such Recall and AUC will still fail that test. In table E.6, we also measure the AUPRC (Sec 5) performance of these new metrics on a subset of our settings, and show that many combination metrics are competitive with top standard metrics. In particular the harmonic mean of Balanced Acc and Inverse Balanced Acc performed well. Overall this shows the promise of using novel combinations of concepts, but a more throughout study needed to confirm their reliability.
of the time. Conversely, combining
metrics that fail
Theoretical Missing Labels Test, Decrease Acc
Activation Frequency 7
0.499
0.1
0.01
0.001
0.0001
Pass
AUC Inverse AUC
100.00%
100.00%
100.00%
100.00%
100.00%
Balanced Acc + Inverse Balanced Acc
100.00%
100.00%
100.00%
100.00%
100.00%
Recall + AUC
100.00%
100.00%
100.00%
100.00%
100.00%
Recall + Inverse AUC
100.00%
100.00%
100.00%
100.00%
100.00%
Precision + Balanced Acc
100.00%
100.00%
100.00%
100.00%
100.00%
Precision Inverse Balanced Acc
100.00%
100.00%
100.00%
0.00%
0.00%
F1-score
100.00%
100.00%
100.00%
100.00%
100.00%
Table E.6. Theoretical Missing Labels test on combination metries.
Theoretical Extra Labels Test, Decrease Acc
Activation Frequency 7/
0.499
0.1
0.01
0.001
0.0001
Pass
AUC Inverse AUC
100.00%
100.00%
100.00%
100.00%
100.00%
Balanced Ace + Inverse Balanced Ace
100.00%
100.00%
100.00%
100.00%
100.00%
Recall + AUC
100.00%
100.00%
100.00%
0.00%
0.00%
Recall + Inverse AUC
100.00%
100.00%
100.00%
100.00%
100.00%
Precision Balanced Acc
100.00%
100.00%
100.00%
100.00%
100.00%
Precision Inverse Balanced Ace
100.00%
100.00%
100.00%
100.00%
100.00%
F1-score
100.00%
100.00%
100.00%
100.00%
100.00%
Table E.7. Theoretical Extra Labels Test for combination metrics.
35

---

Evaluating Neuron Explanations: A Unified Framework with Sanity Checks
Setting 1: VTT-B-16 final layer neuron
Setting 2: Resnet-50 layer4 neurons
Missing Labels Test
Extra Labels Test
Missing Labels Test
Extra Labels Test
Combination Metric
Decrease acc
Decrease acc
Decrease acc
Decrease acc
AUC+ + Inverse AUC
95.64%
99.77%
89.05%
99.90%
Balanced Acc +
99.85%
99.92%
90.75%
100.00%
Inverse Balanced Acc
Recall + AUC
96.99%
19.77%
100.00%
17.32%
Recall + Inverse AUC
96.92%
99.92%
100.00%
87.00%
Precision + Balanced Acc
91.80%
99.70%
64.39%
100.00%
Precision +
48.87%
99.92%
47.38%
100.00%
Inverse Balanced Acc
F1-score
95.86%
99.70%
100.00%
100.00%
Table E.8. Experimental Missing and Extra Labels Test with new combination metrics.
Setting 3: gt oa
Setting SigLIP &t
ViT-B-16(ImageNet)
ViI-B-166ImageNet)
New metries:
AUPRC
Rank
AUPRC
Rank
+UC+ + Inverse AUC
0.8977
0.6551
10
Balanced Ace + Inverse Balanced Acc
0,8885
0.7041
3
Recall + AUC
0,7081
17
0.6129
15
Recall + Inverse AUC
0.8892
0.0077
22
Precision Balanced Acc
0.8679
0.7141
Precision Inverse Balanced Ace
0.8610
0.6468
12
Old metrics:
Recall
0.0832
22
0.5758
17
Precision
0.8592
11
0.6428
13
F1-score/loU
0,8759
0,6992
5
Accuracy
0.7858
0.5853 13
16
Balanced Accuracy
0.7463
0.6158 15
14
Inverse Balanced Acc.
0.8610
0.6474 16
AUC
0.7139
0.6933
Inverse AUC
0.8120
0.5439 12
18
Correlation
0.9399
0.7027
4
Correlation(T&RR
0,4365
18
0.4906
19
Spearman Correlation
0.0125
0.0047 23
23
Spearman (orrelation(T&&R)
0.1783
0.2395 20
20
Cosine
0.9037
0.6948 14
WPMI
0.7570
0,6881
MAD
0.1866
19
0.1268
21
AUPRC
0.8839
6
0.7511
Inverse AUPRC
0.1131
21
0.6688
Table F,9. Performance of new combination on the meta-AUPRC evaluation introduced section 5. metries such as Balanced Acc Inverse Balanced Ace perform well.
can see some combination
36

---

Evaluating Neuron Explanations: A Unified Framework with Sanity Checks
F. Detailed results
F.. Missing and Extra Labels Test:
Experimental Setup Details: We evaluate our experimental results across settings:
Dataset Net, ViT-B-16 (Dosovitskiy et al., 2021) trained on imagenet, 1400 final layer neurons(and superclass neurons), Table F.1
Setting I:
Setting 2:
Dataset D=ImageNet, ResNet-50 trained on imagenet, 2048 layer4 neurons, Table F.1
Setting 3:
Dataset D=Places365, ResNet-18 trained on Places365, 365 final layer neurons, Table F.2
Setting 4: Dataset D=Places365,, ResNet-18 trained Places365, 512 layer4 neurons, Table F.2
Setting 5: Dataset D=CUB200, CBM trained on CUB200, 112 concept neurons, Table F.3
Setting 6: Dataset D=CUB200, CLIP ViT-B-32 image encoder, linear probe trained to detect CUB-concepts, 112 concepts, Table F.3
Setting Dataset D=OppaWebTex((uusst)), GPT-2-small, final(prediction) layer neurons corresponding 500 most common tokens, Table F.4
Setting 8: Dataset D=OOpaWWebTcxx(suusstt) GPT-2-XL, final(peediction) layer neurons corresponding to 500 most common tokens, Table F.4
For all settings we used the ground truth labels from the dataset as cr. The ImageNet (Deng et al., 2009), Places (Zhou et al., 2017) and GPT-2 (Radford et al., 2019) models were pretrained. For CUB-CBL we trained our own model using the code released by (Koh et al., 2020). Our CBM reached 96. 66.75% concept accuracy on the test set which is in line with their reported results. For CLIP, we used the pretrained model from (Radford et al., 2021), and then learned a linear probe on top of frozen image embeddings to minimize binary cross-entropy loss the training split of CUB200 Wah et al., 2011), with early stopping using validation data. Our linear probe reached 89.76% concept accuracy. The CUB dataset is small bird species classification dataset that contains detailed annotations for lower level concepts, such as wing color. Following (Koh et al., 2020), we only used the 112 concepts that are present on at least 5% of the inputs and our CLIP linear probe was trained to predict these concepts, not the final class of inputs.
For the final layer neurons as well as CUB neurons we let "correct" concept by be the ground truth concept for that neuron. We choose the hyperparameter a that maximizes AUPRC(Sec 5) performance on validation neurons, and run the tests on test neurons. For all evaluations used neuron activations after the activation function (i.e. softmax/sigmoid).
For the Language Model evaluations used subset of 204,800 tokens from OpenWebTex, (Gokaslan et al., 2019) D. We evaluated final layer neurons that directly predict next-token, so their explanation was a specific token, which allowed us the extract ground truth directly from the text. We focused on the neurons predicting 500 most common tokens to reduce computational cost and to make sure we focus on concepts where there are enough positive labels available for statistical significance.
For (aferr avg pool) neurons we defined the "correct" concept ty as the concept that maximizes IoU with a 0.005 similar to (Bau al., 2017), using the class(and superclass) labels of the dataset as c.. For these layers we fixed 0.005 for all metrics as that was used to determine the "ground truth".
Interestingly, we find that more methods pass the tests on the CUB dataset than the other datasets we looked at, but the trends in terms of which metrics perform worse are still similar. We believe this is caused by data imbalance, as the concepts CUB are relatively balanced (following (Koh et al., 2020) we only keep concepts that are present on at least 5% of the inputs), while for example ImageNet classes are much more imbalanced (each class is positive on 0.1% of the inputs). This confirmed by our theoretical observations in Section C, which show that poor metrics are much more likely fail the test when the concepts are imbalanced.
37

---

Evaluating Neuron Explanations: A Unified Framework with Sanity Checks
Setting ViT-B-16 final layer neurons
Setting 2: Resmet-50 layer4 neurons
Missing Labels Test
Extra Labels Test
Missing Labels Test
Extra Labels Test
Metric
Decrease Ace
Decrease Acc
Decrease Acc
Decrease Ace
Recall
95,94%
0.00%
100.00%
0.00%
Precision
46.09%
99.62%
47.33%
100.00%
F1-score
35.64%
99.70%
100.00%
100.00%
loU
95.34%
99.62%
100.00%
100.00%
Accuracy
0.00%
63.08%
0.00%
71.07% %
Balanced Acc.
95.86%
23.98%
100.00%
21.12%
Inverse Balanced Acc.
45.56
99.62%
46.35%
100.00%
AUC
95,34%
30.23%
87.41 %
49.02 %
Inverse AUC
21.35%
99.92%
46.61
100.00%
Correlation
99.92%
99.92%
100.00%
100.00%
Correlation(T&&H)
98.12%
44.96%
85.05%
50.51 %
Spearman Correlation
53,91% %
38.05%
56.89%
38.08%
Spearman Correlation( RR)
93.61%
50.75%
67.47%
48.72%
Cosine
100.00%
99.85%
100.00%
97.84 %
WPMI
95, 95.94%
99.70%
86.64%
100.00%
MAD
49,47 19,47%
99.70%
46.30%
100.00%
AUPRC
99.25%
99.62%
99.23%
100.00%
Inverse AUPRC
100.00%
99.62%
98.05%
98.05%
Table F.I. Detailed results of Missing/Extra Labels test on Setting and 2.
Setting 3: RN18(Places365) final layer
Setting 4: RNI88PPaces3655) layer4
Missing Labels Test
Extra Labels Test
Missing Labels Test
Extra Labels Test
Metric
Decrease Ace
Decrease Acc
Decrease Ace
Decrease Acc
Recall
95.68%
0.00%
100.00%
0.00%
Precision
48.13%
98.85%
45.38%
100.00%
F1-score
56.77%
98.85%
99.79%
100.00%
IoU
56.77%
98.85%
99.79%
100.00%
Accuracy
0.00%
100.00%
0.00%
100.00%
Balanced Ace.
95.68%
99.42%
100.00%
65.71 %
Inverse Balanced Acc.
44.67%
98.85%
45.17%
100.00%
AUC
99, 99.14%
59.37%
87.89%
51.33% %
Inverse AUC
41 550%
100.00%
50.31
100.00%
Correlation
99. 99.14%
99.42%
100.00%
100.00%
Correlation
95.10%
40%70%
83.16%
51.33 %
Spearman Correlation
60.23%
37.46%
64.07% %
44.76%
Spearman Correlation(T&&))
89.91%
51.59%
70.43%
50.51 %
Cosine
99, 99.42%
99.42%
100.00%
99.79% 79%
WPMI
95.68%
0.00%
91.17%
100.00%
MAD
46.97%
99.71%
45.38%
100.00%
AUPRC
84.44%
97.98%
97.13%
100.00%
Inverse AUPRC
100.00%
99.71%
99.18%
100.00%
Table F.2. Detailed results of Missing/Extra Labels test on Setting 3 and 4.
38

---

Evaluating Neuron Explanations: A Unified Framework with Sanity Checks
Setting 5: CUB200 CBM neurons
Setting 6: CUB200 Linear Probe
Missing Labels Test
Extra Labels Test
Missing Labels Test
Extra Labels Test
Metric
Decrease Ace
Decrease Acc
Decrease Acc
Decrease Acc
Recall
100.00%
0.00%
100.00%
0.00%
Precision
38.32%
100.00%
43.93%
100.00%
F1-score
100.00%
100.00%
100.00%
100.00%
IoU
100.00%
100.00%
100.00%
100.00%
Accuracy
86.92%
100.00%
100.00%
95.33% %
Balanced Acc.
100.00%
100.00%
100.00%
100.00%
Inverse Balanced Acc.
100.00%
100.00%
98.13%
100.00%
AUC
100.00%
100.00%
100.00%
90.65%
Inverse AUC
100.00%
100.00%
100.00%
100.00%
Correlation
100.00%
100.00%
100.00%
100.00%
Correlation(T&&H)
99. 93.0%% %
93.46%
08.13%
94.39% %
Spearman Correlation
100.00%
90.65%
100.00%
87.85% %
Spearman Correlation(TSs &R)
93. 93.46%
89.72%
95.33%
85.05% %
Cosine
100.00%
100.00%
100.00%
97.20%
WPMI
100.00%
91.59%
100.00%
79.44%
MAD
95.33%
99.07%
96.26%
96.26%
AUPRC
100.00%
100.00%
100.00%
100.00%
Inverse AUPRC
100.00%
93.46%
100.00%
73.83%
Table F.3. Detailed results of Missing/Extra Labels test on Setting 5 and 6.
Setting 7: GPT-2-small final layer
Setting 8: GPT-2-XL final layer
Missing Labels Test
Extra Labels Test
Missing Labels Test
Extra Labels Test
Metric
Decrease Ace
Decrease Acc
Decrease Ace
Decrease Acc
Recall
98.1 11%
0.00%
99.58%
0.00%
Precision
47.16%
100.00%
49.47%
100.00%
F1-score
99.37%
100.00%
97.89%
100.00%
IoU
99.16%
100.00%
97.89%
100.00%
Accuracy
1.47%
16.63%
1.89%
16.84%
Balanced Ace.
98.11 11%
9.68%
99.58%
9.47% %
Inverse Balanced Acc.
45.47%
100.00%
48.84%
99.16%
AUC
93.89%
43.79%
96.00%
49.05%
Inverse AUC
34.11%
100.00%
28.63%
100.00%
Correlation
97.47%
100.00%
98. 74%
100.00%
Correlation
69.26%
48.00%
74.74%
50.74%
Spearman Correlation
38.53%
26.95%
38.74%
29.89% %
Spearman Correlation(T&&))
65.05%
52.21 %
65.05%
49.89 %
Cosine
97. 97.47%
100.00%
98.74%
100.00%
WPMI
98. 11%
0.00%
99.58%
0.00%
MAD
49.68
100.00%
49.05%
100.00%
AUPRC
91.16%
98.32%
93.68%
99.79%
Inverse AUPRC
97.68%
100.00%
98.32%
100.00%
Table F.4. Detailed results of Missing/Extra Labels test on Setting 7 and 8.
39

---

Evaluating Neuron Explanations: A Unified Framework with Sanity Checks
F.2. Meta-AUPRC evaluation:
Experimental Setup Details:
We evaluate our meta-AUPRC test described in Section on 10 different settings:
Setting 1:
Dataset D=lmageNet, ViT-B-16(ImageNett) 1000 final layer neurons, ground truth &
Setting 2:
Dataset D=ImageNet, ViT-B-I6((mageeNe)), 1000 final layer neurons, SigLIP Ct
Setting 3:
Dataset D=ImageNet, ViT-B-16(ImageNet)) 1400 final layer neuronstsuperclas neurons, ground truth Cp
Setting 4: Dataset D=lmageNet, ViT-B-16(ImageNee), 1400 final layer neurons+superclas neurons, SigL.P
Setting 5: Dataset D=Places365, ResNet-18(Pacces365)) 365 final layer neurons, ground truth &r
Setting 6: Dataset D= ResNee-18(Places366) 365 final layer neurons, SigLIP
Setting 7: Dataset D=CUB200, CBM trained on CUB200, 112 concept neurons, ground truth Cz.
Setting 8: Dataset D=CUB200, CLIP ViT-B-32 image encoder, linear probe trained to detect 112 CUB-concepts,
ground truth C
Setting 9: Dataset D=0penWebTTex((ubbee)) GPT-2-small,, final(prediction) layer neurons corresponding 500 most common tokens.
Setting 10: Dataset D=0penWeeTex((subxet)) GPT-2-XL, final(prediction) layer neurons corresponding 500 most common tokens.
SigLIP ct indicates we used Pscudo-labels generated from SigLIP (VIT-S0400M-144 14-g4-0-M (Zhai et al., 2023) as done by (Oikarinen & Weng. 2024). For all metrics and evaluations we choose hyperparameters such as by finding the one with best performance validation neurons (random subset of 5% of the neurons), and use those hyperparameters evaluate test neurons.
Tables F.5, F.6 and F.7 show the detailed results of our AUPRC evaluation experiment. While there is some differences on which metrics perform well on different setups, the overall trends are quite consistent, with Correlation and Cosine outperforming others on almost all settings, and AUPRC being the third best. Spearman Correlation also consistently performed the worst.
40

---

Evaluating Neuron Explanations: A Unified Framework with Sanity Checks
Setting 1: gt c,
Setting 2: SigLIP
Setting 3: gt qi
Setting 4: SigLIP
original
original K and C
original-superclass
original-superelass
K and C
and
K and C
Metric
AUPRC
Rank
AUPRC
Rank
AUPRC
Rank
AUPRC
Rank
Recall
0.9989
0.8638
12
0.0832
16
0.5758
12
Precision
0.9989
0.9640
6
0.8592
6
0.6428
9
F1-score/loU
0.9989
0.9477
0.8759
0.6992
3
Accuracy
0.9989
0.8497
13
0.7858
8
0.5853
11
Balanced Acc.
0.9989
0.9032
0,8610 11
0.7463
10
0.6158
10
Inverse Balanced Acc.
0,9989
0.9664
0.6474
8
AUC
0.9974
0.9438
8
0.7139
11
0.6933
5
Inverse AUC
0.9415
15
0.8452
14
0.8120
0.5439
13
Correlation
0.9998
0.9739 2
0.9399 2
0.7027 12
2
Correlation(T&&
0.9993
0.7257
0.4365 15
0.4906
Spearman Correlation
0.0023
17
0.0213
0.0125 17
0.0047 17
17
Spearman Correlation(T&R)
0.6533
16
0.4100
0.1783 16
0.2395 14
15
Cosine
0.9998
0.9733 2
0,9037
0.6948 2
4
WPMI
0.9989
0.9211
0,7570
0,6881
6
MAD
0.9994
4
0.9716
4
0,1866
13
0.1268
16
AUPRC
0.9989
0.9866
1
0.8839
3
0.7511
1
Inverse AUPRC
0.9999
0.9203
10
0.1131
15
0.6688
7
Table F.5. Meta-AUPRC evaluation results on ImageNet Settings(1-9)
Setting gl er.
Setting SigLIP et
Setting 7: gt es.
Setting gl c
(osseet18(Places)
Pesnee-18(Plaeeee)
CBM(CUB200)
Linear Probe(CUB200)
Metric
AUPRC
Rank
AUPRC
Rank
AUPRC
Rank
AUPRC
Rank
Recall
0.9639
12
0.7623
0.3958
17
0.1078
17
Precision
0,9640
6
0.7771
8
0.6803
9
0.2095
12
F1-score/loU
0.9640
6
0.7916
0.6386
12
0.2792
Accuracy
0.9640
6
0.6490
14
0.5853
13
0.2239
10
Balanced Acc. Inverse Balanced Acc.
0.9640
6
0.7340
11
0.7165
7
0.3186
4
0.9640
0.7813
0.6628
0.2608
AUC
0,9418
14
0.7205
12
0.7045
0.2031
14
Inverse AUC
0.8927
0.6877 15
0.8956
0.3745
Correlation
0.9693
1
0,8326
0.8883
3
0.4683
1
Correlation(T&&H)
0,9662
4
0.5326
15
0.7170
0.2976
Spearman Correlation
0.0213
17
0.1214
0.4233
0.2185
Spearman Correlation(&&RR
0.7424
16
0.4005
16
0.4243
14
0.1607
15
Cosine
0.9693
0.8304 1
0.8985
0.4067
2
WPMI
0.9655
0.7539
0.7323
0.2089
MAD
0,9590
13
0.8321
3
0.7760
4
0.2432
9
AUPRC
0.9640
6
0.8669
0.6712
10
0.2981
5
Inverse AUPRC
0.9678
3
0.8006
0.3975
0.1302
Table E6. Meta-AUPRC evaluation results on Places and CUB models Scttings(5-8)

---

Evaluating Neuron Explanations: A Unified Framework with Sanity Checks
Setting 9: gt
Setting 10: gt ct
GPT2-small
GPT2-XL
Metric
AUPRC
Rank
AUPRC
Rank
Recall
0.9754
6
0.9951
Precision
0.9655
8
0.9780
F1-score/IoU
0.9569
9
0.9875
Accuracy
0.7395
13
0.8334
14
Balanced Acc.
0,9842
0.9973 4
Inverse Balanced Acc.
0,9658
0.9782
AUC
0.8139
12
0.9194
12
Inverse AUC
0.7282
14
0.8481
13
Correlation
0.9918
0.9986
Correlation(T&
0.6705
0.7695
15
Spearman Correlation
0.0137
0.0142
17
Spearman Correlation(T&RR
0.0917
16
0.1168
16
Cosine
0.9912
0.9985 2
WPMI
0.9773
0.9958 5
MAD
0,9014
11
0.9557
11
AUPRC
0.9880
3
0.9951
Inverse AUPRC
0.9314
10
0.9780
Table F7. MctR-AUPRC evaluation results Language Models Scttings(9-10),
42


Reviews:
[{"review_id": "7O5vEUMS4M", "reviewer": "Reviewer_dD28", "summary": "This paper focuses on evaluating neuron-level explanations in deep learning models, particularly in the context of mechanistic interpretability. While many existing methods generate textual explanations for individual neurons, a critical challenge remains: how to assess the quality and reliability of these explanations. To address this issue, the authors introduce NeuronEval, a unified mathematical framework that systematically organizes and compares 18 different evaluation metrics used in previous studies.\n\n## update after rebuttal", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the review! \n\nPlease see https://drive.google.com/file/d/1OHMxyMW1KVIzxd2Rd_Hx34qIjVecUVmo/view for our new experimental results, in particular Tables G1-G4 as those experiments were conducted to address your questions. Below we address your concerns and questions in detail.\n\n> **Weakness 1 - Lacks analysis of some metrics that failed the tests**\n\nWe would like to clarify that in Appendix C of the manuscript, we have conducted extensive theoretical analysis on why metrics fail the tests; and in our motivating example (Sec 4.1, table 2, Figure B.1) we showed why recall and precision fail the tests. \n\nIn particular, failing the extra labels test is caused by the metric not being able to distinguish between the correct concept (pet) and a superclass (animal), while failing the missing labels test is caused by the metric not being able to differentiate the correct concept (pet) from a subclass (dog/cat). Overall, our tests are a mathematically formalized measurement of whether a certain metric can differentiate between a correct concept vs sub/superclass of it. For many metrics, this only happens on imbalanced neurons/concepts that do not activate on most inputs as we show in tables C1-C4, which indicates failure is tied to poor metric performance on imbalanced data. Please let us know if you had a specific kind of analysis in mind that you would like to see.\n\n> **Weakness 2 - Combinations of Metrics**\n\nThank you for the suggestion! We would like to point out that our current result (Table 3, 4) already contains a combination of metrics, since F1-score is the harmonic mean of Recall and Precision: $F1 = 2/(recall^{-1} + precision^{-1})$. In the original manuscript, we also briefly discuss the idea of combining different metrics on lines 900-901 in the Appendix. \n\nFollowing your comment, we have conducted additional experiments evaluating more extensively whether combinations of metrics can work as a good evaluation metric. Specifically, inspired by F1-score, we used the harmonic mean of other existing metrics to see how they perform. Full results are shown in Tables G1-G4 in [our new results](https://drive.google.com/file/d/1OHMxyMW1KVIzxd2Rd_Hx34qIjVecUVmo/view). We can see that many combinations of metrics achieve quite a good performance, and now pass the theoretical missing/extra labels tests. For example the harmonic mean of Balanced Acc and Inverse Balanced Acc performed well. In general, our initial results indicate that combining a metric that passes the extra labels test with a metric that passes the missing labels test will pass both tests. Conversely, combining two metrics that fail the same test, such as Recall and AUC will still fail that test.\n \n> **Weakness 3 - Section 3: Evaluation Metrics can be classified**\n\nWe are not sure which specific part the reviewer is referring to. Do you refer to framing neuron explanation as a binary classification problem in line 114? If so, we have discussed the details in Appendix A.2. We would be happy to discuss further if you have specific questions. \n\n> **Weakness 4 - Intermediate layer explanations**\n\nWe believe there might be some misunderstanding, our results and contributions are not limited to final layer neurons. Our proposed NeuronEval framework in Sec 3 and the theoretical missing/extra labels test (Tab. 3, Fig. 2, App. C) cover explanations of *all* neurons \nregardless of where they are, including hidden layers neurons, final layer neurons and even non-neuron units like directions in activation space.\nIn addition, we did show intermediate or non final-layer results in our experiments, including:\n - intermediate layer neurons (settings 2 & 4, Tab F1-F2), \n - concept neurons in an intermediate layer of a concept bottleneck model (setting 5 in Tab. F3 & setting 7 in Tab. F.6) \n - linear probes trained on hidden layer representations (setting 6 in Tab. F3 and setting 8 in Tab. F6).\n\nOur main text results (Table 3, Table 4) are averaged across settings and contain these results. \n\n> **Weakness 5 - “Input-based explanations” are vague** \n\nThank you for pointing this out. We will change the term to *Input-based neuron explanations* to avoid confusion.\n\n> **Theoretical claims - the theoretical part has some shortcomings.**\n\nCould the reviewer expand on what specific shortcomings in the theoretical part are? We would be happy to discuss and clarify further if you have any specific comments on shortcomings. \n\n> **Summary**\n\nWe believe that we have addressed your major concerns by clarifying our theoretical analysis and experiment results in the intermediate layer neurons (Weakness 1, 4), running additional experiments on combined metrics (Weakness 2) and clarifying our terminology (Weakness 5). We had a hard time understanding a few concerns (Weakness 3, Theoretical Claims), and we hope the reviewer can clarify these if concerns still remain. Otherwise, we hope you consider adjusting the rating if our response has addressed your concerns.", "claims_and_evidence": "\"Input-based explanations\" are vague. Some attribution-based methods also explain the input, but they are not related to neuron explanations.", "methods_and_evaluation_criteria": "The assessment methods used are reasonable.", "theoretical_claims": "The theoretical part has some shortcomings.", "experimental_designs_or_analyses": "The experiments designed under the proposed framework are reasonable.", "relation_to_broader_scientific_literature": "This framework has certain implications for the evaluation of existing black-box explanation work.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "**Strengths：**\n\n1. The idea of using a unified theoretical framework to evaluate metrics is novel and meaningful, and this idea will have a profound impact on the design of future benchmarks.\n\n2. The selected evaluation metrics are sufficient and can reflect the applicability of the framework.\n\n**Weakness:**\n\n1. In terms of experimental analysis, the paper lacks analysis of some evaluation metrics that have not passed the corresponding tests. Such analysis can help the reaction indicators evaluate the rationality of the experiment.\n2. The paper discusses the value of a single metric, but existing work generally considers the performance of a combination of several metrics. Therefore, I think the paper needs to add some experiments to study how the combined metric fits into the proposed theoretical framework.\n3. In the author's description in Section 3, we can see that the author has vaguely seen that the evaluation metrics can be classified, but the experimental results are only briefly mentioned without further analysis.\n4. Neuro Explanation seems to be lacking if it does not consider the evaluation of the explanation of the parameters of the neurons in the intermediate layers, but only considers the explanation of the neurons in the last layer.\n5. \"Input-based explanations\" are vague. Some attribution-based methods also explain the input, but they are not related to neuron explanations.", "other_comments_or_suggestions": "Please see weaknesses, I am willing to raise my score if the author can address my concerns convincingly.", "questions_for_authors": "NA", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "53NP3HCXu6", "reviewer": "Reviewer_7Dju", "summary": "The paper proposes NeuronEval, a unified meta-evaluation formalism for assessing neuron explanation evaluation metrics. It reformulates 19 commonly used metrics under a shared mathematical notation. The authors assess the reliability of these metrics using two diagnostic tests—missing labels and extra labels—and analyze which metrics pass these tests theoretically, empirically, and on neurons with known concepts across different models and modalities. The analysis identifies a subset of metrics that consistently align with neuron-concept correspondence.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the detailed review! We have conducted extensive additional experiments available at: https://drive.google.com/file/d/1OHMxyMW1KVIzxd2Rd_Hx34qIjVecUVmo/view, see in particular Tab G5-G7 as these experiments were conducted to address your questions.\n\n**1.Realistic failure modes**\n\nWe argue that most real world explanation failures are closely connected to one of two failure modes, and that these failure modes are directly captured by our tests:\n - FM-A) Explanation **too generic**: This means the explanation concept is a superset of the “true” neuron concept, e.g. describing a neuron as “animals” when it only activates on dogs. Our Extra Labels Test captures whether a metric can detect this failure mode.\n- FM-B) Explanation **too specific**: Explanation concept is a subset of real neuron activations, i.e. describing a neuron as “black cat” when it activates on all cats. Our Missing Labels Test captures whether a metric can detect this failure mode.\n\nAlso, our idea of a “concept” is very general and includes any text-based description. This means a single concept could be highly specific (e.g. “flying bird”), or a composition of simpler concepts (e.g. “water OR river”). The realistic failure modes you discussed then fit into the above two failure modes:\n- **Polysemanticity:** A popular model of polysemanticity is to model neuron activations as an OR of different concepts. If the explanation only captures one of these concepts, this means the explanation is too specific (FM-B).\n- **Context specific activations:** A context specific neuron activation means the neuron’s “true” concept is a subset of the non-context specific concept. If the explanation is not-context specific, the explanation is too generic(FM-A).\n- **Non activating inputs still contain the concept:** This means that the explanation concept is too generic i.e. a superset of the “true” concept(FM-A).\n\n**2. Random vs semantic sub/supersets**\n\nThanks for the suggestion! Our missing/extra labels tests are a mathematical model of semantic sub/superclasses that can be run without knowledge of the semantics or relationships between concepts. To test this model, we conducted a version of extra labels test on the ImageNet dataset where instead of randomly sampling $c^+$ we used the smallest superclass of the concept according to WordNet hierarchy, and a version of missing labels test where we used the largest subclass of the concept as $c^-$. As shown in new Table G.5, the results are essentially identical to using random sub/supersets, showcasing our random sub/superset is a good model of real semantic relationships for our purposes.\n\n**3. Importance of Epsilon Choice:**\n\nTo clarify, as mentioned on lines 307-309, we normalize each metric so that their values lie in [0,1] before running the tests to ensure fair comparison. To measure the sensitivity of our results to the choice of eps, we ran our tests with different epsilons as shown in Tab G.7. We can see that changing eps by an order of magnitude does not significantly change which metrics pass.\n\nOur theoretical results also suggest our tests are robust to choice of $\\epsilon$. As we show in Tab. C.4 & D.4, the score difference of metrics that fail our tests approaches 0 as the concept/neuron activation frequency $\\gamma$ approaches 0, while passing metrics retain a nonzero score difference regardless of $\\gamma$. This means that for any $\\epsilon > 0$ there exists $\\gamma > 0$ s.t. metrics like accuracy will fail the tests.\n\n**4. Lack of expectation over $c^{-}_{t}$:**\n\nGreat point. We will add an expectation over $c_t^{-}$ in Eq. 10 as this is a more principled definition. While our experimental results only used one sample of $c_t^-$, we conducted a study on how the results change with additional samples as shown in new Tab G.6. We can see averaging over multiple samples has little impact on the results, likely because we already average over a large number of neurons and settings.\n\n**5. Number of neurons**\n\nThanks for pointing this out. For the theoretical missing/extra labels test, we simulate 1k neurons as mentioned in line 1129. For empirical Missing/Extra labels test, we evaluated a total of 5549 neurons, and for meta-AUPRC we evaluated a total of 2989 unique neurons.\n\n**6. Source of $c_t$**\n\nWe would like to clarify we have specified the source of $c_t$ in the manuscript. For missing/extra labels tests, we use ground truth $c_t$, i.e. from a labeled dataset(line 301-302). For meta-AUPRC, we test with both ground truth $c_t$ from a labeled dataset as well as pseudo-labels from SigLIP(App. F.2, lines 1764-1782).\n\n**7. Additional references**\n\nThanks, we’ll cite and discuss these references.\n\n**8. Performance on different modalities**\n\nAs our main contributions are mathematical features of the metric itself and not tied to any particular model or modality, we did not observe significant differences between the modalities.\n\nPlease let us know if you have any remaining questions!", "claims_and_evidence": "Overall, the paper and supplementary materials present a comprehensive theoretical framework, supported by empirical and theoretical evaluations. The unification of evaluation metrics is well-supported. However, claims about metric reliability are weakened by concerns about the design of the sanity tests and fixed parameter choices, which limit the generality of the conclusions. Please see more details below.", "methods_and_evaluation_criteria": "The unification of evaluation metrics under NeuronEval is a valuable and timely contribution. However, the Missing and Extra Labels sanity tests rely on simplistic manipulations of concept labels (random removal or addition) that do not reflect realistic explanation failures, such as semantic drift, polysemanticity, or context-specific activations. Moreover, the use of a fixed epsilon threshold (0.001) ignores their differing scales and sensitivities, potentially skewing the results.", "theoretical_claims": "- Eq. 9 lacks an expectation over the random subset c^{-}_{t}, which would be statistically more rigorous, as the sanity test result would otherwise depend on a single random choice\n- The effect of the epsilon threshold (0.001) is not analyzed, even though it directly affects whether a metric passes or fails the tests. Metrics with small score ranges may unfairly fail.\n- Failure modes are limited to Missing and Extra Labels; other important failures like contextual specificity, polysemanticity, compositions of concepts are not explored. Thus, while the breadth of experiments is good, the design omits key real-world neuron-concept dynamics, limiting the validity of conclusions.", "experimental_designs_or_analyses": "Overall, the experimental section is comprehensive, covering both theoretical and empirical analyses across neurons from different modalities. However, some key information is missing:\n- The total number of neurons analyzed has not been reported.\n- The source of concept labels c_t​ is unspecified—whether human-annotated, model-generated, or otherwise.\n- Both omissions limit the ability to assess the validity and reproducibility of the findings", "relation_to_broader_scientific_literature": "The paper contributes to the growing literature on mechanistic interpretability and neuron explanation evaluation", "essential_references_not_discussed": "In Section 5, it would be useful to refer to existing analyses of neurons with ground-truth descriptions, such as [1, 2].\n\n[1] Schwettmann et al, \"FIND: A Function Description Benchmark for Evaluating Interpretability Methods\", 2023.\n\n[2] Shaham et al, \"A Multimodal Automated Interpretability Agent\", 2025.", "other_strengths_and_weaknesses": "None", "other_comments_or_suggestions": "None", "questions_for_authors": "- How many neurons were used in total in each of the experiments?\n- How was the labels c_t generated in the experiments?\n- How sensitive are the results to the choice of epsilon? Have you tested different epsilon values?\n- Why model incorrect concepts as random subsets/supersets? Did you consider more semantically plausible alternatives (e.g., polysemanticity, partial concepts)?\n- How would your tests handle neurons that correctly activate only for specific aspects of a concept (e.g., \"flying bird\" vs. \"bird\")?\n- Did you explore cases where non-activating inputs may still contain the concept?\n- Did you analyze emperically if some metrics perform better in certain modalities (e.g., vision vs. language), and if so, how do you plan to address these differences?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "X6Hjy4fOme", "reviewer": "Reviewer_3aXR", "summary": "This paper proposes NeuronEval for the meta-evaluation of input-based explanation metrics. Given a textual explanation of an input (and resultant activation), a variety of metrics have been proposed to evaluate how faithfully the description describes the neuron (or “any scalar function of network inputs”). NeuronEval unifies 19 existing explanation evaluation methods (and 18 different metrics) under 1 framework, and characterises differences between them by varying the metric, the source of the concept vector, granularity of activation vectors, and probing dataset. Motivated by the intuition that a reliable metric should be “minimal” (specific) and “exact” (not too general), they propose 2 necessary (but not sufficient) sanity checks: the missing labels test and extra labels test. They find that most metrics fail these tests (lack in sensitivity and fidelity); point out which metrics reliably pass the tests; provide guidelines on reliability desiderata for future metrics in explanation evaluation.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed review and positive feedback!\n\n**Re: Additional Related Work:**\n\nThank you for pointing out these references. While these references focus on a very different type of XAI, in particular local input-importance evaluations and as such cannot be applied to our setting, they contain important ideas about reliable evaluation of explanations and we will cite and discuss them in the related work(Appendix B.4) as follows:\n\nThe field of input-importance explanations has seen an evolution in the metrics used, with initial focus on finding the features that humans think are important. Later metrics such as deletion and insertion proposed by [1] allow for more principled evaluation of the explanation fidelity, i.e. whether it actually matches what the model in vision models, and [2] extends the deletion metric to natural language settings. [3] proposes the Remove And Retrain (ROAR) framework as an alternative method for evaluating the quality of input-importance explanations by evaluating whether a model retrained on data without the most important pixels can still solve the task. [4] propose additional checks, such as measuring whether an explanation method has high sensitivity, with the intuition that similar inputs should have similar explanations.\n\nOverall, we think that this line of research [1]-[4] does not reduce the novelty of our contribution, as they are focused on a different problem and produce no actionable insight for our setting of global neuron-level explanations. In addition, these papers are mostly focused on introducing new evaluation metrics, instead of conducting meta-evaluation between existing metrics which is the focus of our work.  \n\n**Additional Experiments:**\n\nIn response to other reviewer's comments, we have conducted additional experiments such as evaluating novel combinations of metrics, showing the robustness of our results to the choice of $\\epsilon$, and showcasing that our results using a random subset as $c_t^-$ are similar to using a real semantic subset. We have also included a new figure showcasing the Missing and Extra Labels tests. These results are available at https://drive.google.com/file/d/1OHMxyMW1KVIzxd2Rd_Hx34qIjVecUVmo/view.\n\nPlease let us know if you have additional questions and we would be happy to discuss them further! If not, we hope you consider updating your score as you have noted our work is comprehensive, clarifying/unifying and a worthwhile contribution to interpretability research with little weaknesses.\n\n\n\n**References:**\n\n[1] Vitali Petsiuk, Abir Das, and Kate Saenko. RISE: randomized input sampling for explanation of black-box models. In BMVC, pp. 151. BMVA Press, 2018. URL http://bmvc2018.org/contents/papers/1064.pdf.\n\n[2] Thomas Fel, Rémi Cadène, Mathieu Chalvidal, Matthieu Cord, David Vigouroux, and Thomas Serre. Look at the variance! efficient black-box explanations with sobol-based sensitivity analysis. Advances in neural information processing systems, 34:26005–26014, 2021.\n\n[3] Sara Hooker, Dumitru Erhan, Pieter-Jan Kindermans, and Been Kim. A benchmark for interpretability methods in deep neural networks. NeurIPS, 32, 2019.\n\n[4] Umang Bhatt, Adrian Weller, and José MF Moura. Evaluating and aggregating feature-based model explanations. In Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence, pp. 3016–3022, 2021.", "claims_and_evidence": "The claims in this work are that meta-evaluation (of XAI evaluation metrics) is necessary to compare and contrast their reliability, to guide the development and usage of more reliable metrics. Such claims are supported by the introduction and application of NeuronEval, a framework which assesses the reliability, sensitivity and robustness of 18 metrics (across 19 methods) through sanity checks of missing and extra labels. The authors present evidence that the majority of metrics fail or perform suboptimally on one/both tests, which speaks to the importance of this investigation.", "methods_and_evaluation_criteria": "This paper proposes NeuronEval, a framework which unifies major XAI evaluation metrics via plug-and-play, as well as 2 associated sanity checks for the fidelity and sensitivity of various metrics. The framework is sufficiently general and expressive; the sanity checks are well-motivated and consistent with existing intuition that optimal explanations should be minimal and descriptive of the input-activation pair, and that faithful evaluation metrics (of these explanations) should be sensitive to changes in model response. The proposed framework and sanity checks are applicable for the task of meta-evaluating the faithfulness of XAI metrics.", "theoretical_claims": "N/A: there are no theoretical claims in this work.", "experimental_designs_or_analyses": "The experimental details (e.g. specifics of metrics, concept vector, activation vector, probing dataset) for meta-evaluation are expounded upon throughout the paper and supplement. The authors elaborate on how NeuronEval admits different modalities (image and text), network units (e.g. neuron, channel, scalar function of inputs, SAE features, CBM neurons, linear probes), different explanation methods and metrics – I find their descriptions clear and sufficient. They further elaborate on specific setups for the missing/extra labels sanity checks, and discuss 5 possible outcomes and provide hypotheses for failure cases (e.g. concept imbalance, biassed sampling, using generative models in the evaluation pipeline). I find their experimental design reasonable and sound.", "relation_to_broader_scientific_literature": "This work addresses an important issue in interpretability research: the lack of a structured framework / unified basis to evaluate and compare existing methods. It engages well both with established work in explainable AI (XAI) and newer work in mechanistic interpretability. The framework is quite general and contributes to a unified understanding of XAI: it is compatible with 19 different input-based explanation techniques, 18 metrics; it evaluates explanations of network “units”, from a single neuron, single channel to scalar functions of network inputs, e.g. sparse autoencoder (SAE) features, concept bottleneck model (CBM) neurons, linear probes.", "essential_references_not_discussed": "This paper could more deeply with previous work in XAI robustness analysis – RISE [1], Sobol [2], ROAR [3], MuFidelity [4] – which have exhibited similar intuitions regarding the desired sensitivity of XAI methods to perturbations.  \n  \n[1] Vitali Petsiuk, Abir Das, and Kate Saenko. RISE: randomized input sampling for explanation of black-box models. In BMVC, pp. 151. BMVA Press, 2018. URL http://bmvc2018.org/ contents/papers/1064.pdf.   \n[2] Thomas Fel, Rémi Cadène, Mathieu Chalvidal, Matthieu Cord, David Vigouroux, and Thomas Serre. Look at the variance! efficient black-box explanations with sobol-based sensitivity analysis. Advances in neural information processing systems, 34:26005–26014, 2021.  \n[3] Sara Hooker, Dumitru Erhan, Pieter-Jan Kindermans, and Been Kim. A benchmark for interpretability methods in deep neural networks. NeurIPS, 32, 2019.  \n[4] Umang Bhatt, Adrian Weller, and José MF Moura. Evaluating and aggregating feature-based model explanations. In Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence, pp. 3016–3022, 2021.", "other_strengths_and_weaknesses": "In other sections, I have elaborated at length on the positives of this work. Regarding negatives, I find the originality of meta-evaluating XAI methods and metrics based on their sensitivity and faithfulness interesting but not completely novel (see “essential references”). That said, this work is comprehensive, clarifying/unifying and a worthwhile contribution to interpretability research.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CBB3IlJuwt", "reviewer": "Reviewer_UBi6", "summary": "This paper introduces a systematic framework designed for evaluating neural explanations. The core of this framework lies in its ability to quantitatively measure how well a given explanation aligns with neuron behavior across different samples.\n\nThe framework operates on a few key components. First, for any explanation *t* (which could be textual, visual, or any other format), it defines $[c_t]_i$ as a measure that indicates the presence of concept *t* in sample *i*. This presence is potentially determined through methods like crowdsourcing. Second, for a neuron *k*, $[a_k]_i$ represents its activation value on sample *i*. The alignment between $[a_k]_i$ and $[c_t]_i$ is then framed as a binary classification task. In this task, perfect matching suggests that explanation *t* accurately describes neuron *k*'s behavior. The framework also formalizes various evaluation metrics from previous research, such as measuring the **Recall** or **IoU (Intersection over Union)** between $c_t$ and $a_k$.\n\nFurthermore, the paper introduces two meta-evaluation approaches aimed at assessing the reliability of these evaluation metrics. The first is the **Label Manipulation Test**. This test involves artificially setting $[c_t]_i$ to 0 for samples that originally contained the concept. Reliable metrics should then show decreased performance compared to the initial perfect score. Conversely, when $[c_t]_i$ is set to 1 for samples that lacked the concept, scores should similarly decrease from their initial perfect score. The paper conducts these tests both theoretically and empirically for various metrics.\n\nThe second approach is the **Neurons with Known Concepts Test**, which uses the final layer neurons of classifiers with known explanations as ground truth. The framework computes scores for all neuron-concept pairs, including both correct pairs and random pairings. The expectation is that reliable evaluation metrics should consistently yield higher scores for correct neuron-concept pairs when compared to incorrect ones.\n\nThe paper conclude by suggesting some metrices as most reliable options in their framework.\n\n## update after rebuttal\nI kept my score as autors addressed the issues I raised.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review and positive feedback! Your summary is very well written and highlights a strong understanding of our work. We would like to address your concerns below:\n\n**Weakness 1 - Labeling Cost**\n\nThis is a great point. It is true that a significant reason for the use of recall is that it is cheaper to evaluate as we only need to only evaluate $c_{ti}$ on highly activating inputs, i.e. where $a_{ki} = 1$. In contrast, evaluating most other metrics requires knowledge of the full $c_t$. One way to address this issue is to use a combination of metrics, for example evaluating F1-score by combining a crowdsourced evaluation of Recall with a generative model based evaluation of Precision as we suggest in Line 897 of the Appendix. \n\nAs you mentioned, crowdsourced evaluation of the entire $c_t$ will likely give noisy results as most inputs do not contain the concept. To avoid this, another approach is to oversample highly activating inputs similarly to Top-and-random sampling. This can be done without failing the extra labels test if the proper importance sampling correction is applied to correct for the bias introduced by the sampling. However, the effective sampling/user study design for non-recall metrics is a rather large and complex topic that does not fit within the scope of the current paper, but it is something we are actively looking into and we are confident that there are ways to evaluate the other metrics with a cost not much higher than recall evaluation. We will include discussion of this under limitations/future work.\n\n**Weakness 2 - Missing citation**\n\nThank you for pointing this out. We will cite [1] as well as other papers[2, 3] conducting similar studies based on neurons with known concepts as suggested by Reviewer 7Dju.\n\n[1] Soltani Moakhar, A., Iofinova, E., Frantar, & Alistarh, D. SPADE: Sparsity-Guided Debugging for Deep Neural Networks. ICML 2024.\n\n[2] Schwettmann et al, \"FIND: A Function Description Benchmark for Evaluating Interpretability Methods\", 2023.\n\n[3] Shaham et al, \"A Multimodal Automated Interpretability Agent\", 2025.\n\n**Weakness 3 - Limited novelty i.e. not introducing new metrics**\n\nSimilar to how you acknowledge this is as “not necessarily a weakness”, we do not think this is a weakness but a conscious decision as we thought it would be a more valuable contribution to the field to rigorously analyze existing and standard statistical metrics instead of focusing on creating new ad-hoc metrics. This way our contributions are more clearly defined, and designing metrics particularly to do well on our tests risks overfitting them to these specific tasks, or worse tuning the tests themselves in a way that our new metrics would do well.\n\n**Other comments or suggestions 1- Examples**\n\nWe have included an example with dataset inputs and concepts values Appendix Figure B.1., to improve clarity similar to your suggestion. Please let us know if you would find additional examples useful or have any additional suggestions.\n\n**Other comments or suggestions 2 - Typo**\n\nThank you for pointing this out, we have corrected the typo in Line 1636 to: “… we defined the “correct” concept $t_k$ as the …”\n\n**Additional Experiments:**\n\nIn response to other reviewer's comments, we have conducted additional experiments such as evaluating novel combinations of metrics, showing the robustness of our results to the choice of $\\epsilon$, and showcasing that our results using a random subset as $c_t^-$ are similar to using a real semantic subset. We have also included a new figure showcasing the Missing and Extra Labels tests. These results are available at https://drive.google.com/file/d/1OHMxyMW1KVIzxd2Rd_Hx34qIjVecUVmo/view.\n\n**Summary**\n\nWe are happy to hear that you find our work is addressing a crucial problem and produces logical and valuable contributions to help solve it. We believe we have addressed you main remaining concerns regarding labeling cost and missing citations and will include a discussion of these in the revised manuscript. Please let us know if you have any remaining questions and we would be happy to discuss them further.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "I checked the theoretical analysis in section D of appendix which is related to **Label Manipulation Test** Meta-evaluation. No issue found.", "experimental_designs_or_analyses": "The experiment on Meta-evaluation **Label manipulation Test** is valid. However its contribution to the paper claims is very minimal. Their Theoretical analysis can fully support those claims without the need of the expeiments.", "relation_to_broader_scientific_literature": "This paper formalized previous evaluation metrices in an unified framework.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "# Strengths\n\n1.  The paper addresses a very crucial problem. Evaluation in machine learning interpretability is currently inconsistent, with each work often employing unique, new evaluation methods. As a result comparison of different methods is very hard.\n2.  The paper's formalization is simple yet comprehensive, effectively encompassing the evaluation setups used in many other works.\n3.  The proposed meta-evaluation metrics are both logical and a valuable contribution to the field.\n4. The paper is very well written and easy to follow. \n\n# Weaknesses\n\n1.  The paper neglects to discuss the labeling costs associated with each metric, a significant consideration. For example, while the paper criticizes the use of **recall**, it's important to recognize that **recall** can reduce labeling costs because it only requires concept labels ($[c_t]_i$) for samples that highly activate a neuron. Given that calculating $c_t$ typically involves crowdsourcing or LLMs (the most expensive part of the evaluation process), **recall** offers a practical advantage. Furthermore, crowdsourcing labels for $c_t$ across *all* samples, where the label is predominantly 0 (due to concepts appearing in only a small percentage of samples), can lead to lower-quality labels due to extreme data imbalance (labelers may become inattentive and default to labeling most samples as 0).\n2.  The paper's meta-evaluation using **Neurons with Known Concepts** lacks proper citation. Soltani Moakhar et al. [1] present a very similar method for evaluating interpretability, employing the final layer of a ResNet (where neuron/concept pairs are known) for evaluation purposes. While this paper uses it for meta-evaluation (distinguishing it somewhat), it's still a relevant prior art to acknowledge.\n3.  The paper's novelty is limited. While not necessarily a weakness, a completely novel and unintuitive evaluation metric might face resistance to adoption within the field.\n\n[1]: Soltani Moakhar, A., Iofinova, E., Frantar, & Alistarh, D. SPADE: Sparsity-Guided Debugging for Deep Neural Networks. ICML 2024.", "other_comments_or_suggestions": "1. The addition of examples in Appendix F (e.g., a sample dataset input, a concept present in that input, corresponding $c_t$ and $a_k$ values for a neuron) would enhance understanding. \n\n2. Also, there appears to be a writing issue on Line 1636 in the appendix that needs to be addressed.", "questions_for_authors": ".", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper studies how to evaluate neuron explanations. The authors propose NeuronEval, a framework for the meta-evaluation of input-based explanation metrics. Given a textual explanation of an input, various metrics have been proposed to assess how faithfully the explanation captures neuron behavior. NeuronEval unifies these existing evaluation methods and characterizes their differences by varying factors such as the evaluation metric, the source of the concept vector, the granularity of activation vectors, and the probing dataset.

Overall, reviewers found the unified framework meaningful and considered this work a valuable and timely contribution to the field of interpretability.
Decision: Accept (poster)