Evaluating Neuron Interpretation Methods of NLP Models
Authors: Yimin Fan
Fahim Dalvi
Nadir Durrani
Hassan Sajjad
Venue: NeurIPS 2023 poster (2023)
Keywords: Neuron interpretation
NLP
Interpretability
Machine Learning
TL;DR: We conduct a thorough evaluation of neuron interpretation methods and propose a novel benchmarking framework with many applications.

Abstract: Neuron interpretation offers valuable insights into how knowledge is structured within a deep neural network model. While a number of neuron interpretation methods have been proposed in the literature, the field lacks a comprehensive comparison among these methods. This gap hampers progress due to the absence of standardized metrics and benchmarks. The commonly used evaluation metric has limitations, and creating ground truth annotations for neurons is impractical. Addressing these challenges, we propose an evaluation framework based on voting theory. Our hypothesis posits that neurons consistently identified by different methods carry more significant information. We rigorously assess our framework across a diverse array of neuron interpretation methods. Notable findings include: i) despite the theoretical differences among the methods, neuron ranking methods share over 60% of their rankings when identifying salient neurons, ii) the neuron interpretation methods are most sensitive to the last layer representations, iii) Probeless neuron ranking emerges as the most consistent method.

Evaluating Neuron Interpretation Methods of NLP Models
Yimin Fan 2
Nadir Durrani Hassan Sajjad* The Chinese University of Hong Kong. Hong Kong, China Qatar Computing Research Institute, HBKU, Qatar
Fahim Dalvi
Faculty Computer Science, Dalhousie University, Canada
Abstract
Neuron interpretation offers valuable insights into how knowledge is structured within a deep neural network model. While a number of neuron interpretation methods have been proposed in the literature, the field lacks comprehensive comparison among these methods. This gap hampers progress due to the absence of standardized metries and benchmarks. The commonly used evaluation metric has limitations, and creating ground truth annotations for neurons is impractical. Addressing these challenges, we propose evaluation framework based voting theory. Our hypothesis posits that neurons consistently identified by different methods carry more significant information. We rigorously assess our framework across diverse array of neuron interpretation methods. Notable findings include: i) despite the theoretical differences among the methods, neuron ranking methods share over 60% of their rankings when identifying salient neurons, ii) the neuron interpretation methods are most sensitive to the last layer representations, iii) Probeless neuron ranking emerges as the most consistent method.
Introduction
The advent of deep neural network (DNN) models and their success in Natural Language Processing (NLP) has opened new research frontier to interpret the representations learned within these models. Representation analysis provides a holistic view of the knowledge learned within the representation. Whereas neuron analysis gives insight into how knowledge is structured within the representation. To this end, an ample amount of work has been done to understand the knowledge captured within the learned representations (Belinkov al. 2017 Liu al. [019a]] Tenney et al.. 2019; Dalvi et al.) 2022) and individual neurons (Karpathy et al. 2015 Kádár et al. 2017a Dalvi et al. 2019))
Neuron analysis provides fine-gras fine-grained interpretation of the representation. It discovers salient neu- rons of the network w.r.t to a concept such as morphological or syntactic concepts (Lakretz et al. 2019 Durrani al. 2020). and provides insight into how they may be used during inference (Lundberg &
Lee| |ee| 2017 Dhamdhere et al. 2018). Beyond understanding the inner dynamics of the DNN models, neuron analysis has various potential applications such network manipulation (Bau al. 2019).
domain adaptation (Antverg et al. 2022), model distillation (Dalvi et al.) 2020) and architectural
search (Prasanna et al.) 2020).
number of neuron analysis methods have been proposed in the literature to interpret deep NLP models. While these methods have shown promising results, there is a notable absence of systematic comparisons among them in existing studies. Conducting fair comparison of neuron interpretation methods is crucial for comprehending the merits and limitations of each technique. Such comparisons
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

serve as a guiding light for the community, steering efforts towards the development of improved and more effective interpretation methodologies.
In this work, we study a variety of neuron interpretation methods under a controlled setting and provide thorough comparative analysis. There are two main challenges in comparing neuron interpretation methods: i) absence of a standard evaluation metric, ii) lack of an evaluation benchmark (Sajjad al. 2022b). The most common evaluation metric used in the neuron analysis literature is to train a classitier using the discovered neurons as features. The performance of the classifier in predicting the concept of interest serves as evidence of the quality of the discovered neurons, in other words, the correctness of the neuron interpretation method.
evaluation metric is suboptimal for two reasons: i) training classifier provides an opportunity to learn the task and it may result in good performance irrespective of the quality of the neurons used features (Hewitt & Liang] 2019), ii) the classifier an evaluation method may favor the neuron interpretation methods that are methodologically similar (for instance, methods that use classifier to discover salient neurons). [Antverg & Belinkov (2021) shed light on the latter issue and showed that the methodological similarity between the neuron interpretation methods and the evaluation methods may result in unreasonably high accuracy scores. We empirically demonstrate the issue in Section 3 Another bottleneck in comparing neuron interpretation methods the absence of evaluation benchmark. Neurons are distributive in nature and exhibit polysemy (capturing multiple concepts at the same time). Moreover, there can be different sets of neurons learning& concept, and there exists no single correct answer. therefore intricately challenging to create the ground truth annotations and infeasible to scale to large set of models and concepts.
We argue that the classifier
Given the lack of standard evaluation metric and the infeasibility of creating ground truth annota- tions, how can we compare neuron interpretation methods? ? We rely on the voting theory OOCConoor & Robertson 2003) that performs a systematic aggregation of results in order to achieve a consen- sus (Patrão Neves 2016). It is effectively used in political, social, and machine-learning communities for cases wherein there are distinct preferences among voters (models). Examples of specific applica- tions of voting theory include ranking search results obtained from various algorithms in information retrieval (Lin et al., 2017 Yilmaz et al., 2008), finding the best machine translation system using pair-wise ranking (Lapata 2006) and aggregating results of several NLP tasks (Colombo et al.) 2022).
We hypothesize that neurons that commonly discovered by different interpretation methods are more informative than others, and may serve as a signal of correctness for the selected neurons. Motivated by this, devise two voting- based compatibility metrics. Our first metric, Avgover Lap, is based on plurality voting (Cooper & Zillante| 2012). AvgOverlap computes an average overlap of the discovered neurons across all voters. Our second method, NeuronVote, inspired by Borda Count (Lippman 2012). takes into account the ranking of neurons of each method while calculating the average pair- wise intersection over union with all the other methods. The compatibility metrics provide single score for each method, where a higher score refers to a neuron interpretation method whose discovered neurons are more compatible with the other methods. We further extend the evaluation of neuron interpretation methods to pair-wise comparison, providing insights into how any two given methods relate in terms of their discovered neurons.
We conduct a comparative analysis of six neuron interpretation methods using diverse concepts consisting of morphological, semantic, and syntactic properties and using three pre-trained models. The analysis suggests that the best performing methods, despite their methodological differences, share more than 60% of the top neurons. Probeless ranking Antverg & Belinkov (2022) consistently exhibited the highest overlap with other methods. Because neurons in the last layer representations are highly correlated, they are the most challenging to interpret. Finally. we present a case study that demonstrates the usefulness of our evaluation methodology for any new method proposed in the future (Appendix 7.1 Our contributions are as follows:
We conduct the first thorough comparative analysis of a large set of neuron interpretation methods of NLP across 3 pre-trained models and using diverse set of linguistic concepts.
Our proposed evaluation methodology consisting of two compatibility metrics and a pair- wise comparison targets one of the critical limitations of the neuron interpretation studies.
We provide the evaluation methodology as a framework to facilitate future studies and comparison of methods.
2

---

2 Neuron Interpretation Methods
The goal neuron interpretation methods is to rank N neurons with respect to some concept These methods can be broadly classified into two classes: corpus-base based methods and probing- -baed methods (Sajjad et al,, 2022b). The former discovers the role of neurons by aggregating statistics neuron activations and the latter trains classifiers achieve the same.
Let D be dataset of sentences (represented as a list of tokens), where each token w appears in specific context/sentence and has contextual representation associated with it. Let 2(n, w) be the activation of the n² neuron and the wh token in D. The neuron n E - can be from any component of the original model, such as specific layer. Also note that the activation value a(n, w) in context-dependen, so the same w can result in different z(n,, w) values depending on context.
A concept C' represents property we want to discover neurons for and represents a random concept, ie. containing words that are not associated with the concept C. For example, Ccountry (Rome, Paris, London, New York, represent a concept of country names and would consist of other words in the corpus that are not country names. More formally, = D C.
Let R(n, C)) represents the score of neuron with respect to concept C. Each neuron interpretation method implements R. and provides ranked list of neurons with respect to concept. In following, present six interpretation methods studied in this paper.
2.1 Corpus-based Methods
Corpus-based methods align neurons with a concept by accumulating co-occurrence patterns between neuron activations and the presence of the concept of interest. We use two corpus-based methods in this work presented as follows.
2.1.1 Probeless
The Probeless method (Antverg & Belinkov 2021 obtains neuron rankings based on an accumulative strategy. The score of given neuron defined as follows:
R(n, = x(C) -(())
where (CC) is the average of activations =(n, w), C. u(C) is the average of activations over the random concept č. Note that the ranking for each neuron computed independently.
2.1.2 ToU Method
Mu & Andreas (2020) proposed IoU to generate compositional explanations for neurons. For each token in the dataset, they create i) a binary mask of a neuron by thresholding activations above percentile and ii) binary mask of a concept by checking its presence the token. They compute Intersection over Union (IoU) over the two masks:
R(n, C) = E. 1(2(n, w) 8) 1 C) En 1(=(n, w) > 8) V 1(w E CC
where 1 is the indicator function. =(11, w)) > 8 is binary mask over neuron activations based on binary mask, created by checking if the current word represents concept C of interest. Mu & Andreas (2020) used loU scores to generate alternative explanations for neurons on the task of image classification. Here we apply their method in the NLP domain to generate neuron ranking for any concept of interest. Like Probeless, the ranking for every neuron is computed independently of other neurons.
threshold 5. w
2.2 Probing Methods
Probing methods train a classifier using neuron activations as features for the task of predicting the concept of interest (Belinkov et al.] 2017]] Hupkes et al. 2018). The internals of the trained classifier (e.g. weights) are used rank the features (neurons)
Available in the NeuroX toolkit Dalvi et al 2023

---

2.2.1 Lasso Regularizer (L1)
Radford et al. (2019) trained linear classifier with LI regularization and used the weights of the classifier as proxy for the importance of neurons for the given concept. The loss function of the
classifier is as follows:
[()) Elog log Po(clew((w) + |101
where, E (C,C) is the correct class for w. (x) is the vector of all neuron activations for word w, z(n,, w)], and Po(c]zn(w)) is the probability that word w belongs to the
i.e. [=(0, w²), =(1, class The learned weights 0 serve the ranking of neurons. Specifically, R(n, C) is defined the absolute weights for the class i.c. [8c(.)].
LI adds the "absolute value of magnitude" of the coefficient as penalty term the L(O), which shrinks the less important weights to zero, leading to sparse weight distribution. Radford et al. (2019) used this regularization to force the classifier learn spiky weights, indicating the selection of few specialized neurons learning a concept, while setting the majority of neurons' weight to zero. Such an assumption is useful in discovering focused neurons that learned particular concept only.
2.2.2 Ridge Regularizer (I.2)
Lakretz et (2019) used 1.2 regularization train the linear classifier. The loss is as follows:
[()) =
Σ log Po(clen(w)) + 22
L.2 forces weights to be close to zero (but not zero). It is useful to deal with multicollinearrty (neurons that are highly correlated) scenarios, through constricting the coefficient while keeping all the features. Intuitively, this encourages grouping of features, thus discovering group neurons that jointly learn concept. The score R(n, C) is computed in similar fashion as the Lasso Regularizer method.
2.2.3 ElasticNet Regularizier (LCA)
Both 1.1 and 1.2 regularizations capture properties that are desirable when selecting the most important neurons. L1 facilitates sparsity, identifying focused neurons while L2 encourages identifying groups of highly correlated features into account. Dalvi et al. (2019) used ElasticNet regularization (Zou & Hastie 2005) that balances the trade-off between them.
(()) Σ
log Po(clzx(w)) + A 1111 + >2110112
X and A are hyperparameters that are tuned to optimize the effect of I.1 and 1.2 regularization.
2.2.4 Gaussian Classifier
Hennigen et al. (2020) assumes that neurons exhibit a Gaussian distribution. They fit multivariate Gaussian, say P over all neurons. Since multivariate Gaussian is decomposable by nature, they able to extract individual probes for any subset of input features without additional work. Formally, they are able to extract Py, where F N. The classifier itself trained using the Bayesian framework, specifically the maximum a posteriori estimate to compute the parameters 6. Once multivariate Gaussian is trained, the neuron selection performed in greedy fashion:
F= F), F + arg max P(C (m)) for i in 1..[A]
R(n, = [W| - inder F,
In essence, probes for individual neurons are first extracted, and the neuron probe with the highest log- likelihood is considered the most important neuron. Next, probes are extracted for pairs neurons, with one of them being the selected neuron. The pair with the highest log-likelihood then contributes the second most important neuron to the ranking. The full ranking is compared iteratively, with each step being greedy selection of the next best neuron to add based on the log ligeliiooooodd. This method makes two fundamental assumptions: i) neuron activations follow a Gaussian distribution and ii) neuron ranking can be done in a greedy fashion.

---

3 Evaluation Methodology
3.1 Classifier Accuracy
The commonly used evaluation metric i.e. classification accuracy of the selected neurons is inadequate: because it favors the neuron ranking methods that are based the probing framework (Antverg Belinkov 2021), ). ii) it is not clear whether the classifier performance is a reflection of the knowledge learned in the discovered neurons or it is due the capacity of the classifier to learn and memorize the task (Hewitt & Liang. 2019; Zhang & Bowman 2018).
We demonstrate these issues empirically, by using linear classifier without regularization as an evaluation metric to evaluate various neuron interpretation methods. Given discovered neurons with respect a concept C, we train the classifier using these discovered neurons as features. The performance of the classifier serves as a measure of the correctness of s. Table presents the average accuracy across all parts of speech (POS) concepts and across all layers of BERT for 30, 50, 70, 100. Random refers to a random selection of neurons. The details of the dataset and the experimental setup are provided in Section 4 few notable observations are: the classifier accuracy is higher for the probing methods (Lasso and Ridge) than the corpus-based methods (Probeless and loU), ii) the classifier has the capacity to memorize as a Random selection of neurons performs reasonably well when more than certain number of neurons These results complement the issues raised earlier i.c. the classifier may result in high performance irrespective of the correctness of the neurons, and it favors the interpretation methods that are methodologically similar to itself.
used in the evaluation.
Table Task: POS, Model: BERT, Accuracy scores using classifier accuracy as an evalua- tion metric. Bold and underline are the first and second best results.
Table 2: Task: POS, Model: BERT, Av- erage compatibility scores AvgOverlap and NeuronVote when selecting the top 10, 30, 50 neurons from layers 1, 6, 12. Bold and underline are the first and second best scores.
Neurons
30
70
100
AvgOverlap
NeuronVote
Probeless
0.92
0.94
0.95
0.96
Random
0.021
0.021
IoU
0.91
0.93
0.94
0.95
Gaussian
0.086
0.169
LCA
0.258
0.514
Lasso Ridge
0.93
0.95
0.95
0.96
Lasso
0.240
0.473
0.95
0.97
0.98
0.98
Ridge
0.177
0.362
Random
0.81
0.86
0.89
0.92
Probeless
0.269
0,532
IoU
0.156
0.365
3.2
Compatibility Metrics
Given the absence of good evaluation metric and gold annotations, rely on consensus-based voting theory. The consensus serves as a tool to measure the relation between rankings. In NLP. has been widely used in information retrieval to combine search results of various algorithms (Lin et al. 2017; Yilmaz et al. 2008), in machine translation to compare different systems' output using pair- wise ranking (Lapata, 2006 and to aggregate results of NLP tasks (Colombo al. 2022).
One way to compare rankings to perform pairwise comparison using rank correlation or similarity functions such as Kendall's 7 (Kendall) 1938) and Spearman correlation (Daniell 1990). These methods assume conjoint rankings where all rankings contain identical elements. the case of neuron rankings, for every neuron interpretation method, we only consider the 8 top-ranked neurons with respect to a concept. The rationale behind this setting is to minimize the noise and randomness in rankings that may occur for neurons that are not learning the concept. This results in disjoint sets and the rank correlation methods can not be directly applied in such cases (Lin et al. 2017).
Motivated by the voting theory, we devise two metrics to compare disjoint rankings. Our first metric, AvgOverlap, is based on plurality voting (Cooper & Zillante 2012). AvgOver computes the pair-wise average neuron overlap across methods. It is a set-based method and it does not consider the internal ranking neurons into consideration. For example, the first top neuron and fifth top neuron in a ranking will get equal weight when compared with another ranking. Our second method, NeuronVote, inspired by Borda Count (Lippman) 2012), considers the order of neurons into consideration in comparing rankings. We call our methods compatibility metries, which when given
5

---

neuron ranking, provide its compatibility score with respect to other methods. The metrics give high score method that is most aligned with the rankings of the other neuron interpretation methods. To build deeper understanding of how any two methods relate in terms of the resulting neurons, we also present pair-wise comparison of the discovered neurons across methods.
Formally, let Sm R1, .... n. represent the s top-ranked neurons for a method m, where is hyperparameter that be adjusted vary the number of top neurons selected for evaluation. We describe the compatibility metrics and the pair-wise comparison method as follows.
3.2.1 Average Overlap
AvgOverlap scores an interpretation method based on its average neuron overlap with other methods. We define the overlap o(m1, m2) between two methods the intersection union of their respective top-neuron sets:
|Smy A ASm!l
o(m1, m2)
Smm
Sma
AvgOverlap is defined as:
M Σ j=1,f
Avgoverlapm.
o(mi, m2)
(8)
M
where M is the set of all methods. A large value of AvgOverlap means that the ranking of that method is well aligned with the ranking of other neuron interpretation methods.
3.2.2 NeuronVote Vote
AvgOverlap treats each neuron equally in a set of discovered neurons and ignores the ranking assigned by the methods. However, intuitively neuron interpretation method whose top choice is endorsed by all other methods should get higher score than an interpretation method whose 10th top choice is endorsed by all other methods. To take this into account, we follow the Borda count strategy (Lippman 2012) and propose NeuronVote. It considers each neuron's 's ranking in the voting process. We first create Shest, an aggregated ranking based on the position of individual neurons in the rankings of various methods.
M
Sbest = argsort
inder((m,.)) for n2 E N Σ
(9)
where argsort is ascending sort function and index gives the position of neuron in the ranked list. The accumulated Shest represents an aggregation on weighted votes, with neurons appearing at the top of various methods' ranked lists getting higher weight than others. Following the notation in AvgOverlap we define NeuronVote,,. o(Sm,, Sheee). The method with high NeuronVote implies that its ranking of neurons is most endorsed by other interpretation methods.
3.3 Pairwise Comparison
The compatibility metrics provide a holistic evaluation of a neuron interpretation method with respect to other methods. We further extend the analysis to pair-wise comparison to understand how the discovered neurons of method related to another method. The pair-wise wise comparison is calculated as an intersection between the output of two methods i.c. o(m1, m2) = Sm, A Smz
4 Evaluation
4.1
Settings
We consider three 12-layered pre-trained models: BERT-base-cased (BERT, Devlin et al. 2019))
RoBERTa-base- (RoBERTa, Liu et al., 2019b) and XLMM-Roerrt-bbar (XLMR, Conneau et al. 2019). Each layer consists of 768 neurons. We consider concepts from three linguistic tasks: parts of speech tags (POS, Marcus et al.) 1993), semantic tags (SEM, Abzianidze et al. 2017) and syntactic chunking (Chunking) using CoNLL 2000 shared task dataset (Tjong Kim Sang Buchholz] 2000). Dataset details are provided in Appendix 7.2 We consider every tag as concept and identify neurons methods, we use binary classification setup where the contextualized representation of words belonging to the concept serves as positive
with respect to the concept. In the case of classifier-

---

Table 3: Task: POS, Average NeuronVote compatibility scores across concepts when selecting the top 10, 30, and 50 neurons from layers 1, 6 and 12. Bold numbers, underline numbers, and dashed numbers show the first, second, and third best scores respectively
BERT
RoBERTa
XLMR
Layers
6
12
6
12
6
12
Random
0.019
0.021
0.023
0.020
0,019
0.017
0.020
0.017
0.023
Gaussian
0.147
0.177
0.183
0.140
0.179
0.179
0.118
0.135
0.153
LCA
0.501
0.544
0.391
0.550
0.495
0.380
0.450
0.531
0.373
Lasso
0.496
0.545
0.395
0.475
0.453
0.350
0.410
0.471
0.267
Ridge
0.332
0.405
0.360
0.452
0.491
0,510
0.358
0,446
0.455
Probeless
0.497
0.550
0.515
0.590
0.589
0.580
0.534
0.590
0.497
IoU
0.343
0.380
0.348
0.315
0.338
0.275
0.334
0.320
0.344
examples. We randomly select the negative class words from the rest of the data, equal in size to the positive class examples. We drop the concepts that have less than 200 examples to ensure stable training. We split the binary classification dataset into train/dev/test splits of 70/15/15 percent. We use Å₁ = 0.01 and A2 = 0.01 for regularization-ban based methods. We present the results of POS only in the main paper and provide the rest in Appx. 7.3
Number of top neurons: We did not optimize the number of neurons s selected by each method as this would result in varying number of neurons and will make the compatibility score incompatible across methods. We consider = 10, 30, 50 which covers diverse range to generalize the findings.
Baseline: We generate a random ranking of neurons to compare the behavior of a neuron interpretation method with a random selection of neurons. We refer to it as Random.
Procedure: We extract layer-wise contextualized representations of words in the dataset. For every layer representation, we discover the top neurons with respect to concept using each neuron interpretation method. We follow leave-one-out strategy to calculate the compatibility score of each method with respect to other methods. More specifically, we consider method as a test method and select the rest as the database of the sets of top neurons. We then calculate the AvgOver lap and MeuronVote scores of the test method.
neuron interpretation
4.2 Compatibility Scores
Table 2 presents the average compatibility scores across all concepts and when selecting the top 10, 30, and 50 neurons from layers 1, 6, and 12.
AvgOverlap and NeuronVote show similar pattern: While the scale of the compatibility score is different between both metrics, they are consistent in highlighting the top interpretation methods i.e. Lasso, LCA, and Probeless. In the rest of the paper, we mainly present the results of NeuronVote, and discuss the AvgOverlap only when it is necessary to show a different trend.
Overall comparison across methods: Table presents the NeuronVote scores averaged over all POS concepts and when selecting the top 10, 30, and 50 neurons. The bold, underline, and dashed numbers represent the first, second, and third best compatibility scores respectively. Probeless consistently achieves the highest compatibilityy across all layers and models. other words, the neurons selected by Probeless have the highest overlap with the neurons selected by other methods. We did not observe a similar trend for IoU which also a corpus-based method.
LCA and Lasso achieve the second-best compatibility score (see underlined and dashed numbers in Table Ridge, although methodologically similar to LCA and Lasso, did not consistently show high compatibility across models. We discuss these methods later in Section Gaussian achieved the worst compatibility score which is lower by 0.354 points from the best NeuronVote score on layer and is only 0. 1.12 better than Random for BERT. The result of Gaussian presents an interesting scenario where the compatibility score suggests that the neurons identified by Gaussian are least similar to all other methods. However, the difference with Random, although small, suggests that the selection of neurons is not random. The low results of Gaussian are inline with Antverg & Belinkov (202T) where they found that the Gaussian method memorizes the probing task and it may not provide most faithful ranking of neurons. These scores, therefore, suggest that further investigation may be required to confirm the efficacy of the Gaussian method.
7

---

A notable point about the top 3 methods (Probeless, LCA, and Lasso) is their methodological diversity. despite of which they selected similar neurons other methods. On the other hand, methodologically similar techniques such as IoU and Probeless did not result in similar compatibility scores. This shows that the compatibility metric is not biased by methodological similarity, which was one of the primary concerns with the commonly used evaluation metrics.
Layer-wise trend: Ethayarajh 2019) showed that each layer is different in terms of representation
geometry. Saijad et al. (2022a) revealed that due to the geometry of the representation space, particularly in the last layers, knowledge of concept may be readily available. We hypothesize that the performance of certain neuron interpretation methods may vary based on the nature of the representation space. Our compatibility methods facilitate quantitative evidence to support this hypothesis by showing that certain neuron analysis methods suffer in selecting the most compatible neurons from the higher layers. LCA and Lasso both showed substantial drop in their compatibility scores for the 12th layer (for example, Table ( NeuronVote LCA 0.544 for layer 6 and 0.391 for layer 12 for BERT). In contrast, Probeless did not show any substantial difference in the scores of the last layers and earlier layers (NeuronVote: Probeless 0.550 vs 0.515 for layer 6 and 12 respectively for BERT). This further support Probeless as the most reliable method and encourages further investigation into the robustness of Lasso and LCA with varying representational space.
Compatibility across models: The overall trend of top-performing neuron interpretation methods is similar across BERT, RoBERTa, and XLMR. One notable difference is the compatibility score of Ridge for the last layer of the RoBERTa (0.510) and XLM-R (0.439) models which is substantially higher compared to the BERT model (0.360). Moreover, it achieved the second best score for XLM-R after Probeless and is substantially better than LCA and Lasso. Ridge prefers correlated features in contrast to Lasso, which prefers a few spiky features highly predictive of the concept. Based on the results, we hypothesize that the last layers of RoBERTa and XLM-R consist of highly correlated neurons and Ridge is effective in discovering salient neurons of correlated nature. LCA balances between spiky and correlated neurons, and results higher scores than Lasso in these cases.
Bendom
daams
damm
Gureddnn
Cauedden
1CA
1CA
1CA
R.dge
Rsgge
Rggge
Protedess
Proteless
100
I0U
Randton
Govssaeo
ECA
Ridge
LCA
Rodget
Prodetens
LC4
Lasto
Pubbates
120
(a) Layer 1
(b) Layer 6
(c) Layer 12
Figure 1: Task: POS. Model: BERT, Comparing average overlap of top 10-50 neurons
baseline that provides a perspective of the correctness of a newly proposed neuron interpretation method. It gave substantially lower scores than all neuron interpretation methods (Table 3). Even though the Gaussian showed the lowest compatibility score among interpretation methods, the difference with Random adds a factor of confidence its ranking. Moreover, low compatibility score for Random also implies that each of the other methods are bringing in important neurons to some degree.
Random selection of neurons: The compatibility score of Random serves
4.3 Pairwise Comparison
The compatibility metrics provide single score to understand an interpretation method in the context of a large set of other interpretation methods. We further extend the amalysis to pair-wise comparison and provide insights into the relationship between different methods. Figure presents the heatmaps comparing the average overlap of the top 10 - 50 neurons across methods using BERT. The light color in the heatmap shows higher matches while the dark color shows lower matches. We added Random a random selection of neurons as baseline. It showed an average overlap of less than 3% only (see the first row in heatmaps).
Comparing Regularization-bsee Methods: LCC and Lasso showed a high overlap of up to 64% in the discovered neurons (see Figure ((a) l(b)). However, Ridge results in lower overlap of at most
8

---

29% compared to other methods. Note that the Ridge regularization does not force the coefficients of neurons to zero, thus considers most of the neurons during the training phase. The resulting top neurons may represent correlated neurons capturing similar and redundant knowledge (Dalvi eet al. 2020) and may not be representative of the neurons with the most predictive power with respect to the understudied concept. Lasso prefers individual neurons most predictive of the concept and ElasticNet (LCA) provides a good balance between selecting individual neurons and correlated neurons highly predictive of the concept.
Methods: Probeless and IoU both directly rely on the activation value of neuron. However, did not observe consistent overlap between their discovered neurons (at most 30% in Figurd](c)). In comparison to other classes of methods, Probeless showed better and consistent overlap with LCA and Lasso. For example, in Figure [(a] we observed an overlap of 66% between Probeless and LCA, and 55% between Probeless and Lasso. The relatively
Corpus-based vs. Regularization-
high overlap of Probeless with methods using Lasso points towards the selection of identical focused neurons by these methods despite their methodological differences.
Both Ridge and IoU show smaller but consistent overlap with most of the methods. As discussed earlier, Ridge works well for correlated features which may not be highly predictive of the concept. IoU relies on activation values and it may select neurons that consistently activate higher than the threshold irrespective of the concept. Gaussian did not show substantial overlap with any methods or with any combination of methods. Further investigation needed evaluate its' efficacy.
Outliers:
5 Related Work
The area of interpreting deep learning models constitutes a broad expanse of research. This section provides synthesized overview of diverse interpretability subareas within deep learning models applied to Natural Language Processing (NLP), while also outlining the scope of our study.
Feature importance and atfribution methods identifies the contribution of input features to predictions. These methodologies predominantly rely on the gradient of the output concerning the input feature and determine input feature importance by evaluating the magnitude of gradient values (Denil et al.) 2014 Sundararajan et al. 2017 Danilevsky et al.) (020))
Counterfactual Intervention revolves around an intricate analysis of the interplay between input features and predictions. This approach involves manipulating inputs and quantifying resulting output alterations. Diverse intervention strategies, including erasing input words and substituting words with different meanings have been used (Li et al. 2016b Ribeiro et al., 2018).
Atlention Weights: Numerous investigations have been directed towards interpreting components of deep learning models at varying levels of granularity. For instance, attention weights have emerged as viable metric to gauge the interrelation between input instances and model outputs (Martins & Astudillo 2016 Vig 2019). Along these lines, Geva et al. (022)) delved into the analysis of feedforward neural network components within the transtormer model, revealing their functionality as key-value memories. Additionally, Voita et al. (2019) demonstrated that pruning many attention
heads has minimal impact performance.
Mechanistic Interpretability puts a focus on reverse engineering of network weights to comprehend their behavior (Nanda et al., (023). Building upon the Distill Circuits thread, [Elhage et al. (2021)
investigated two-layered transformer models with attention blocks, identifying attention heads con- tributing to in-context learning. This understanding was further extended to larger wranfoormerbbaeed
language models by Olsson et al. (2022). To enhance neuron interpretability. |Ehhage et al. (2022)
introduced Softmax Linear unit as activation function replacement. |Wang et al. (202aa) attempted to bridge mechanistic interpretability findings in small networks to large ones, particularly GPT-2 small. Their approach involved iteratively tracing influential model components from predictions using causal intervention. They showcased the potential of mechanistic interpretability in understand- ing extensive models, while also highlighting associated challenges. Similarly, Bricken et al. (2023) uses a sparse autoencoder to disentangle polysemantic neurons.
Representation Analysis involves probing network representations concerning predefined concepts. to quantify the extent of knowledge captured in these representations (Conneau al., 2018 Liu et al.)
2019a Tenney et al., 2019; Durrani et al. 2019 Arps et al. 2022). This is often realized through

---

training diagnostic classifiers for linguistic concepts, wherein classifier accuracy serves an indicator of concept knowledge within representations. See Belinkov & Glass (2019) for a comprehensive survey.
Neuron Interpretation A more intricate form of representation analysis, termed neuron interpretation, delves into how knowledge is structured within the network (Sajjad et al., 2022b). This approach establishes connections between neurons and predefined concepts, offering insights into where and how specific concept knowledge is assimilated. Work done on neuron analysis can be broadly classified into three groups: Neuron visualization involves manual identification of patterns across a set of sentences (Li et al., 2016a, Karpathy et al. (015)) More recently Foote et al. (2023) proposed an automated upproach to enhance interpretability of Large Language Models (LLMs) by extracting and visualizing individual neuron behaviors as interpretable graphs. Corpus- based Methods explore the role of neuron through techniques such as ranking sentences a corpus (Kádár et al. 2017b),
generating synthetic sentences (Poerner et al. 2018) that maximize its activation, or computing neuron-level statistics over a corpus (Mu& Andreas 2020 [uau al. [020) Antverg & Belinkov
2022). Bills et al. (2023); Mousi et al. (2023) proposed the use of LLM to interpret neurons. Probing Methods identify salient neurons for a concept by training a classifier using neuron activations as
features (Radford et al.) 2019 Lakretz et al. 2019; Durrani et al. 2022) or fitting a multivariate
Gaussian over all neurons and then extracting individual probes for single neurons (Hennigen et al.
2020).
number of works identify neurons with respect to the output class (Wang et al. 2022b Dai et al. 2022). They are effective in finding neurons that play a role in the prediction. In this paper, we focus on the neuron interpretation methods that take a concept as input and find neurons with respect to the concept. We propose an evaluation framework to formalize the evaluation and comparison of results across methods. Moreover, we propose a novel method, MeanSelect and present a case study of using the evaluation framework.
Conclusion and Limitation
We provided a thorough comparative analysis of neuron interpretation methods in NLP. We overeame the challenge of the lack of a standard evaluation metric and gold annotations by proposing an evaluation strategy consisting of two compatibility metrics and a pair-wise comparison. In addition to developing capability to evaluate new neuron analysis method, we presented various insights into existing neuron interpretation methods. For example, due to the correlated nature of last layer representations, they are most challenging for interpretation methods. The selection of top neurons overlaps substantially irrespective of the methodological differences among techniques. We made the evaluation framework available to the research community.
Limitations: Our methodology relied on consensus and thus inherits the limitations of that paradigm. The consensus may mislcad the evaluation under certain settings e.g. if the majority of the voters form lobby then they can skew the results of the consensus. This scenario can happen if set of methods always produce close to identical ranking, they will cause a decrease in the compatibility score of new method that produces a ranking different from theirs. Another possible issue is if a new method discovers an entirely different set of neurons than the ones discovered by other methods. The compatibility score of such a method will be low. We intend to mitigate these issues by including a large number of diverse neuron interpretation methods that are theoretically different from each other. We believe that under these settings, the scenario of an entirely different set of discovered neurons is more theoretical than practical. We further present a pair-wise wise analysis that provides an analysis at a more granular level and bring insight into how a method compares with other methods. The pairwise comparison heatmap will highlight if a set of methods form a lobby.
Another limitation of current neuron interpretation methods is that they do not explicitly target the discovery of neurons of diverse nature such as polysemantic, and superposition. Theoretically, only the ElasticNet regularization is capable of discovering neurons learning a singular function and multiple functions. The other methods such as Probeless are incapable of discovering multifunction neurons. None of the neuron interpretation methods explicitly identify superposition neurons. Explicit modeling of neurons of different nature in a neuron interpretation method may result in discovering novel sets of neurons. The evaluation methods including our proposed framework do not explicitly compare neurons of different types.
10

---

References
Abzianidze, L., Bjerva, J., Evang. K., Haagsma, H., van Noord, R., Ludmann, P., Nguyen, D.-D., and Bos, J. The parallel meaning bank: Towards multilingual corpus of translations annotated with compositional meaning representations. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguisties, EACL '17, pp. 242-247, Valencia, Spain, 2017.
Antverg, O. and Belinkov, Y. On the pitfalls of analyzing individual neurons in language models. In International Conference on Learning Representations, 2021.
Antverg, O. and Belinkov, Y. On the pitfalls of analyzing individual neurons in language models. In International Conference Learning Representations, 2022. URL http/tttss//opeeeeeeew....o forrmmiid-BuzDENPPIMM
Antverg. O., David, E., and Belinkov, Y. Idani: Inference-time domain adaptation via neuron- level interventions, 2022. URL https: wwwoorgg//ao..eoeo abs//20b..2..... 00259
Arps, D.. Samih, Y., Kallmeyer, L., and Sajjad, H. Probing for constituency structure in neural language models. In Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 6738-6757, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10.18653/ 2.11653/v1/0022 1/2022. http:///.2.....nnnee...... fidding--mmllp502 2022.
URL www.hhtsts//aaattoo.llaaoooo// org/
5..
Bau, A., Belinkov, Y., Sajjad, H., Durrani, N., Dalvi, F., and Glass, J. Identifying and control- ling important neurons in neural machine translation. In International Conference on Learning Representations, 2019, URL https:///////ttteeeeee http://///opeeeeeeeeeeeeeerrrrro -
Belinkov, Y. and Glass, J. Analysis methods in neural language processing: A survey. Transactions of the Association for Computationnal Linguisttics, 7:49-72, March 2019. doi: 10.11622tacla__00254
URL http::tpps/ttttm //wtp:::/wwww.... wwwwww.wwbb..... www.leb.org//atthoogy////// http//annoooooo11-110000 1004
Belinkov, Y., Durrani, N., Dalvi. F., Sajjad, H., and Glass, What do Neural Machine Translation
Models Learn about Morphology? In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL.), Vancouver, July 2017. Association for Computational I Lin- coli.... uni saar land dw/pdd hteepdd//PPP1//P17-1100
guistics. URL
pdf
Bills, S., Cammarata, N., Mossing, D., Tillman, H., Gao, L., Goh, G., Sutskever, I., Leike, J.,
Wu, J., and Saunders, W. Language models can explain neurons in language models. https //openaipublic..loo wlo.bll......b....oooo core. windows..... www.//eee/nneeeeeeeeeee http://eeeurrooeellaaaaaarrrreee
html| 2023.
Bricken, T., Templeton, A., Batson, J.,, Chen, B., Jermyn, A., Conerly, T., Turner, N., Anil, C., Denison, C., Askell, A., Lasenby, R., Wu, Y., Kravec, S., Schiefer, N., Maxwell, T., Joseph, N., Hatfield- Dodds, Z.,, Tamkin, A., Nguyen, K., McLean, B., Burke, E.,, Hume, T., Carter, S., Henighan, T., and Olah, C. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2023. naaureesindex..tm..
Colombo, P., Noiry, N., Irurozki, E., and CLEMENCON, S. What the best systems? new perspectives on NLP benchmarking. In Oh, A. H., Agarwal, A., Belgrave. D., and Cho, K. (eds.), Advances Neural Information Processing Systems, 2022. URL forum?id=kktVvzqPvg
net/
Conneau, A.,
Kruszewski, G., Lample, G., Barrault, I.,, and Baroni, M. What you can cram into single vector: Probing sentence embeddings for linguistic properties. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL), July 2018.
Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzmán, F., Grave, E.,, Ott, M., Zettlemoyer, L., and Stoyanov, V. Unsupervised cross-lingual representation learning at scale. CoRR, abs/1911.099 16, 2019. hRRL/http//arxiwoorr//// hatp//bbbs/1111..//// 109991011111
Cooper, D. and Zillante, A. A comparison of cumulative voting and generalized plurality voting. In Public Choice, 2012. doi: 10.10077/11127-0009970

---

Dai, D., Dong, L., Hao, Y.. Sui, Z.,. Chang, B., and Wei, F. Knowledge neurons in pretrained transformers. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pp. 8493-8502, Dublin, Ireland, May 2022. Association for Computational Linguisties. doi: 10. 10.18665311/222222cclooorgg
URL,
org/2022. -cleaaclllonnn. 581
Dalvi, F., Durrani, N., Sajjad, H., Belinkov, Y., Bau, D. A., and Glass, J. What is one grain of sand in the desert? analyzing individual neurons in deep nlp models. In Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence (AAAI, Oral presentation), January 2019.
Dalvi, F., Saijad, H., Durrani, N., and Belinkov, Y. Analyzing redundancy in pretrained transformer models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP-2020) Online, November 2020.
Dalvi, F., Khan, A. R., Alam, F., Durrani, N., Xu, J., and Sajjad, H. Discovering latent concepts learned in BERT. In International Conference on Learning Representations, 2022. URL /ttps //openreview net/forum?id-PPTMopP
Dalvi, F, Durrani, N., and Sajjad, H. Neurox library for neuron analysis of deep nlp models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pp. 75-83, Toronto, Canada, July 2023, Association for Computational Linguistics.
Daniel, W. Applied Nonparametric Statistics. Duxbury advanced series in statistics and decision sciences. PWS-KENT Pub., 1990. ISBN 9780534919764. URL https://books. google ca/ books?id=0hPAAMAAMA
Danilevsky, M., Qian, K., Aharonov, R., Katsis, Y., Kawas, B., and Sen, P. A survey of the state of explainable AI for natural language processing. In Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pp. 447-459, Suzhou, China, December 2020. Association for Computational Linguistics. URL httph:tts////// aww.aclloel aclweb. .rrg wwt.oorg/////ooo/oo httgg//athloog/2200 aacl moce-main... 46
Denil, M., Demiraj, A., and de Freitas, N. Extraction of salient sentences from labelled documents. CoRR, abs/1412.68155 2014. URL http://arxiv.. org/ hatgg//b///// 11411.6.1155
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. BERT:
Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapler of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long and Short Papers), Minneapolis, Minnesota, 2019, Association for Computational LinguuttionnaLinguuuiiiicc...
Dhamdhere, K., Sundararajan, M., and Yan, Q. How important is neuron? CoRR, abs/1805, 12233, 2018. http:httt///arxw.orrrg http://axxivv.rgabbb/////ss 805. 12233
Durrani, N., Dalvi, F., Sajjad, H., Belinkov, Y., and Nakov, P. One size does not fit all: Comparing NMT representations of different granularities. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pp. 1504-1516, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 10.18633/11/N19-11522 URL https www.w//wwww...........e.ee/eeeee www.orrgggtttooooooroooooooo 9-11111
Durrani, N., Sajjad, H., Dalvi, F., and Belinkov, Y. Analyzing individual neurons in pre- trained language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). pp. 4865 4880, Online, November 2020. Associa- tion for Computational Linguistics. doi: //www. www.ww.lweb.com org/
1tt.:///1122200..mmmmmaiaaaa www.eb..rgg/ntoooogo//22/// awwaannggooooonooo emnlp-maill wannll-mmii... 395
URL https
Durrani, N., Dalvi, F. and Sajjad, H. Linguistic correlation analysis: Discovering salient neurons in 13288
deepnip models, 2022. URL
12

---

Elhage, N., Nanda, N., Olsson, C., Henighan, T., Joseph, N., Mann, B., Askell, A., Bai, Y., Chen, A.. Conerly, T., DasSarma, N., Drain, D., Ganguli, D., Saatfield-Dodike Z., Hernandez, D., Jones, A.. Kernion, J., Lovitt, L.., Ndousse, K., Amodei, D., Brown, TT, Clark, J., Kaplan, J., McCandlish, S., and Olah, C. mathematical framework for transformer circuits. Transformer Circuits Thread, 2021.
Elhage, N., Hume, T., Olsson, C., Nanda, N., Henighan, T., Johnston, S., ElShowk, S., Joseph, N., DasSarma, N., Mann, B., Hernandez. D., Askell, A., Ndousse, K., Drain, D., Chen, A., Bai, Y., Ganguli, D., Lovitt, L.,, Hatfield- Doddss, Z., Kernion, J., Conerly, T., Kravec, S., Fort, S., Kadavath, S., Jacobson, J., Tran-Johnson, E., Kaplan, J., Clark, J., Brown, T., McCandlish, S., Amodei, D., and Olah, C. Softmax linear units. Transformer Circuits Thread, 2022. https:/transformer- circuits. Drtuuits.pub/2222200 2ttp//22///ll/iidxx.tttlll
Ethayarajh, K. How contextual are contextualized word representations? comparing the ge- ometry of BERT, ELMo, and GPT-2 embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Con- ference on Natural Language Processing CEMNLP-IJCNLE pp. 55-65, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v//D19-10000 URL https://www aclweb. org http//hhoooogg/1-110000
Foote, A., Nanda, N., Kran, E., Konstas, I., Cohen, S., and Barez, F. Neuron to graph: Interpreting language model neurons at scale, 2023.
Geva, M. Schuster, R., Berant, J., and Levy, O. Transformer feed-forward layers are key-value memories. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 5484-5495, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653//1/2021emmmm// //aclanthology.org///
URL https
www.e-nwmmm......... main. 446
Hennigen, L. T., Williams, A., and Cotterell, R. Intrinsic probing through dimension selection. In Pro- ceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 197-216, Online, November 2020. Association for Computational Linguistics. doi: 10. 18653/ w1/2020.emnlp-main.n
URL https:t/pt//ttttt ac www..w.w........................ org/ antho Logy 2020. emnlp-main. 15
Hewitt, J. and Liang. P. Designing and interpreting probes with control tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 2733-2743, Hong Kong, China, November 2019. Association for Computational Linguisties. doi: 10. 10.18653/11/D11-117.
URL httts:://cllanthologg. 11110//900000//11-117775
Hupkes, D., Veldhoen, S., and Zuidema, W. Visualisation and 'diagnostic classifiers' reveal how recurrent and recursive neural networks process hierarchical structure, 2018.
Kádár, Á., Chrupala, G., and Alishahi, A. Representation of linguistic form and function in recurrent neural networks. Computational Linguistics, 43(4):761-780, Decembe 2017a. doi: 10.1 COLL_a00030M URL hhttpp::/wwww.arrr http://wwwwwwwoooooerr/atttttaoa
Kádár, A., Chrupata, G., and Alishahi, A. Representation of linguistic form and function in recurrent neural networks. Computational Linguistics, 43(4):761-780, 2017b.
Karpathy, A., Johnson, J., and Fei-Fei, L. Visualizing and understanding recurrent networks. arXiv preprint TXXiv:1506600776 2015.
Kendall, M. A new measure of rank correlation. Biometrika, 1938.
Lakretz, Y., Kruszewski, G., Desbordes, T., Hupkes, D., Dehaene, S., and Baroni, M. The emergence of number and syntax units in LSTM language models. In Proceedings of the 2019 Conference of the North American Chupter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 11-20, Minneapolis, Minnesota, June 2019, Association for Computational Linguistics. doi: 10. 18153///1//N11110022 URL https
13

---

Lapata, M. Automatic evaluation of information ordering: Kendall's tau. Computational Linguistics, 32(4):471 1484, 2006. doi: 10.1162//c01.220..224444444 URL https://aclanthoooogy org/ J06-4002
Li, J., Chen, X., Hovy, E., and Jurafsky, D. Visualizing and understanding neural models in NLP. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics; Human Language Technologies, pp. 681-691 San Diego, California, June 2016a. Association for Computational Linguistics. doi: 10.18653/w1/N16. 1082. URL https: htwps://wwww... wwwwww.wwww..................... org http//anhhooog//N6-11....
Li, J.. Monroe, W. and Jurafsky, D. Understanding neural networks through representation erasure. CoRR, abs/1612.08220 2016b. URL http::/arxiv...o org hwtpg//bs//11..... 06222
http://arxxi..ooras////1111
Lin. Z., Li, Y., and Guo, X. Consensus of rankings. CoRR, abs/1704.08464, 2017. URL http //arxiv.org/abtt www..org//bb//oogeo farxiv.org/abs//0e.086666 08464.
Lippman, D. Voting Theory, Math in Society, Pierce College Ft Steilacoom, 2012.
Liu, N. F., Gardner, M., Belinkov, Y., Peters, M. E., and Smith, N. A. Linguistic knowledge and transferability of contextual representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pp. 1073-1094, Minneapolis, Minnesota, June 2019a. Association for Computational Linguistics. URL htthhtts:://ww ac awwebb...bb............ orggantholog... N19-1112
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, I.., and Stoyanov, V. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11699 2019b. URL http:://aaxiii org/abs./////oooooooo.t....... /19ss///97... 11692
Lundberg, S. M. and Lee, S.-I. A unified approach to interpreting model predictions. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (eds.), Advances Neural Information Processing Systems 30, pp. 4765-4774. Curran Associates, Inc., 2017. URL wwwpuu/iiieaaappaaaa--------e
nips. cc/paper/ model -wwoedeppeedccionnss..eeeee
Marcus, M. P., Santorini, B., and Marcinkiewicz, M. A. Building large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19(2)3113-330, 1993. URL https
//www. www.aaclwe..org/aat wwtp://wwww.../..aaoooooo///////
Martins, A. and Astudillo, R.
From softmax to sparsemax: A sparse model of attention and multi-label classification. In Balcan, M. F. and Weinberger, K. Q. (eds.), Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pp. 1614-1623, New York, New York, USA, 20-22 Jun 2016. PMLR. URL https: httpss///ooceeiigg..oo ml www.ppressseeeeeee watpsss//////rr//// tins1...innnn..mmm
Mousi, B., Durrani, N., and Dalvi, F. Can Ilms facilitate interpretation of pre-trained language models? In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), Singapore, dec 2023. Association for Computational Linguistics.
Mu, J. and Andreas, J. Compositional explanations of neurons. CoRR, ahs/20066.14033 2020. URL https://arxiw.orr/abb////
14032
Na. S., Choe, Y. Y.J. Lee, D., and Kim, G. Discovery of natural language concepts in individual units of CNNs. CoRR, abs//102.072240 2019. URL http:://rxi.....////////// wttp://wwwwggg///////ee// U.L.ittt//aaak..oor////////222. 07249
Nanda, N., Chan, L., Lieberum, T., Smith, J., and Steinhardt, J. Progress measures for grokking via
mechanistic interpretability. The Eleventh International Conference on Learning Representa- tions, 2023. URL
O' O'Connor, J. and Robertson, E. The mactutor history of mathematics archive, 2003.
14

---

Olsson, C., Elhage, N., Nanda, N., Joseph. N., DasSarma. N., Henighan, T.. Mann, B., Askell, A.. Bai, Y., Chen, A.. Conerly, T., Drain, D., Ganguli, D., Hatfield-Dodde Z., Hernandez, D.,, Johnston, S., Jones, A., Kernion, J., Lovitt, L.., Ndousse, K., Amodei, D., Brown, T., Clark, J., Kaplan, J., McCandlish, S., and Olah, C. In-context learning and induction heads. Transformer Circuits Thread, 2022. https://ransformercciruutt http://ssppppee//ooooooooooeeeee hawpr/eeeeeeeeinnnnnnnnnnnitro-e
html..
Patrão Neves, M. Consensus, pp. 19-29. Encyclopedia of Global Bioethics, Springer, 06 2016. doi: 00000/9978-331-055544--33 119-1.
Poemer, N., Roth, B., and Schütze, H. Interpretable textual neuron representations for NLP. In Proceedings of the 2018 EMNLP Workshop BlackboxNI.P: Analyzing and Interpreting Neural Networks for NLP, pp. 325-327, Brussels, Belgium, November 2018. Association for Computa- tional Linguistics. doi: 10. 10.18663/11/ 1A11///115537777 URL httph: http::://www..cll.e.. http://w...og////ooooooeo /nghaogoology//
5437
Prasanna, S., Rogers, A., and Rumshisky, A. When BERT Plays the Lottery, All Tickets Are Winning. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pp. 3208-3229, Online, November 2020. Association for Computational Linguistics. URL httLh / htwp:/hhhooogyooo//222//2// org/ wwt.:///.....mmmmmmm.......... emnlp-main
doi: 10. 10./6653ww122200mmmllp..mm
259
Radford, A., Wu, J., Child, R., Luan, D.,, Amodei, D., Sutskever, I,, et al. Language models unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Ribeiro, M. T., Singh, S., and Guestrin, C. Anchors: High-pu precision model-agnostic explanations. In Proceedings of the Thirty-Secone AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications Artificial Intelligence Conference and Eighth AAAI Symposium Educational Advances in Artificial Intelligence, AAAI' 18/LAAI 18/EAAT// 18. AAAI Press, 2018.
ISBN 978 108-1--7773-800-1
Sajjad, H., Alam, F., Dalvi, F., and Durrani, Effect of post-processing on contextualized word representations. In Proceedings of the 29th International Conference Computational Linguis- tics, pp. 3127-3142, Gyeongju, Republic of Korea, October 2022a. International Committee on Computational Linguistics. /RLL/htts::///caaattooo/
org/ 202. coling-1.2
Sajjad, H., Durrani, N., and Dalvi, F. Neuron-level Interpretation of Deep NLP Models: A Survey. Transactions of the Association for Compulational Linguistics, 2022b.
Suau, X., Zappella, L., and Apostoloff, Finding experts in transformer models. abs/2005.07647. 2020. URL
CoRR,
07647
Sundararajan, M., Taly, A., and Yan, Axiomatic attribution for deep networks. In Proceedings of the 34th International Conference on Machine Learning Volume 70, (CML'17, pp. 3319-3328. JMg.. 2017.
Tenney, I., Das, D., and Pavlick, E. BERT rediscovers the classical NLP pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4593-4601 Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10. N0.18653/v1/P19.2 1452. URL https:// aclweb. www.eee.orr/atthhhoooooo wwgg/anhhlogy//19-14555
Tjong Kim Sang, E. F. and Buchholz, S. Introduction to the CoNLL-2000 shared task chunking. In Fourth Conference Computational Natural Language Learning and the Second Learning Language in Logic Workshop, 2000. UtL/httss: //wp::::///wwww. aclweb. org wttgg//thhllog//ooo0000000
Vig, J. Visualizing attention in transformer-bassed language representation models. CoRR, Abss/194.02669, 2019. URL http:://arxxv..org/aa//11104 02677 02679
Voita, E., Talbot, D., Moiseev, F., Semnrich, R., and Titov, I. Analyzing multi-heal self-attention: Specialized heads the heavy lifting, the rest can pruned. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 5797-5808, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10. 10.18653/v1/P11-1558 URL
https: / wwwp ac aclweb. www.ee..ogg//atttooooo//////--5s
15

---

Wang, K., Variengien, A., Conmy, A., Shlegeris, B., and Steinhardt, J. Interpretability in the wild: circuit for indirect object identification gpt-2 small. 2022a.
Wang, X., Wen, K., Zhang, Z., Hou, L., Liu, Z., and Li, J. Finding skill neurons pre-trained transformer-based language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 11132-11 11152, Abu Dhabi, United Arab Emirates, December 2022b. Association for Computational Linguistics. doi: 10.18633/1/2202..mmm// mainn........65.... URL https://aclanthologg.oooo org 2822. emnlpp wwwne-mnnalmmee main.
Yilmaz, E., Aslam, J. A., and Robertson, S. A new rank correlation coefficient for information retrieval. In Proceedings of the 3 Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '08, pp. 587-594, New York, NY, USA, 2008. SE049990331113943555 1390435.
Association for Computing Machinery. ISBN 978160558
doi: 10.1 10.1145/1390334,
URL https::/dooi orgorog// 10. 1145/1390334 1390435
Zhang, K. and Bowman, S. Language modeling teaches you more than translation does: Lessons learned through auxiliary syntactic task analysis. Proceedings of the 2018 EMNLP Workshop BlackboxNI.P. Analyzing and Interpreting Neural Networks for NLP, pp. 359-361, Brussels, Bel- gium, November 2018, Association for Computational Linguistics. doi: 10.18653/1//18-555 URL phttps://wwet wwwwwb.wwb.ccc org/ aggpbbooo///tt..oo////// -
Zou, H. and Hastie, T. Regularization and variable selection via the elastic net. Journal of the Royal Statistical Society, Series B, 67:301-320, 2005.
16

---

7
Appendix
0.05
3:45
6:41
0.23
0.06
0.26
2.08
0.35
0.31
6.28
8:46
3.02
0.28
0.25
6.22
0.44
0.26
Rgw 12
0:32
9.29
0.24
0.2
0.54
0.32
Myer
22
3.12
0:9
0:17
6.33
0.34
0:20
Myer 12
3.02
0.23
0.39
4:24
029
0.25
Goossay
Lasse
Hldge
Grostec
Lok
Roger
Probeds
Groptec
Lok
Rode
(a) BERT
(b) RoBERTa
XLMR
Figure 2: Task: POS, Pairwise comparison of MeanSelect with other methods using 10-50 neurons
7.1 Case Study
Mean Select: We propose corpus-based method, MeanSelect, for neuron ranking case-study to illustrate how researchers can use our proposed methodology. Kádár et al. (2017b) generated explanations for neurons by extracting top-N 5-gram context for each neuron based on the magnitude of their activations, followed by human annotation. Na et al.) (2019) removed the human-in-the-loopp by extracting concepts of various granularities from a parsed tree and aligned highly activating neurons to the concepts. Inspired by these works, we propose a novel method, MeanSelect, that generates a ranking of neurons with respect concept. The intuition is that neuron learning a concept will have consistently high activations across different contexts where that concept appears. However, there may be neuron that always activates with high value irrespective of the concept. A difference in the mean value of the neuron activating the concept and all other concepts will provide the behavior of the neuron for the concept. Following our notation in Section the score of given neuron is defined as follows:
(dC) - (c(C)
R(n, C)
(10)
Tmmoz Remón
where nlC) is the average, Rmar is the max and 1yyin the min of activations z(n, w) where w C, and p(C) is the average of activations over the random concept set.
Compatibility Metrics Table 4 shows the compatibility score of MeanSelect using the repre- sentation of different layers and compared with the Random selection of neurons. The high compatibility scores show that MeanSelect discovers neurons that are endorsed by other neuron interpretation methods. This serves as a measure of confidence in the proposed method. Moreover, one may observe that the method has a relatively lower score for the last layer compared other layers, giving insight into potential improvements that can be made to the behavior of this method.
Pairwise Comparison The pairwise comparison of methods further provides insights into how the new method relates to other methods in terms of the resulting neurons. Figure]Jshows the heatmaps of three pre-trained models. MeanSelect has the highest overlap with the Probeless method and except Gaussian, it shows an overlap of at least 0.23 points with other methods. While the high overlap of MeanSelect with Probeless is surprising given both based on similar intuitions, the overlap with LCA and LI shows that the method selecting diverse set of neurons captured by variety of methods. Similar to the discussion on compatibility score, here we observe a substantial overlap drop in the last layer and this highlights the potential vulnerability of the method to certain representations.
7.2 Concept Datasets
We consider concepts from three linguistic tasks: parts of speech tags (POS, Marcus et al. 1993).
semantic tags (SEM, Abzianidze et al. (017) 7 and syntactic chunking (Chunking) using CoNL. I. 2000 shared task dataset (ionng Kim Sang & Buchholz] 2000). For the POS dataset, used 20 concepts
17

---

Table 4: Task: POS, Model: BERT, Average NeuronVote score of MeanSelect using 10-50 neurons
BERT
RoBERTa
XLMR
Layers
12
12
6
12
Random 0.019
0.021
0.023
0.020
0.019
0.017
0.019
0.017
0.022
MeanSelect
0.464
0.476
0.402
0.392
0.451
0.328
0.368
0.408
0.253
Table Task:
Semantic tagging, Average NeuronVote compatibility scores across Semantic tagging concepts when selecting the top 10, 30, and 50 neurons from layers 1, 6 and 12. Bold numbers, underline numbers, and dashed numbers show the first, second, and third best scores respectively
BERT
RoBERTa
XLMR 6
Layers
6
12
6
12
Random
0.018
0.013
0.017
0.011
0.026
0.017
0.026
0.014
0.026
Gaussian
0.257
0.256
0.222
0.237
0.282
0.245
0.176
0.256
0.195
LCA
0.474
0.541
0.385
0.488
0.493
0.347
0.309
0.455
0.367
Lasso
0.407
0.492
0.322
0.391
0.460
0.328
0.294
0.396
0.358
Ridge
0.316
0.343
0.361
0.468
0.492
0.575
0.372
0.430
0.473
Probeless
0.450
0.501
0.476
0.547
0.586
0.640
0.495
0.571
0.464
loU
0.344
0.332
0.380
0.288
0.287
0.242
0.277
0.262
0.282
which have a total dataset size of 40137. These concepts include VBG (777), VBZ (908), NNPS (204), DT (4015), TO (1177), CD (1935), JJ (2836), PRP (801), MD (463), RB (1348). VBP (534), VB (1244), NNS (3021), VBN (1082), POS (433), IN (5039), NN (6660), CC (1220), NNP (4698), and VBD (1742).
For the SEM dataset, used three concepts which have total dataset size of 120941. These concepts include IST (72240), NOW (24137) and EXS (24564). We used 10 concepts from Chunking which have total dataset size of 220606. These concepts include B-ADJP (2493), B-ADVP (5081 1), B-NP (67285), B-PP (26005), B-VP (26078), I-ADJP (805), I-ADVP (532), I-NP (77368) I-PP (339) and I-VP (14620). For all the datasets used in the experiments, and 15%.
train/valid/test split 70%, 15%
7.3 Results
7.3.1
Semantic Tagging Concepts
For the SEM dataset, sample 20000 sentences for experimental validation with train/valid/test split 70%, 15% and 15%. We select three tags: IST (intersective), NOW (present tense) and EXS (untensed simple event). Table p]eresenss the average NeuronVote scores across three models. We observed identical trends to that of POS i.e. Probeless is the most consistent method, LCA and Lasso second best methods but they suffer the last layers.
7.4 Chunking Concepts
Figures and show layer-wise results for the two voting methods proposed the paper, across three understudied models (BERT, RoBERTa and XLM-R). The results show that voting methods consistently rank the Probeless method as the most compatible terms of neuron rankings across the layers. We including detailed results in Tables 5-10 give detailed results with exact numbers.
18

---

Table 6: Task: Chunking, Average NeuronVote compatibility scores across Chunking concepts when selecting the top 10, 30, and 50 neurons from layers 1, 6 and 12. Bold numbers, underline numbers, and dashed numbers show the first, second, and third best scores respectively
BERT
RoBERTa
XLMR
Layers
6
12
6
12
6
12
Random
0.018
0.017
0.023
0.022
0.018
0.022
0.023
0.019
0.014
Gaussian
0.122
0.181
0.174
0.111
0.163
0.164
0.110
0.121
0.143
LCA
0.395
0.469
0.300
0.440
0.422
0.328
0.336
0.447
0.388
Lasso
0.396
0.472
0.301
0.399
0.395
0.322
0.366
0.425
0.395
Ridge
0.235
0.255
0.256
0.303
0.330
0.386
0.254
0.289
0.410
Probeless
0.465
0.506
0,422
0.502
0.500
0.514
0.499
0.507
0,463
IoU
0.346
0.361
0.321
0.285
0.298
0.244
0.319
0.283
0.352
0.30
64-
CA-
0.25
0.3
03
8.5
530
0.0
(a) BERT
(b) RoBERTa
(c) XLMR
Figure 3: AvgOverlap score across different Layers in different models. All methods perform better than the baseline, Random. Probeless is the most consistent method across all models, concepts and across all layers. among the top methods with LCA and Lasso. However, LCA and Lasso show low score on last layers. The performance of Gaussian deteriorates significantly and is closer to Random when applied on XLMR.
0.6
05-
0.5
6A
0.4
03
0.1
61
01-
0.0
Layer
Layer
Laver
(a) BERT
(b) RoBERTa
XLMR
different Layers in different models. All methods perform better than the baseline. Random. Probeless is the most consistent method across all models, concepts and across all layers. It among the top methods with LCA and Lasso. However, LCA and Lasso show low score on last layers.
Figure 4: NeuronVote
19

---

Table 7: This is extension of Table Average Avgoverlap compatibility when selecting the top 10, 30, and 50 neurons. Bold numbers show the best scores. Probeless, LCA the top performing methods. However, Probeless most consistent performing method. LCA drops substantially for the last layers.
concepts
BERT
7
8
10
Kandom
0.022
0.021
0.019
0.018
0.019
0.024
0.020
0.017
0.021
0.022
0.021
0.022
0.023
0.081
LCA
0.189
0.263
0.296
0.296
0.302
0.307
0.303
0.290
0.27
0.266
0.254
0.242
0.245
Lasso
0.171
0.246
0.271
0.270
0.287
0.289
0,289
0,269
0.245
0.247
0.24
0.253
0.243
Ridge
0.118
0.16
0.187
0.192
0.213
0.212
0.204
0.199
0.184
0.161
0.163
0.202
0.224
Probelcas
0.195
0.262
0.292
0.293
0.3
0.305
0,303
0.29
0.273
0.265
0.258
0.325
0.346
Table 8: This is extension of Table Average NeuronVote compatibility scores across concepts when selecting the top 10, 30, and 50 neurons.
BERT
Layers
4
5
6
7
8
10
11
12
Random
0.022
0.019
0.018
0.016
0.019
0.025
0.021
0.018
0.014
0.022
0.026
0.020
0.023
LCA
0,454
0.501
0.577
0.573
0.593
0.581
0.544
0.574
0.572
0.529
0.530
0,512
0.391
Lasso
0.413
0.496
0.518
0.522
0.553
0.546
0,545
0.546
0.511
0.468
0.487
0.465
0.395
Ridge
0.277
0.332
0,364
0.384
0.405
0.392
0.405
0.382
0.379
0.361
0.313
0.338
0.360
Proheless
0.451
0.497
0,543
0.558
0.567
0.563
0.550
0.566
0.562
0.538
0.531
0.509
0.515
IoU
0.381
0.385
0.380
0.380
0.379
0.365
0.348
Table This is an extension of Table Average AvgOverlap compatibility scores across concepts when selecting the top 10, 30, and 50 neurons.
RoBERTa
Layers
5
6
7
8
9
10
11
12
Random
0.021
0.020
0.019
0.023
0.016
0.017
0.020
0.021
0.023
0.014
0.018
0.024
0.018
0.265
LCA
0.322
0.452
0.457
0.457
0.449
0.446
0.441
0.441
0.445
0.450
0.439
0.428
0.365
Lasso
0.314
0.425
0.438
0.430
0.428
0.427
0.425
0,423
0.425
0.433
0.419
0.418
0.355
Ridge
0.338
0.387
0.407
0.414
0.409
0.406
0.402
0.390
0.386
0.405
0.384
0.384
0.391
Proheless
0.388
0.462
0.474
0.474
0.469
0.467
0.461
0.457
0.457
0.460
0.449
0.445
0.424
IoU
0.306
0.318
0.319
0.324
0.324
0.320
0.321
0.324
0.319
0.313
0.305
0.299
0.282
Table 10: This is extension of Table Average NeuronVote compatibility scores concepts when selecting the top 10, 30, and 50 neurons.
RoBERT's
Layers
3
4
5
6
7
8
10
11
Random
0.021
0.020
0.018
0.025
0.013
0.015
0.019
0.021
0.024
0.013
0.016
0.026
0.017
0.189
LCA
0.338
0.550
0,544
0.532
0.513
0.506
0.495
0.518
0.522
0.520
0.509
0.477
0.382
Lasso
0.475
0.482
0.433
0.468
0.451
0.453
0.467
0.478
Ridge
0.405
0.452
0.488
0.496
0.505
0.495
0.491
0.462
0.461
0.477
0.464
0.468
0.516
Proheless
0.524
0.590
0.606
0.597
0.583
0.599
0.589
0.580
0.597
0.587
0.575
0.563
0.580
IoU
0.315
0.314
0.329
0.333
0.335
0.338
0.335
0.333
0.317
0.275
Table 11: This is extension of Table Average AvgOver compatibility scores across concepts when selecting the top 10, 30, and 50 neurons.
XLMR
Layers
2
3
4
5
6
7
8
9
10
11
12
Random
0.020
0.021
0.022
0.021
0.018
0.018
0.019
0.019
0.019
0.020
0.021
0.022
0.023
Gaussian
0.185
0.184
0.181
0.181
0.180
I.A
0.264
0.377
0.428
0.437
0.439
0.442
0.438
0.435
0.438
0.433
0.431
0.408
0.245
Lasso
0.257
0.362
0.408
0.414
0.412
0.421
0.420
0.418
0.415
0.412
0.409
0.389
0.243
Ridge
0.292
0.312
0.327
0.343
0.357
0.355
0.361
0.367
0.370
0.356
0.353
0.338
0.324
Proheless
0.335
0.403
0.434
0.433
0.440
0.441
0.440
0.436
0.439
0.437
0.429
0.412
0.346
InUl
0.279
0.291
0.282
0.281
0.285
0.292
0.299
0.301
0.303
0.296
0.293
0.284
0.263
20

---

Table 12: This an extension of Tablle Average NeuronVote compatibility scores across concepts when selecting the top 10, 30, and 50 neurons.
XLMR
Layers
10
Random
0.022
0.020
0.022
0.022
0.019
0.017
0.017
0.019
0.021
0.017
0.019
0.022
0.023
Gaussiun
0.051
0.118
0.044
0.038
0.037
0.037
0.135
0.033
0.032
0.032
0.041
0.033
0.153
I.CA
0.262
0.450
0.542
0.562
0.554
0.558
0.531
0.523
0.547
0.547
0.528
0.485
0.373
Lasso
0.241
0.410
0.498
0.501
0.476
0.502
0.471
0.495
0.496
0.488
0.478
0.446
0.267
Ridge
0.391
0.446
Probeless
0.491
0.534
0,578
0.567
0,586
0,587
0,590
0,574
0.583
0.575
0.560
0.548
0.497
0.311
Table 13: This is extension of Table 10, 30, and 50 neurons for all layers
Average AvgOverlap score of McanSelect when selecting
Layers
BERT
Random
0.022
0.021
0.019
0.018
0.019
0.022
0,020
0,017
0.021
0.022
0.021
0.022
0.023
MeanSelect
0,266
0.351
0,363
0.362
0.365
0.369
0.374
0.370
0.356
0.349
0.343
0.327
0.283
RoBEKTa
0.019
0.016
0.017
0.020
MeanSclect
0.252
0.294
0.307
0.317
0.326
0.328
0.323
0.320
0.314
0.308
0,283
0.269
0.239
XI.M
Random
0.020
0.021
0.022
0.021
0.018
0.018
0.019
0.019
0.019
0.020
0.022
0.023
MeanSelect
0.233
0.246
0.245
0.247
0.264
0.273
0.294
0,292
0.281
0.261
0.259
0.241
0.159
Table 14: This is an extension of Table 10, 30, and 50 neurons for all layers
Average NeuronVote score of MeanSelect when selecting
Layers
6
Random
0.022
0.019
0.018
0.016
0.019
0.025
0,021
0.018
0.014
0.022
0.026
0.020
0.023
McanSelect
0,361
0.476
0.462
0.488
0.470
0.468
0.479
0.488
0.486
0.463
0.457
0.432
0.400
RoBERTa
Random
0.021
0.020
0.018
0.025
0.013
0.015
0.019
0.021
0.024
0.013
0.016
0.026
0.017
McanSelect
0.455
0,438
XLMR
Random
0.022
0.020
0.022
0.022
0.019
0.017
0.017
0.019
0.021
0.017
0.019
0.022
0.023
MeanSelect
0.376
0.357
0.356
0.342
0.364
0.392
0.414
0.416
0.400
0.372
0.381
0.360
0.243
Comparison of Neuron Vote score of Probeless and LCA under different number of consensus methods. majority of the cases, Probeless outperformed LCA irrespective of the methods used for consensus. Moreover, in three cases where LCA is better than Probeless, their results are comparable.
Table 15:
Num. Methods
Consensus Methods
Probeless
LCA
Gaussian
0.110
0.096
loU
0.274
0.216
Lasso
0.449
0,516
Ridge
0.270
0.240
Gaussian,Lol
0.233
0.177
Gaussian,Ridee
0.202
0.174
Gaussian,Lasso
0,272
0.275
Los
0.351
0.329
Lidso,Ridge
0.404
0.410
Ridge.IoU
0.292
0.243
3
Beuusian,Riige..h
0.265
0,229
LaassiannLassoRRdgee
0.337
0.327
Lasssian
0.327
0.305
Lasso.Ridge.JoU
0.373
0.351
Lanssian_Lsso.RRdge...ol
0.368
0.342
21

---

Sandum
Sandimm
Sandomn
4090
4003
Glessaan
LC
Lacto
Redee
Prodeioss
Ramaion
Cantalan
LCs
Podger
Proaeins
Gesssiann
Larto
Bodge
Prodecss
(a) Layer
(b) Layer
(c) Layer 3
Gandum
Gandorn
Gandomi
Leumstn
Caurmten
Ridge
Rigee
Rldge
Procaless
Pribaless
Probaless
Namstam
LGA
Prigee
Prosmiers
Rarssom
LGA
Langer
Phager
Prodmess
Rarmion
LGc
Lamor
Peggee
Probmees
(d) Layer 4
(c) Layer 5
(f) Layer 6
Sancom
Cousstan
Cousilan
Coussian
LCK
LCA
LCA
CGB
Ridge
Ridge
Ridge
Phocllsss
Probeless
Phoceloss
X2U
XQU
Recoun
Lol
Lesso
Reage
Proseaec
Rensum
Lca
Lensu
Radee
Prodeatu
Ransau
LGa
Lessa
Redgee
Probeiss
gg) Layer
(h) Layer
Layer
Sambom
LC4
LC4
LC4
.2250
10U
10U
10U
Ronsoon
Lok
Lakao
Rogge
Ransium
Lok
Ruage
Lesto
Ragel
Layer
(k) Layer II
Layer 12
Figure 5: This is extension of Figure Comparing average overlap of top 10-50 neurons across methods for BERT
22

---

Sandum
Sandimm
Sandom
4090
4003
Glessaan
LC
Lacto
Redee
Prodeioss
Ransion
Cantalan
LCS
Podger
Proaeins
ind
Gesssiann
Larto
Bodge
Prodecss
(a) Layer
(b) Layer
(c) Layer 3
Gandum
Gandorn
Gandomm
Leumstn
Coummten
Ridge
Rige
Rldge
Procaless
Pribaless
Protaless
Namstam
LGA
Prige
Prosmirr
Rarssom
LGA
Langer
Phager
Prodmess
Rarmion
LGc
Lamor
Pegge
Probmees
(d) Layer 4
(c) Layer 5
(f) Layer 6
Sancom
Cousston
Cousilan
Coussian
LCK
LCA
LCA
CCB
Ridge
Ridge
Ridge
Phocelss
Probeless
Phoceloss
X2U
X2U
Recoun
Loc
Lesso
Redge
Probeaec
Rensum
Lca
Lesso
Radee
Prodeatu
Ransum
LGa
Lessa
Redge
Prodeiss
gg) Layer
(h) Layer
Layer
Sambom
LC4
LC4
LC4
.2250
10U
10U
X0U
Ronston
Lok
Lakao
Rogge
Ransium
Lok
Ruage
Rosdom
Lesto
Raggl
Layer
(k) Layer II
Layer 12
Figure 6: This is extension of Figure Comparing average overlap of top 10-50 neurons across methods for RoBERTa
23

---

Sandum
Sandimn
Sandom
4090
4003
KIt
Glessaan
LCK
Lacto
Redee
Prodeioss
Ramsion
Cantalan
LCs
Podger
Proaeins
Grsssian
Larto
Bodge
Prodecss
(a) Layer
(b) Layer
(c) Layer 3
Gandum
Gandorn
Gandomt
Leumstn
Coummten
Ridge
Rigee
Rldge
Procaless
Pribaless
Protaless
Namstam
LGA
Proge
Prosmirr
Rarssom
LGA
Langer
Phager
Prodeess
Rarmion
LGc
Lamor
Pegge
Probmees
(d) Layer 4
(c) Layer 5
(f) Layer 6
Sancom
Coussion
Cousiian
Coussian
LCK
LCA
LCA
Ridge
Ridge
Ridge
Phocelss
Probeless
Phoceloss
X2U
XQU
Recoun
Loa
Lesso
Redge
Prodeaec
Rensum
Rodee
Prodeats
Ransum
LGa
Lecsa
Redge
Prodeiss
gg) Layer
(h) Layer
Layer
Sambom
LC4
LC4
LC4
.2250
10U
10U
X0U
Ronsoon
Lok
Lakao
Rogge
Ransium
Lol
Ruage
Rondom
Lesto
Rugel
Layer
(k) Layer II
Layer 12
Figure 7: This is extension of Figure Comparing average overlap of top 10-50 neurons across methods for XLMR
24

---

Table 16: Comparison of Neuron Vote score Probeless and Lasso under different number of consensus methods. In all cases, Probeless shows better results irrespective of the methods used for consensus.
Num of Consensus Methods
Consensus Methods
Probeless
Lasso
Gaussian
0.110
0.092
IoU
0.274
0.195
LCA
0.524
0.516
Ridge
0.270
0.237
Gaussian,
0.233
0.159
Gaussian, LCA
0.291
0.267
Gaussian, Ridge
0.202
0.170
LCA,IoU
0.357
0.300
Ridge
0.387
0.329
Ridge,IoU
0.292
0.227
Gaussian.LCA.Jog
0.346
0.289
Gaussian,L.CA.
0.356
0.321
Gaussian.Ridge.Iol
0.265
0.214
LCA, ICA,Ridge,loU
0.424
0.396
4
Gaussian.CCAARidgg..lol
0.382
0.324
Table 17: Comparison of Neuron Vote score of Probeless and IoU under different number of consensus
methods. all cases, Probeless shows better results irrespective of the methods used for consensus. Num of Consensus Methods Consensus Methods
Probeless
loU
Gaussian
0.110
0,085
IoU
0.449
0.195
Lasso-01
0.524
0,216
Ridge-01
0.270
0.160
Caussian.IoU
0.272
0.155
Coussian.Ridge---
0.291
0.158
Gaussian,Lasson
0.202
0.136
Lasso-O1,1o
0.404
0.206
Lasso-01 0idgoeo-01
0.498
0.209
Ridge-01.IoU
0.424
0.217
Gaussian,Ridge IoU
0.337
0.194
Gaussian, Caassian_Lasso-000 -010000110000 Ridge-01
0.356
0.200
Gaussian,Lasse Lasso-O01IIO
0.475
0.224
Lasso-01, Ridgel,,.dlccllloll
0.434
0.206
Gaussian, LuussianLasooRRiggoO...ood
0.426
0.221
25


Reviews:
[{"review_id": "x4kpHSGECm", "reviewer": "Reviewer_z745", "summary": "This paper proposes a standardized evaluation metric and benchmark for comparing various neuron interpretation methods, based on ideas from majority voting. The benchmark is based on the hypothesis that \"neurons that are commonly discovered by different interpretation methods are more informative than others\", and uses this hypothesis to rank & score methods by their alignment with the majority (in terms of the neurons they assign behaviors to). I discuss some thoughts/limitations of this hypothesis below (and the authors discuss it as well in their limitations section). The authors demonstrate empirically why the current commonly-used evaluation metric (fitting a classifier to the neurons to predict the concept) favors certain methods over others and suffers from the same common issues as probing classifiers in general. Then then propose two \"compatibility\"-based metrics, one set-based and one rank-based, for comparing the top-k neurons deemed informative for explaining a concept by each method, and provide in-depth analysis using the metrics to compare 6 different existing neuron interpretation methods (3 corpus-based, and 3 classifier-based).\n\nOverall, I believe this paper is sound, well-executed, thorough, and clear, and will serve a useful role in the neural model interpretability sub-community. Based on the authors' response to my below questions, I may be inclined to raise my score further.\n\nEdit: I read the authors' rebuttal and it addressed my concerns. I did not change my score as I originally insinuated above, in part due to finding validity in the other reviewers' critiques. I still think the paper should be accepted.", "questions": "- L12: colon instead of semicolon?\n- L20: ungrammatical sentence\n- Would be good to introduce the term \"probe\" somewhere in l.37-43 of the introduction (or earlier). You mention the \"Probeless\" classifier multiple times in the intro/abstract, but don't define this term anywhere.\n- L102: represents --> represent\n- L150: extra \"i\"\n- L205-206 describe AvgOverlap as set-based (not taking ranking into account); this contradicts lines 222-223, where I think the word \"ranking\" is meant as \"set\". Can you clarify?\n- I don't understand the role of Section 3.3 and its associated equation. Why wouldn't you just use Eqns. 7 and 8 with $|\\mathcal{M}| = 2$?\n- L255: the choice of hyperparameter values here seems like it could have a big impact on the results; did you do any search?\n- Table 2 is never referred to in the text", "limitations": "Yes- see one suggestion above.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "Originality & Significance:\n- The paper serves an important role in the mechanistic interpretability sub-community, and is likely to be of good value to this community.\n- I am not aware of any other attempts to organize methods in this manner, or propose a standardized benchmark.\n\nQuality & Clarity:\n- The experimental section is sound. The lack of ground-truth in makes evaluating explanation evaluation metric \"faithfulness\" difficult, but the authors have nonetheless provided a convincing set of results.\n- The paper is well-organized and well-written. The paper uses proper mathematical notation throughout.\n- The authors have promised to release the code upon publication.", "weaknesses": "Agreement with hypothesis underlying the benchmark:\n- I am not sure I am totally convinced by the claim that the most informative neurons will be discovered by the most methods, given that neurons that are easily discoverable can often be those which are perhaps more simplistic in what they encode (e.g., only serve a singular function rather than multiple, concept is encoded only in that singular neuron rather than spread over multiple, etc.) However, the granularity and assumptions underlying neuron interpretation methods is a nascent research question in the interpretability community overall, so I will not fault the authors for their hypothesis (i.e., I think the paper is appropriately scoped and the hypothesis is reasonable in the context of existing/popular neuron interpretation methods). It would be nice for the authors to discuss limitations of the hypothesis (or neuron interpretation methods in general) in the limitations section.\n \nMore minor:\n- The paper is lacking a related works section. While the methods studied in the paper are described in detail in Section 2, it would be nice to have some sort of summary paragraph of the field of neuron-level interpretation as a whole after the introduction to situate the work (and potentially mention other methods that are not tested in the paper). \n- There is some minor conflation it seems between the decision to treat each neuron interpretation method as producing a ranked list vs. performing binary set membership classification. This is particularly confusing with the classifier methods, which are directly trained to do the latter (4.1) but tested on the former via the NeuronVote method. However, the inclusion of the set-based metric, AvgOverlap, mitigates this issue to some extent. I think it would be good for the authors to clarify early in the paper that each method *can* be viewed as producing a ranking, even though this is not always how the methods have been proposed or used in practice, and that the inclusion/comparison of *both* evaluation metrics is designed to provide a view that does not unnecessarily favor one method over the other, by considering both the set membership and the rank views.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**R: ….It would be nice for the authors to discuss limitations of the hypothesis…**\n\n\nA: We appreciate your comment. Given the variety of methods we considered, we anticipate discovering neurons with diverse properties, including polysemous characteristics. For instance, the ElasticNet regularization (Section 2.2.3) is theoretically capable of identifying both singular and multiple function neurons. We also found in the pairwise analysis (Figure 1) that neuron interpretation methods discover partly different neurons. In other words, the discoverability of a neuron varies depending on the neuron interpretation method used.\nWe acknowledge the value of discussing these findings as it would offer insights into the potential limitations of each neuron interpretation method discussed in the paper and will provide future research directions. We have added the following limitations of interpretation methods in the paper.\n```\nA limitation of current neuron interpretation methods is that they do not explicitly target the discovery of neurons of diverse nature such as polysemantic, and superposition. Theoretically, ElasticNet regularization is capable of discovering neurons learning a singular function and multiple functions. The other methods such as Probeless are incapable of discovering multifunction neurons. An explicit modeling of neurons of different nature in a neuron interpretation method may result in discovering novel sets of neurons. \n```\n\n**R: The paper is lacking a related works section….**\n\n\nA: We agree that having a related work section will be useful to understand the work in the context of the other work on interpretability and neuron interpretation. We have added a related work section that provides a broad view of the field, clearly describes the scope of our paper and mentions other works on neuron interpretation. We have provided the related work section in the overall rebuttal.\n\n\n**R: I think it would be good … each method can be viewed as producing a ranking …**\n\n\nA: That’s a great suggestion. We will clarify this in the paper and explicitly mention that the evaluation metrics are designed in a way that won’t result in biased evaluation with respect to set-based and ranked-based methods. \n"}, {"review_id": "ESDur8vs40", "reviewer": "Reviewer_QcFR", "summary": "This paper provides a comparative analysis of six neuron interpretation methods utilizing diverse concepts across three distinct pre-trained models and introduces an evaluation framework predicated on voting theory. Importantly, it offers the first comprehensive examination of multiple neuron interpretation methods and strives to mitigate the challenges in this field.  The authors note similarities among the most proficient techniques within layers of neurons, with these resemblances remaining regardless of methodological deviations. It is thereby suggested, existing neuronal interpretation methodologies might have focused on a reciprocal group of top-performing neurons. \n\nThe paper also discusses the lack of no recognised evaluative metrics along with the lack of gold annotations by suggesting an evaluation strategy that consists of two compatible metrics with a pairwise comparison. This approach could facilitate the creation of a means to evaluate new neuron interpretation methods. ", "questions": "Do I understand it right that if the methods under study are sub-optimal, the voting results can also be sub-optimal? In other words, the voting method does not provide any indications of how well a method may perform. Rather, it simply allows for a comparison between them.\n\nConsidering that we are in the early phases of understanding the black-box nature of ML models, is it appropriate to start discussing benchmarks now? I have my reservations due to our lack of comprehensive hypotheses regarding their inner mechanisms. Hence, it presents a dilemma because it's as if we might not be in the position to start comparing or benchmarking. ", "limitations": "The authors provide some insightful limitations of thier work.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The major strength of the paper lies in its originality; being the first work to create a ranking metric using voting theory for comparing the various neuron interpretation methods available. This methos should allow for autumatic comparison between existing and new neuron actiavtion methos. Furthermore, it embarked on a comprehensive comparative analysis via this new evaluation framework", "weaknesses": "The primary weakness of the paper lies in its disregard for significant, influential studies previously published in the domain, thus lacking a comprehensive deliberation on an extensive range of neuron activation methods studied by the mechanistic interpretability community. In addition, the paper grievously overlooks the complexities associated with polysemantic, superposition and neuroplastic neuron behaviors encountered in neuron activation.\n\nWork worth considering are:\n\n1- Elhage, et al - https://transformer-circuits.pub/2021/framework/index.html\n\n2 - Olsson et al - https://transformer-circuits.pub/2022/in-context-learning-and-induction-heads/index.html\n\n3- Elhage, et al - https://transformer-circuits.pub/2022/in-context-learning-and-induction-heads/index.html\n\n4- Henighan et al - https://transformer-circuits.pub/2023/toy-double-descent/index.html\n\n5 - Foote et al - https://arxiv.org/pdf/2305.19911.pdf\n\n6 - Bills et al - https://openaipublic.blob.core.windows.net/neuron-explainer/paper/index.html \n\n7 - wang et al - arXiv preprint arXiv:2211.00593\n\n8 - Chan et al - https://www.alignmentforum.org/posts/JvZhhzycHu2Yd57RN/causal-scrubbing-a-method-for-rigorously-testing\n\n9 - Tenney et al - https://arxiv.org/pdf/2008.05122.pdf\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**R: re: missing references**\n\nA: Thank you for pointing out the missing references. We are certainly open to including a broader view of the interpretation field in the paper to enhance its scope. We acknowledge that some relevant references on neuron interpretation, such as Foote et al and Bills et al, appeared on arxiv in May 2023, and unfortunately, we couldn't include them in the paper at the time of submission. We have added the related work in the paper that provides a broad overview of the interpretation field and defines the scope of our work (included in the overall rebuttal). \n\n**R: …the paper grievously overlooks the complexities associated with polysemantic, superposition and neuroplastic neuron behaviors encountered in neuron activation.**\n\n\nA: Thank you for raising this point. It is indeed worth noting that the neuron interpretation studies in NLP, that come under the scope of this paper, have this limitation that they do not adequately consider polysemantic, superposition and neuroplastic neuron behaviors. Among the current methods, only ElasticNet regularization holds the theoretical expectation of identifying polysemantic neurons, and a few works have provided empirical evidence to support this claim. However, a detailed analysis in this particular line of work is currently lacking. We plan to address this in the limitations section to provide a comprehensive perspective on the research landscape. Thank you again for the invaluable feedback in refining the paper's clarity and purpose. \nFollowing is the text added to the paper to bring attention to the limitation of current neuron interpretation methods.\n\n\n```\nA limitation of current neuron interpretation methods is that they do not explicitly target the discovery of neurons of diverse nature such as polysemantic, and superposition. Theoretically, ElasticNet regularization is capable of discovering neurons learning a singular function and multiple functions. The other methods such as Probeless are incapable of discovering multifunction neurons. An explicit modeling of neurons of different nature in a neuron interpretation method may result in discovering novel sets of neurons. \n```\n\n**R: Do I understand it right that if the methods under study are sub-optimal, the voting results can also be sub-optimal?**\n\nA: Given the lack of an objective evaluation criteria or gold annotations, we believe that the next best thing is a consensus based method like the one we have proposed. Iif all the methods employed are sub-optimal, the overall voting outcome will also be sub-optimal. However, we have carefully selected a range of well-established and methodologically diverse neuron interpretation methods as well as concepts for our study, to have a variety of discovered neurons for comparison, and provide the best evaluation currently possible. Given a new interpretation method, our setup provides its evaluation from the perspective of other neurons discovered by several other methods. This is the case with the Gaussian method presented in the paper where its ranking score is quite low but better than Random selection of neurons and it encourages the inventors to perform further evaluation of their method. The conclusion and limitation section (Section 5) discussed this limitation in detail.\n\n**R: …early phases of understanding the black-box nature of ML models, is it appropriate to start discussing benchmarks now…**\n\nA: We respect the opinion of the reviewer.  Interpretability is a very wide field with various subfields and every subfield is at a different level of maturity. For example, there are hundreds of papers within the last five years on representation analysis while the work on mechanistic interpretability is in its early stages. Neuron Interpretation (that our work is aiming at) has seen a number of methods in the last five years, but any comparison between these has been cursory because of a lack of a benchmark. Without any comparison, it is difficult to assess the progress of neuron analysis methods, understand their limitations, and guide further research endeavors.We believe that a benchmark like this is a first step towards a standardized yardstick to move forward as a subfield.\n"}, {"review_id": "v3npEhvQWT", "reviewer": "Reviewer_x9Fa", "summary": "This work evaluates six different interpretation methods from a unified perspective. The authors focus on two challenges in this field: the absence of standard metrics and the lack of benchmarks. They propose two voting-based metrics to evaluate the compatibility among these six methods. Probeless consistently achieves the highest compatibility across all models based on their evaluation methodology. Further analytics experiments provide insight to the research community.", "questions": "1. Are there additional performance comparison experiments to practically demonstrate the effectiveness of this method? For example, try to deactivate the most important 10% of neurons calculated by each interpretation method, and then observe whether Probeless exhibits the most significant performance decline.\n\n2. To assess the efficacy of an interpretation method across various fields, is it essential to replicate multiple interpretation methods within those fields before applying the voting metrics? If so, this process can be quite challenging and should be acknowledged as a limitation in this work.", "limitations": "There is no potential negative societal impact.", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The findings of the interpretation methods mentioned in this paper may foster research on the topic.\n\n2. The idea of this paper is slightly novel.", "weaknesses": "1. The selected six methods need to be more novel. More recent advancements in this field may exist rather than relying on L1 & L2 regularization.\n\n2. The validation of the superiority of Probeless needs to be more comprehensive.\n\n3. The mentioned problems (metrics and benchmarks) were not effectively addressed, making this method difficult to be reproduced in future applications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**R: The selected six methods need to be more novel. More recent advancements in this field may exist rather than relying on L1 & L2 regularization.**\n\nA: We selected the widely used and established neuron interpretation methods for NLP models, drawing from the existing literature. Please see the recent published survey on neuron interpretation \n(https://direct.mit.edu/tacl/article/doi/10.1162/tacl_a_00519/113852/Neuron-level-Interpretation-of-Deep-NLP-Models-A). Our selection of neuron interpretation methods is based on it. \nHowever, kindly note that the crux of our contribution lies not in the selection of these interpretation methods but rather in the introduction of a novel evaluation framework. This framework is specifically designed to streamline the assessment of new interpretation methods and facilitate meaningful comparisons of their results. By offering this evaluation tool, we aim to foster advancements in the field of neuron interpretation of NLP models, and encourage further research in this important area.\n\n**R: The validation of the superiority of Probeless needs to be more comprehensive.**\n\nA: We conducted a large set of experiments using three pre-trained models, across all layers, and using 50 concepts of diverse linguistic annotations such as parts of speech tagging, chunking and semantic tagging. This makes approximately 2000 settings in total. All of our results showed Probeless to be the most consistent method. LCA is another competitive method to Probeless with the exception of last layer representations. Appendix Tables 6 and 7 present the results using semantic tagging and chunking tasks. We included the set of experiments that you have proposed and we would be happy to consider and discuss if there are any suggestions of more experiments.\n\n**R: The mentioned problems (metrics and benchmarks) were not effectively addressed**\n\nA: We kindly seek clarification on the matter. We have integrated all methods into a single codebase, shared the evaluation metric code, and provided discovered neurons associated with various concepts to ensure the  reproducibility of the results. We are confident that replicating our findings for future applications will be a straightforward process.\n\n**R: Q1: …additional experiments…  For example, try to deactivate the most important 10% of neurons calculated by each interpretation method, and then observe whether Probeless exhibits the most significant performance decline.**\n\nA: Thanks for proposing an interesting experiment. As suggested, we compared the compatibility score by iteratively removing top N neurons from the ranking of each method and calculated the compatibility score for the next top neurons. Figure 1 in the PDF file showed the average results across layer 1, 6 and 12. Probeless maintains the top or competitive to the top compatibility score for all three models, BERT, RoBERTa and XLMR. This shows that the top neurons discovered by Probeless are consistently better.\n\n**R: Q2 To assess the efficacy of an interpretation method across various fields, is it essential….**\n\nA: This is correct. The compatibility metric relies on the availability of several methods targeting a common goal. The metric is of significant value when gold standard annotations are not available and are harder to make. We have added the following text in the limitation section to acknowledge this limitation. \n\n```\nWhile the proposed framework is agnostic to methods used to produce ranking, in order to adapt it to other fields, it requires the presence of various methods targeting an identical goal.\n```\n"}, {"review_id": "IEGiRPOYAF", "reviewer": "Reviewer_F8Vs", "summary": "This work investigates interpretation methods in NLP that identify which neurons in a neural network are most related to particular concepts (e.g. a specific part of speech). The key idea is to compare how consistent one method's ranking of neurons is (w.r.t. a specific concept) with that of all other considered methods. The assumption is that the method that produces the most consistent rankings is the one that is closest to the true ranking of important neurons. The authors consider a number of interpretation methods and pretrained language models and show that across several settings, the method that is most consistent is Probeless (Antverg and Belinkov, 2021).", "questions": "How do the insights about which interpretability methods is the most consistent change according to the considered interpretability methods? For example, can the authors remove 1,..n-1 interpretability methods from the consensus calculation and compare the results with those from the full set of n interpretability methods? \n\nMinor questions / notes:\n\nL100: are the examples supposed to be country names? The provided examples are cities.\n\nThere are a lot more \"random\"concepts in a corpus than target concepts. Are the random concepts that are used for evaluation sampled such that they are equal in number to the target concepts? How consistent is this evaluation over different random samples?\n\nL185: why does the performance of a probe using random 100 neurons being higher than other neurons mean that the probe is memorizing? As the authors point out earlier in the manuscript, the concept knowledge is distributed so it makes sense that as one increases the number of random neurons that are being considered, the prediction performance would improve. If the authors were looking at randomly initialized models rather than pretrained models, then I would agree with their conclusion, but in the current setup, there seem to be alternate explanations for their results.\n", "limitations": "See under Weaknesses", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "- The work considers the problem of interpreting the neuron-level representations of neural networks, which seems to be a recently emerging subarea of interpretability within NLP. I'm not fully convinced by its utility but there are indeed several recent works in this area so perhaps there would be interest in this work\n- The paper is mostly well written and motivated", "weaknesses": "The main weakness is the assumption that consensus with existing interpretability methods is desirable. This consensus of course depends on the other considered interpretability methods, and it's not clear that the number of total interpretability methods (6) is sufficient to lead to reliable results that are based on consensus.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**R: …his consensus of course depends on the other considered interpretability methods, and it's not clear that the number of total interpretability methods (6) is sufficient to lead to reliable results…**\n\nA: We acknowledge the concern. We discussed the limitation of our approach in detail in Section 5. We would also like to emphasize the significance and critical nature of the problem we are addressing. Currently, there is no standardized metric available for comparing and evaluating neuron interpretation methods. This lack of standardization has led to scattered efforts in proposing new interpretation methods. Our work aims to address this gap by establishing a foundation for evaluating and comparing neuron interpretation methods, which will be continuously updated to include new interpretation methods.\nRegarding the choice of six interpretability methods: we did not explicitly reject an interpretation method from inclusion into our framework. The choice of six interpretability methods is based on the recently published survey on neuron interpretation (link: https://direct.mit.edu/tacl/article/doi/10.1162/tacl_a_00519/113852/Neuron-level-Interpretation-of-Deep-NLP-Models-A). Additionally, the six methods utilized in our paper were chosen for their theoretical diversity, resulting in a diverse array of discovered neurons.\nWe welcome any discussion on alternative methods that the reviewer believes could help address the limitations of evaluating neuron interpretation techniques. We are open to considering and incorporating such suggestions to further enhance the robustness of our approach.\n\n**R: For example, can the authors remove 1,..n-1 interpretability methods from the consensus calculation and compare the results with those from the full set of n interpretability methods?**\n\nA: Thank you for suggesting an insightful experiment. In the paper, we considered a leave-one-out strategy to calculate compatibility of each method so for a set of N methods, we considered one method as a test method and used the rest N-1 methods to serve as the database of sets of discovered neurons. What you have suggested is opposite to this. We tried it for the rebuttal. We considered a combination of one, two, three and four methods to serve as a database of the sets of neurons and calculate the compatibility scores of the other methods. The trends in Table 1, 2 and 3 showed consistent results to that reported in the paper. We observe that only the inclusion of Lasso in the database, when there is only one or two methods, provides an edge to LCA during evaluation. However, this effect is minimized with the inclusion of more methods (greater than 2) in the consensus database. Rest, we did not find any correlation between the presence of a method in the database with the high compatibility score of a test method. \n\n\n**R: L100: are the examples supposed to be country names? The provided examples are cities.**\n\nA: Thank you for pointing out the typo. We will fix it to be city names.\n\n\n**R: Are the random concepts that are used for evaluation sampled such that they are equal in number to the target concepts?**\n\nA: Yes, we randomly selected “random” concepts equal in number to the target concept. We ran three random selection runs and found the results to be consistent. \n\n\n**R: L185: why does the performance of a probe using random 100 neurons being higher than other neurons mean that the probe is memorizing?**\n\nA: Thank you for pointing out the error. It is indeed due to the distributiveness of knowledge of the concept. Durrani et al. (2020) presented an empirical evidence of this using the controlled task. We have corrected it in the paper. \n\n\n"}]

Author rebuttal:
We thank the reviewers for their insightful comments, questions and suggestions. We have incorporated their suggestions, and answered specific concerns below each review. At a high level, we have significantly overhauled the related work (*included below for reviewers z745 and QcFR*), and defined the scope of our work within the field. We have run additional experiments to further confirm the results (*pdf attached*), as well as revamped the limitations section to incorporate the thoughtful suggestions by the reviewers. Following is the related work section that we have added to the paper.

```
Related Work
The area of interpreting deep learning models constitutes a broad expanse of research. This section provides a synthesized overview of diverse interpretability subareas within deep learning models applied to Natural Language Processing (NLP), while also outlining the scope of our study.
Attribution Methods Feature importance and attribution methods endeavor to identify the contribution of input features to predictions. These methodologies predominantly rely on the gradient of the output concerning the input feature and determine input feature importance by evaluating the magnitude of gradient values (Denil et al., 2014; Sundararajan et al., 2017). Please see Danilevsky et al. (2020) for a comprehensive survey.
Counterfactual Intervention revolves around an intricate analysis of the interplay between input features and predictions. This approach involves manipulating inputs and quantifying resulting output alterations. Diverse intervention strategies, including erasing input words, removing multiple input words, and substituting input words with different meanings, have been scrutinized (Li et al., 2016b; Ribeiro et al., 2018).
Attention Weights Numerous investigations have been directed towards interpreting components of deep learning models at varying levels of granularity. For instance, attention weights have emerged as a viable metric to gauge the interrelation between input instances and model outputs (Martins & Astudillo, 2016; Vig, 2019). Along these lines, Geva et al. (2021) delved into the analysis of feedforward neural network components within the transformer model, revealing their functionality as key-value memories. Additionally, Voita et al. (2019) demonstrated that pruning many attention heads has minimal impact on performance.
Mechanistic Interpretability puts a focus into the reverse engineering of network weights to comprehend their behavior. Building upon the Distill Circuits thread, Elhage et al. (2021) investigated two-layered transformer models with attention blocks, identifying attention heads contributing to in-context learning. This understanding was further extended to larger transformer-based language models by Olsson et al. (2022). To enhance neuron interpretability, Elhage et al. (2022) introduced a Softmax Linear unit as an activation function replacement. Wang et al. (2022) attempted to bridge mechanistic interpretability findings in small networks to large ones, particularly GPT-2 small. Their approach involved iteratively tracing influential model components from predictions using causal intervention. They showcased the potential of mechanistic interpretability in understanding extensive models, while also highlighting associated challenges.
Representation Analysis involves probing network representations concerning predefined concepts, particularly linguistic, to quantify the extent of knowledge captured in these representations Belinkov et al. (2017); Conneau et al. (2018); Liu et al. (2019a); Tenney et al. (2019). This is often realized through training diagnostic classifiers for specific concepts, wherein classifier accuracy serves as an indicator of concept knowledge within representations. See Belinkov & Glass (2019) for a comprehensive survey.
Neuron Interpretation A more intricate form of representation analysis, termed neuron interpretation, delves into how knowledge is organized within the network (Sajjad et al., 2022b). This approach establishes connections between neurons and predefined concepts, offering insights into where and how specific concept knowledge is assimilated. Work done on neuron analysis can be broadly classified into 3 groups: Neuron visualization involves manual identification of patterns across a set of sentences (Li et al., 2016a; Karpathy et al., 2015).More recently Foote et al. (2023) proposed an automated approach to enhance interpretability of Large Language Models (LLMs) by extracting and visualizing individual neuron behaviors as interpretable graphs. Corpus-based Methods explore the role of a neuron through techniques such as ranking sentences in a corpus Kádár et al. (2017b), generating synthetic sentences Poerner et al. (2018) that maximize its activation, or computing neuron-level statistics over a corpus Mu & Andreas (2020); Suau et al. (2020); Antverg & Belinkov (2022). (Bills et al., 2023) recently proposed an algorithm to generate neuron explanations, simulating activations using a simulator model (an LLM), and scoring the results. Probing Methods identify salient neurons for a concept by training a classifier using neuron activations as features Radford et al. (2019); Lakretz et al. (2019); Dalvi et al. (2019) or fitting a multivariate Gaussian over all neurons and then extracting individual probes for single neurons Torroba Hennigen et al. (2020). In this paper, we focus on the neuron interpretation methods that take a concept as input and find neurons with respect to the concept. We considered all methods mentioned in the recent survey on neuron interpretation Sajjad et al. (2022b) in our study. We propose an evaluation framework to formalize the evaluation and comparison of results across methods. Moreover, we propose a novel method, MeanSelect and present a case study of using the evaluation framework.
```


Meta-review: The reviewers overall agree there is a significant contribution of neural interpretability. The authors have addressed reviewer concerns very well, but I highly recommend they ensure there are no further mistakes in their results or their presentation in the tables.
Decision: Accept (poster)