Title: Evaluating Neuron Interpretation Methods of NLP Models
Venue: NeurIPS 2023
Accepted: yes
Keywords: Neuron interpretation

NLP

Interpretability

Machine Learning
Abstract: Neuron interpretation offers valuable insights into how knowledge is structured within a deep neural network model. While a number of neuron interpretation methods have been proposed in the literature, the field lacks a comprehensive comparison among these methods. This gap hampers progress due to the absence of standardized metrics and benchmarks. The commonly used evaluation metric has limitations, and creating ground truth annotations for neurons is impractical. Addressing these challenges, we propose an evaluation framework based on voting theory. Our hypothesis posits that neurons consistently identified by different methods carry more significant information. We rigorously assess our framework across a diverse array of neuron interpretation methods. Notable findings include: i) despite the theoretical differences among the methods, neuron ranking methods share over 60% of their rankings when identifying salient neurons, ii) the neuron interpretation methods are most sensitive to the last layer representations, iii) Probeless neuron ranking emerges as the most consistent method.

Evaluating Neuron Interpretation Methods of NLP Models

Abstract

Neuron interpretation offers valuable insights into how knowledge is structured within a deep neural network model. While a number of neuron interpretation methods have been proposed in the literature, the field lacks a comprehensive comparison among these methods. This gap hampers progress due to the absence of standardized metrics and benchmarks. The commonly used evaluation metric has limitations, and creating ground truth annotations for neurons is impractical. Addressing these challenges, we propose an evaluation framework 1 based on voting theory. Our hypothesis posits that neurons consistently identified by different methods carry more significant information. We rigorously assess our framework across a diverse array of neuron interpretation methods. Notable findings include: i) despite the theoretical differences among the methods, neuron ranking methods share over 60% of their rankings when identifying salient neurons, ii) the neuron interpretation methods are most sensitive to the last layer representations, iii) Probeless neuron ranking emerges as the most consistent method.

1 Introduction

The advent of deep neural network (DNN) models and their success in Natural Language Processing (NLP) has opened a new research frontier to interpret the representations learned within these models. Representation analysis provides a holistic view of the knowledge learned within the representation. Whereas neuron analysis gives insight into how knowledge is structured within the representation. To this end, an ample amount of work has been done to understand the knowledge captured within the learned representations (Belinkov et al., 2017; Liu et al., 2019a; Tenney et al., 2019; Dalvi et al., 2022) and individual neurons (Karpathy et al., 2015; Kádár et al., 2017a; Dalvi et al., 2019).

Neuron analysis provides a fine-grained interpretation of the representation. It discovers salient neurons of the network w.r.t to a concept such as morphological or syntactic concepts (Lakretz et al., 2019; Durrani et al., 2020), and provides insight into how they may be used during inference (Lundberg & Lee, 2017; Dhamdhere et al., 2018). Beyond understanding the inner dynamics of the DNN models, neuron analysis has various potential applications such as network manipulation (Bau et al., 2019), domain adaptation (Antverg et al., 2022), model distillation (Dalvi et al., 2020) and architectural search (Prasanna et al., 2020).

A number of neuron analysis methods have been proposed in the literature to interpret deep NLP models. While these methods have shown promising results, there is a notable absence of systematic comparisons among them in existing studies. Conducting a fair comparison of neuron interpretation methods is crucial for comprehending the merits and limitations of each technique. Such comparisons

serve as a guiding light for the community, steering efforts towards the development of improved and more effective interpretation methodologies.

In this work, we study a variety of neuron interpretation methods under a controlled setting and provide a thorough comparative analysis. There are two main challenges in comparing neuron interpretation methods: i) absence of a standard evaluation metric , ii) lack of an evaluation benchmark (Sajjad et al., 2022b). The most common evaluation metric used in the neuron analysis literature is to train a classifier using the discovered neurons as features. The performance of the classifier in predicting the concept of interest serves as evidence of the quality of the discovered neurons, in other words, the correctness of the neuron interpretation method.

We argue that the classifier as an evaluation metric is suboptimal for two reasons: i) training a classifier provides an opportunity to learn the task and it may result in good performance irrespective of the quality of the neurons used as features (Hewitt & Liang, 2019), ii) the classifier as an evaluation method may favor the neuron interpretation methods that are methodologically similar (for instance, methods that use a classifier to discover salient neurons). Antverg & Belinkov (2021) shed light on the latter issue and showed that the methodological similarity between the neuron interpretation methods and the evaluation methods may result in unreasonably high accuracy scores. We empirically demonstrate the issue in Section 3. Another bottleneck in comparing neuron interpretation methods is the absence of evaluation benchmark . Neurons are distributive in nature and exhibit polysemy (capturing multiple concepts at the same time). Moreover, there can be different sets of neurons learning a concept, and there exists no single correct answer. It is therefore intricately challenging to create the ground truth annotations and infeasible to scale to a large set of models and concepts.

Given the lack of a standard evaluation metric and the infeasibility of creating ground truth annotations, how can we compare neuron interpretation methods? We rely on the voting theory (O'Connor &Robertson, 2003) that performs a systematic aggregation of results in order to achieve a consensus (Patrão Neves, 2016). It is effectively used in political, social, and machine-learning communities for cases wherein there are distinct preferences among voters (models). Examples of specific applications of voting theory include ranking search results obtained from various algorithms in information retrieval (Lin et al., 2017; Yilmaz et al., 2008), finding the best machine translation system using pair-wise ranking (Lapata, 2006) and aggregating results of several NLP tasks (Colombo et al., 2022).

We hypothesize that neurons that are commonly discovered by different interpretation methods are more informative than others , and may serve as a signal of correctness for the selected neurons. Motivated by this, we devise two voting-based compatibility metrics. Our first metric, AvgOverlap , is based on plurality voting (Cooper & Zillante, 2012). AvgOverlap computes an average overlap of the discovered neurons across all voters. Our second method, NeuronVote , inspired by Borda Count (Lippman, 2012), takes into account the ranking of neurons of each method while calculating the average pair-wise intersection over union with all the other methods. The compatibility metrics provide a single score for each method, where a higher score refers to a neuron interpretation method whose discovered neurons are more compatible with the other methods. We further extend the evaluation of neuron interpretation methods to pair-wise comparison, providing insights into how any two given methods relate in terms of their discovered neurons.

We conduct a comparative analysis of six neuron interpretation methods using diverse concepts consisting of morphological, semantic, and syntactic properties and using three pre-trained models. The analysis suggests that the best performing methods, despite their methodological differences, share more than 60% of the top neurons. Probeless ranking Antverg & Belinkov (2022) consistently exhibited the highest overlap with other methods. Because neurons in the last layer representations are highly correlated, they are the most challenging to interpret. Finally, we present a case study that demonstrates the usefulness of our evaluation methodology for any new method proposed in the future (Appendix 7.1). Our contributions are as follows:

We conduct the first thorough comparative analysis of a large set of neuron interpretation methods of NLP across 3 pre-trained models and using a diverse set of linguistic concepts.

Our proposed evaluation methodology consisting of two compatibility metrics and a pairwise comparison targets one of the critical limitations of the neuron interpretation studies.

We provide the evaluation methodology as a framework to facilitate future studies and comparison of methods.

2 Neuron Interpretation Methods

The goal of neuron interpretation methods is to rank N neurons with respect to some concept C . These methods can be broadly classified into two classes: corpus-based methods and probing-based methods (Sajjad et al., 2022b). The former discovers the role of neurons by aggregating statistics over neuron activations and the latter trains classifiers to achieve the same.

Let D be a dataset of sentences (represented as a list of tokens), where each token w appears in a specific context/sentence and has a contextual representation associated with it. Let z ( n, w ) be the activation of the n th neuron and the w th token in D . The neuron n ∈ N can be from any component of the original model, such as a specific layer. Also note that the activation value z ( n, w ) in context-dependent, so the same w can result in different z ( n, w ) values depending on context.

Aconcept C represents a property we want to discover neurons for and ˆ C represents a random concept, i.e. containing words that are not associated with the concept C . For example, C country = {Rome, Paris, London, New York, . . . } represent a concept of country names and ˆ C would consist of all other words in the corpus that are not country names. More formally, ˆ C = D\C .

Let R ( n, C ) represents the score of a neuron n with respect to a concept C . Each neuron interpretation method implements R , and provides a ranked list of neurons with respect to a concept. In the following, we present six interpretation methods 2 studied in this paper.

2.1 Corpus-based Methods

Corpus-based methods align neurons with a concept by accumulating co-occurrence patterns between neuron activations and the presence of the concept of interest. We use two corpus-based methods in this work presented as follows.

2.1.1 Probeless

The Probeless method (Antverg & Belinkov, 2021) obtains neuron rankings based on an accumulative strategy. The score of a given neuron n is defined as follows:

where µ ( C ) is the average of activations z ( n, w ) , w ∈ C . µ ( ˆ C ) is the average of activations over the random concept ˆ C . Note that the ranking for each neuron n is computed independently.

2.1.2 IoU Method

Mu & Andreas (2020) proposed IoU to generate compositional explanations for neurons. For each token in the dataset, they create i) a binary mask of a neuron by thresholding activations above a percentile and ii) a binary mask of a concept by checking its presence in the token. They compute Intersection over Union (IoU) over the two masks:

✶ ✶ where ✶ is the indicator function. z ( n, w ) > δ is a binary mask over neuron activations based on the threshold δ . w ∈ C is a binary mask, created by checking if the current word represents the concept C of interest. Mu & Andreas (2020) used IoU scores to generate alternative explanations for neurons on the task of image classification. Here we apply their method in the NLP domain to generate neuron ranking for any concept of interest. Like Probeless, the ranking for every neuron is computed independently of other neurons.

2.2 Probing Methods

Probing methods train a classifier using neuron activations as features for the task of predicting the concept of interest (Belinkov et al., 2017; Hupkes et al., 2018). The internals of the trained classifier (e.g. its weights) are used to rank the features (neurons).

2.2.1 Lasso Regularizer (L1)

Radford et al. (2019) trained a linear classifier with L1 regularization and used the weights of the classifier as a proxy for the importance of neurons for the given concept. The loss function of the classifier is as follows:

where, c ∈ {C , ˆ C} is the correct class for w , z N ( w ) is the vector of all neuron activations for word w , i.e. [ z (0 , w ) , z (1 , w ) , ... , z ( n, w ) ], and P θ ( c | z N ( w )) is the probability that word w belongs to the class c . The learned weights θ serve as the ranking of neurons. Specifically, R ( n, C ) is defined as the absolute weights for the class i.e. | θ C ( n ) | .

L1 adds the 'absolute value of magnitude' of the coefficient as a penalty term to the L ( θ ) , which shrinks the less important weights to zero, leading to sparse weight distribution. Radford et al. (2019) used this regularization to force the classifier to learn spiky weights, indicating the selection of a few specialized neurons learning a concept, while setting the majority of neurons' weight to zero. Such an assumption is useful in discovering focused neurons that learned one particular concept only.

2.2.2 Ridge Regularizer (L2)

Lakretz et al. (2019) used L2 regularization to train the linear classifier. The loss is as follows:

L2 forces weights to be close to zero (but not zero). It is useful to deal with multicollinearity (neurons that are highly correlated) scenarios, through constricting the coefficient while keeping all the features. Intuitively, this encourages grouping of features, thus discovering group neurons that jointly learn a concept. The score R ( n, C ) is computed in a similar fashion as the Lasso Regularizer method.

2.2.3 ElasticNet Regularizier (LCA)

Both L1 and L2 regularizations capture properties that are desirable when selecting the most important neurons. L1 facilitates sparsity, identifying focused neurons while L2 encourages identifying groups of highly correlated features into account. Dalvi et al. (2019) used ElasticNet regularization (Zou & Hastie, 2005) that balances the trade-off between them.

λ 1 and λ 2 are hyperparameters that are tuned to optimize the effect of L1 and L2 regularization.

2.2.4 Gaussian Classifier

Hennigen et al. (2020) assumes that neurons exhibit a Gaussian distribution. They fit a multivariate Gaussian, say P over all neurons. Since multivariate Gaussian is decomposable by nature, they are able to extract individual probes for any subset of input features without additional work. Formally, they are able to extract P F , where F ⊂ N . The classifier itself is trained using the Bayesian framework, specifically the maximum a posteriori estimate to compute the parameters θ . Once a multivariate Gaussian is trained, the neuron selection is performed in a greedy fashion:

In essence, probes for individual neurons are first extracted, and the neuron probe with the highest loglikelihood is considered the most important neuron. Next, probes are extracted for pairs of neurons, with one of them being the selected neuron. The pair with the highest log-likelihood then contributes the second most important neuron to the ranking. The full ranking is compared iteratively, with each step being a greedy selection of the next best neuron to add based on the log-likelihood. This method makes two fundamental assumptions: i) neuron activations follow a Gaussian distribution and ii) neuron ranking can be done in a greedy fashion.

3 Evaluation Methodology

3.1 Classifier Accuracy

The commonly used evaluation metric i.e. classification accuracy of the selected neurons is inadequate: i) because it favors the neuron ranking methods that are based on the probing framework (Antverg & Belinkov, 2021), ii) it is not clear whether the classifier performance is a reflection of the knowledge learned in the discovered neurons or it is due to the capacity of the classifier to learn and memorize the task (Hewitt & Liang, 2019; Zhang & Bowman, 2018).

We demonstrate these issues empirically, by using a linear classifier without regularization as an evaluation metric to evaluate various neuron interpretation methods. Given s discovered neurons with respect to a concept C , we train the classifier using these discovered neurons as features. The performance of the classifier serves as a measure of the correctness of s . Table 1 presents the average accuracy across all parts of speech (POS) concepts and across all layers of BERT for s = 30 , 50 , 70 , 100 . Random refers to a random selection of neurons. The details of the dataset and the experimental setup are provided in Section 4. A few notable observations are: the classifier accuracy is higher for the probing methods (Lasso and Ridge) than the corpus-based methods (Probeless and IoU), ii) the classifier has the capacity to memorize as a Random selection of neurons performs reasonably well when more than a certain number of neurons are used in the evaluation. These results complement the issues raised earlier i.e. the classifier may result in high performance irrespective of the correctness of the neurons, and it favors the interpretation methods that are methodologically similar to itself.

3.2 Compatibility Metrics

Given the absence of a good evaluation metric and gold annotations, we rely on consensus-based voting theory. The consensus serves as a tool to measure the relation between rankings. In NLP, it has been widely used in information retrieval to combine search results of various algorithms (Lin et al., 2017; Yilmaz et al., 2008), in machine translation to compare different systems' output using pair-wise ranking (Lapata, 2006) and to aggregate results of NLP tasks (Colombo et al., 2022).

One way to compare rankings is to perform pairwise comparison using rank correlation or similarity functions such as Kendall's τ (Kendall, 1938) and Spearman correlation (Daniel, 1990). These methods assume conjoint rankings where all rankings contain identical elements. In the case of neuron rankings, for every neuron interpretation method, we only consider the s top-ranked neurons with respect to a concept. The rationale behind this setting is to minimize the noise and randomness in rankings that may occur for neurons that are not learning the concept. This results in disjoint sets and the rank correlation methods can not be directly applied in such cases (Lin et al., 2017).

Motivated by the voting theory, we devise two metrics to compare disjoint rankings. Our first metric, AvgOverlap , is based on plurality voting (Cooper & Zillante, 2012). AvgOverlap computes the pair-wise average neuron overlap across methods. It is a set-based method and it does not consider the internal ranking of neurons into consideration. For example, the first top neuron and fifth top neuron in a ranking will get equal weight when compared with another ranking. Our second method, NeuronVote , inspired by Borda Count (Lippman, 2012), considers the order of neurons into consideration in comparing rankings. We call our methods compatibility metrics, which when given

a neuron ranking, provide its compatibility score with respect to other methods. The metrics give a high score to a method that is most aligned with the rankings of the other neuron interpretation methods. To build a deeper understanding of how any two methods relate in terms of the resulting neurons, we also present a pair-wise comparison of the discovered neurons across methods.

Formally, let S m = n 1 , n 2 , ..., n s represent the s top-ranked neurons for a method m , where s is a hyperparameter that can be adjusted to vary the number of top neurons selected for evaluation. We describe the compatibility metrics and the pair-wise comparison method as follows.

3.2.1 Average Overlap

AvgOverlap scores an interpretation method based on its average neuron overlap with other methods. We define the overlap o ( m 1 , m 2 ) between two methods as the intersection over union of their respective top-neuron sets:

̸

where M is the set of all methods. A large value of AvgOverlap means that the ranking of that method is well aligned with the ranking of other neuron interpretation methods.

3.2.2 NeuronVote

AvgOverlap treats each neuron equally in a set of discovered neurons and ignores the ranking assigned by the methods. However, intuitively a neuron interpretation method whose top choice is endorsed by all other methods should get a higher score than an interpretation method whose 10th top choice is endorsed by all other methods. To take this into account, we follow the Borda count strategy (Lippman, 2012) and propose NeuronVote . It considers each neuron's ranking in the voting process. We first create S best , an aggregated ranking based on the position of individual neurons in the rankings of various methods.

where argsort is an ascending sort function and index gives the position of neuron n in the ranked list. The accumulated S best represents an aggregation on weighted votes, with neurons appearing at the top of various methods' ranked lists getting a higher weight than others. Following the notation in AvgOverlap , we define NeuronVote m i = o ( S m i , S best ) . The method with high NeuronVote implies that its ranking of neurons is most endorsed by other interpretation methods.

3.3 Pairwise Comparison

The compatibility metrics provide a holistic evaluation of a neuron interpretation method with respect to other methods. We further extend the analysis to pair-wise comparison to understand how the discovered neurons of a method related to another method. The pair-wise comparison is calculated as an intersection between the output of two methods i.e. o ( m 1 , m 2 ) = S m 1 ∧ S m 2

4 Evaluation

4.1 Settings

We consider three 12-layered pre-trained models: BERT-base-cased (BERT, Devlin et al., 2019), RoBERTa-base-cased (RoBERTa, Liu et al., 2019b) and XLM-Roberta-base (XLMR, Conneau et al., 2019). Each layer consists of 768 neurons. We consider concepts from three linguistic tasks: parts of speech tags (POS, Marcus et al., 1993), semantic tags (SEM, Abzianidze et al., 2017) and syntactic chunking (Chunking) using CoNLL 2000 shared task dataset (Tjong Kim Sang & Buchholz, 2000). Dataset details are provided in Appendix 7.2. We consider every tag as a concept and identify neurons with respect to the concept. In the case of classifier-based methods, we use a binary classification setup where the contextualized representation of words belonging to the concept serves as positive

AvgOverlap is defined as:

examples. We randomly select the negative class words from the rest of the data, equal in size to the positive class examples. We drop the concepts that have less than 200 examples to ensure stable training. We split the binary classification dataset into train/dev/test splits of 70/15/15 percent. We use λ 1 = 0 . 01 and λ 2 = 0 . 01 for regularization-based methods. We present the results of POS only in the main paper and provide the rest in Appx. 7.3.

Number of top neurons: We did not optimize the number of neurons s selected by each method as this would result in a varying number of neurons and will make the compatibility score incompatible across methods. We consider s = 10 , 30 , 50 which covers a diverse range to generalize the findings.

Baseline: Wegenerate a random ranking of neurons to compare the behavior of a neuron interpretation method with a random selection of neurons. We refer to it as Random .

Procedure: We extract layer-wise contextualized representations of words in the dataset. For every layer representation, we discover the top neurons with respect to a concept using each neuron interpretation method. We follow a leave-one-out strategy to calculate the compatibility score of each method with respect to other methods. More specifically, we consider one neuron interpretation method as a test method and select the rest as the database of the sets of top neurons. We then calculate the AvgOverlap and NeuronVote scores of the test method.

4.2 Compatibility Scores

Table 2 presents the average compatibility scores across all concepts and when selecting the top 10, 30, and 50 neurons from layers 1, 6, and 12.

AvgOverlap and NeuronVote show a similar pattern: While the scale of the compatibility score is different between both metrics, they are consistent in highlighting the top interpretation methods i.e. Lasso, LCA, and Probeless. In the rest of the paper, we mainly present the results of NeuronVote , and discuss the AvgOverlap only when it is necessary to show a different trend.

Overall comparison across methods: Table 3 presents the NeuronVote scores averaged over all POS concepts and when selecting the top 10, 30, and 50 neurons. The bold, underline, and dashed numbers represent the first, second, and third best compatibility scores respectively. Probeless consistently achieves the highest compatibility across all layers and models. In other words, the neurons selected by Probeless have the highest overlap with the neurons selected by other methods. We did not observe a similar trend for IoU which is also a corpus-based method.

LCA and Lasso achieve the second-best compatibility score (see underlined and dashed numbers in Table 3). Ridge, although methodologically similar to LCA and Lasso, did not consistently show high compatibility across models. We discuss these methods later in Section 4.3. Gaussian achieved the worst compatibility score which is lower by 0.354 points from the best NeuronVote score on layer 1 and is only 0.128 better than Random for BERT. The result of Gaussian presents an interesting scenario where the compatibility score suggests that the neurons identified by Gaussian are least similar to all other methods. However, the difference with Random, although small, suggests that the selection of neurons is not random. The low results of Gaussian are inline with Antverg & Belinkov (2021) where they found that the Gaussian method memorizes the probing task and it may not provide the most faithful ranking of neurons. These scores, therefore, suggest that further investigation may be required to confirm the efficacy of the Gaussian method.

Anotable point about the top 3 methods (Probeless, LCA, and Lasso) is their methodological diversity, despite of which they selected similar neurons to other methods. On the other hand, methodologically similar techniques such as IoU and Probeless did not result in similar compatibility scores. This shows that the compatibility metric is not biased by methodological similarity, which was one of the primary concerns with the commonly used evaluation metrics.

Layer-wise trend: Ethayarajh (2019) showed that each layer is different in terms of representation geometry. Sajjad et al. (2022a) revealed that due to the geometry of the representation space, particularly in the last layers, knowledge of a concept may not be readily available. We hypothesize that the performance of certain neuron interpretation methods may vary based on the nature of the representation space. Our compatibility methods facilitate quantitative evidence to support this hypothesis by showing that certain neuron analysis methods suffer in selecting the most compatible neurons from the higher layers. LCA and Lasso both showed a substantial drop in their compatibility scores for the 12th layer (for example, Table 3: NeuronVote LCA - 0.544 for layer 6 and 0.391 for layer 12 for BERT). In contrast, Probeless did not show any substantial difference in the scores of the last layers and earlier layers ( NeuronVote : Probeless - 0.550 vs 0.515 for layer 6 and 12 respectively for BERT). This further support Probeless as the most reliable method and encourages further investigation into the robustness of Lasso and LCA with varying representational space.

Compatibility across models: The overall trend of top-performing neuron interpretation methods is similar across BERT, RoBERTa, and XLMR. One notable difference is the compatibility score of Ridge for the last layer of the RoBERTa (0.510) and XLM-R (0.439) models which is substantially higher compared to the BERT model (0.360). Moreover, it achieved the second best score for XLM-R after Probeless and is substantially better than LCA and Lasso. Ridge prefers correlated features in contrast to Lasso, which prefers a few spiky features highly predictive of the concept. Based on the results, we hypothesize that the last layers of RoBERTa and XLM-R consist of highly correlated neurons and Ridge is effective in discovering salient neurons of correlated nature. LCA balances between spiky and correlated neurons, and results in higher scores than Lasso in these cases.

CA

(a) Layer 1

(b) Layer 6

(c) Layer 12

Random

OT

0.02

0.02

0.02

0.02

0.02

0.02

Random

1.0

0.03

0.02

0.02

0.02

0.02

0.02

Random

1.0

0.02

0.02

0.02

0.02

0.02

0.02

0.02

0.06

0.06

0.11

0.05

0.05

0.03

01

0.1

0.09

0.1

0.1

0.09

0.02

0.09

0.08

0.15

0.09

0.07

LCA

0.02

0.06

0.58

0.26

0.66

0.24

LCA

0.02

0.1

1.0

0.64

0.29

0.62

0.24

0.02

0.09

0.43

0.2

0.51

0.21

0.02

0.06

850

0.26

0.55

0.2

0.02

0.09

0.64

0.28

0.52

0.21

0.02

0.08

0.43

0.22

0.38

0.16

0.02

0.11

0.26

0.26

1.0

0.23

0.17

0.02

0.1

0.29

0.28

1.0

0.29

0.16

Ridge-

0.02

0.15

0.2

0.22

1.0

0.24

0.13

Probeless

0.02

0.05

0.66

0.55

0.23

1.0

0.25

Probeless

0.02

0.1

0.62

0.52

0.29

1.0

0.28

Probeless

0.02

0.09

0.51

0.38

0.24

0.3

IOU

0.02

0.05

0.24

0.2

0.17

0.25

1.0

IOU-

0.02

0.09

0.24

0.21

0.16

0.28

1.6

IOU-

0.02

0.07

0.21

0.16

0.13

0.3

1.0

CA

CA

Random

Probeless

Random

Probeless

Random

Probeless

Random selection of neurons: The compatibility score of Random serves as a baseline that provides a perspective of the correctness of a newly proposed neuron interpretation method. It gave substantially lower scores than all neuron interpretation methods (Table 3). Even though the Gaussian showed the lowest compatibility score among interpretation methods, the difference with Random adds a factor of confidence to its ranking. Moreover, a low compatibility score for Random also implies that each of the other methods are bringing in important neurons to some degree.

4.3 Pairwise Comparison

The compatibility metrics provide a single score to understand an interpretation method in the context of a large set of other interpretation methods. We further extend the analysis to pair-wise comparison and provide insights into the relationship between different methods. Figure 1 presents the heatmaps comparing the average overlap of the top 10 - 50 neurons across methods using BERT. The light color in the heatmap shows higher matches while the dark color shows lower matches. We added Random - a random selection of neurons as a baseline. It showed an average overlap of less than 3% only (see the first row in heatmaps).

Comparing Regularization-based Methods: LCA and Lasso showed a high overlap of up to 64% in the discovered neurons (see Figure 1(a), 1(b)). However, Ridge results in a lower overlap of at most

29% compared to other methods. Note that the Ridge regularization does not force the coefficients of neurons to zero, thus considers most of the neurons during the training phase. The resulting top neurons may represent correlated neurons capturing similar and redundant knowledge (Dalvi et al., 2020) and may not be a representative of the neurons with the most predictive power with respect to the understudied concept. Lasso prefers individual neurons most predictive of the concept and ElasticNet (LCA) provides a good balance between selecting individual neurons and correlated neurons highly predictive of the concept.

Corpus-based vs. Regularization-based Methods: Probeless and IoU both directly rely on the activation value of a neuron. However, we did not observe a consistent overlap between their discovered neurons (at most 30% in Figure1(c)). In comparison to other classes of methods, Probeless showed a better and consistent overlap with LCA and Lasso. For example, in Figure 1(a) we observed an overlap of 66% between Probeless and LCA, and 55% between Probeless and Lasso. The relatively high overlap of Probeless with methods using Lasso points towards the selection of identical focused neurons by these methods despite their methodological differences.

Outliers: Both Ridge and IoU show a smaller but consistent overlap with most of the methods. As discussed earlier, Ridge works well for correlated features which may not be highly predictive of the concept. IoU relies on activation values and it may select neurons that consistently activate higher than the threshold irrespective of the concept. Gaussian did not show substantial overlap with any methods or with any combination of methods. Further investigation is needed to evaluate its' efficacy.

5 Related Work

The area of interpreting deep learning models constitutes a broad expanse of research. This section provides a synthesized overview of diverse interpretability subareas within deep learning models applied to Natural Language Processing (NLP), while also outlining the scope of our study.

Feature importance and attribution methods identifies the contribution of input features to predictions. These methodologies predominantly rely on the gradient of the output concerning the input feature and determine input feature importance by evaluating the magnitude of gradient values (Denil et al., 2014; Sundararajan et al., 2017; Danilevsky et al., 2020).

Counterfactual Intervention revolves around an intricate analysis of the interplay between input features and predictions. This approach involves manipulating inputs and quantifying resulting output alterations. Diverse intervention strategies, including erasing input words and substituting words with different meanings have been used (Li et al., 2016b; Ribeiro et al., 2018).

Attention Weights: Numerous investigations have been directed towards interpreting components of deep learning models at varying levels of granularity. For instance, attention weights have emerged as a viable metric to gauge the interrelation between input instances and model outputs (Martins & Astudillo, 2016; Vig, 2019). Along these lines, Geva et al. (2021) delved into the analysis of feedforward neural network components within the transformer model, revealing their functionality as key-value memories. Additionally, Voita et al. (2019) demonstrated that pruning many attention heads has minimal impact on performance.

Mechanistic Interpretability puts a focus on reverse engineering of network weights to comprehend their behavior (Nanda et al., 2023). Building upon the Distill Circuits thread, Elhage et al. (2021) investigated two-layered transformer models with attention blocks, identifying attention heads contributing to in-context learning. This understanding was further extended to larger transformer-based language models by Olsson et al. (2022). To enhance neuron interpretability, Elhage et al. (2022) introduced a Softmax Linear unit as an activation function replacement. Wang et al. (2022a) attempted to bridge mechanistic interpretability findings in small networks to large ones, particularly GPT-2 small. Their approach involved iteratively tracing influential model components from predictions using causal intervention. They showcased the potential of mechanistic interpretability in understanding extensive models, while also highlighting associated challenges. Similarly, Bricken et al. (2023) uses a sparse autoencoder to disentangle polysemantic neurons.

Representation Analysis involves probing network representations concerning predefined concepts, to quantify the extent of knowledge captured in these representations (Conneau et al., 2018; Liu et al., 2019a; Tenney et al., 2019; Durrani et al., 2019; Arps et al., 2022). This is often realized through

training diagnostic classifiers for linguistic concepts, wherein classifier accuracy serves as an indicator of concept knowledge within representations. See Belinkov & Glass (2019) for a comprehensive survey.

Neuron Interpretation A more intricate form of representation analysis, termed neuron interpretation, delves into how knowledge is structured within the network (Sajjad et al., 2022b). This approach establishes connections between neurons and predefined concepts, offering insights into where and how specific concept knowledge is assimilated. Work done on neuron analysis can be broadly classified into three groups: Neuron visualization involves manual identification of patterns across a set of sentences (Li et al., 2016a; Karpathy et al., 2015). More recently Foote et al. (2023) proposed an automated approach to enhance interpretability of Large Language Models (LLMs) by extracting and visualizing individual neuron behaviors as interpretable graphs. Corpus-based Methods explore the role of a neuron through techniques such as ranking sentences in a corpus (Kádár et al., 2017b), generating synthetic sentences (Poerner et al., 2018) that maximize its activation, or computing neuron-level statistics over a corpus (Mu & Andreas, 2020; Suau et al., 2020; Antverg & Belinkov, 2022). Bills et al. (2023); Mousi et al. (2023) proposed the use of LLM to interpret neurons. Probing Methods identify salient neurons for a concept by training a classifier using neuron activations as features (Radford et al., 2019; Lakretz et al., 2019; Durrani et al., 2022) or fitting a multivariate Gaussian over all neurons and then extracting individual probes for single neurons (Hennigen et al., 2020).

A number of works identify neurons with respect to the output class (Wang et al., 2022b; Dai et al., 2022). They are effective in finding neurons that play a role in the prediction. In this paper, we focus on the neuron interpretation methods that take a concept as input and find neurons with respect to the concept. We propose an evaluation framework to formalize the evaluation and comparison of results across methods. Moreover, we propose a novel method, MeanSelect and present a case study of using the evaluation framework.

6 Conclusion and Limitation

We provided a thorough comparative analysis of neuron interpretation methods in NLP. We overcame the challenge of the lack of a standard evaluation metric and gold annotations by proposing an evaluation strategy consisting of two compatibility metrics and a pair-wise comparison. In addition to developing a capability to evaluate a new neuron analysis method, we presented various insights into existing neuron interpretation methods. For example, due to the correlated nature of last layer representations, they are most challenging for interpretation methods. The selection of top neurons overlaps substantially irrespective of the methodological differences among techniques. We made the evaluation framework available to the research community.

Limitations: Our methodology relied on consensus and thus inherits the limitations of that paradigm. The consensus may mislead the evaluation under certain settings e.g. if the majority of the voters form a lobby then they can skew the results of the consensus. This scenario can happen if a set of methods always produce close to identical ranking, they will cause a decrease in the compatibility score of a new method that produces a ranking different from theirs. Another possible issue is if a new method discovers an entirely different set of neurons than the ones discovered by other methods. The compatibility score of such a method will be low. We intend to mitigate these issues by including a large number of diverse neuron interpretation methods that are theoretically different from each other. We believe that under these settings, the scenario of an entirely different set of discovered neurons is more theoretical than practical. We further present a pair-wise analysis that provides an analysis at a more granular level and bring insight into how a method compares with other methods. The pairwise comparison heatmap will highlight if a set of methods form a lobby.

Another limitation of current neuron interpretation methods is that they do not explicitly target the discovery of neurons of diverse nature such as polysemantic, and superposition. Theoretically, only the ElasticNet regularization is capable of discovering neurons learning a singular function and multiple functions. The other methods such as Probeless are incapable of discovering multifunction neurons. None of the neuron interpretation methods explicitly identify superposition neurons. Explicit modeling of neurons of different nature in a neuron interpretation method may result in discovering novel sets of neurons. The evaluation methods including our proposed framework do not explicitly compare neurons of different types.

References

Abzianidze, L., Bjerva, J., Evang, K., Haagsma, H., van Noord, R., Ludmann, P., Nguyen, D.-D., and Bos, J. The parallel meaning bank: Towards a multilingual corpus of translations annotated with compositional meaning representations. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics , EACL '17, pp. 242-247, Valencia, Spain, 2017.

Antverg, O. and Belinkov, Y. On the pitfalls of analyzing individual neurons in language models. In International Conference on Learning Representations , 2021.

Antverg, O. and Belinkov, Y. On the pitfalls of analyzing individual neurons in language models. In International Conference on Learning Representations , 2022. URL https://openreview.net/ forum?id=8uz0EWPQIMu .

Antverg, O., Ben-David, E., and Belinkov, Y. Idani: Inference-time domain adaptation via neuronlevel interventions, 2022. URL https://arxiv.org/abs/2206.00259 .

Arps, D., Samih, Y., Kallmeyer, L., and Sajjad, H. Probing for constituency structure in neural language models. In Findings of the Association for Computational Linguistics: EMNLP 2022 , pp. 6738-6757, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.findings-emnlp.502. URL https://aclanthology.org/ 2022.findings-emnlp.502 .

Bau, A., Belinkov, Y., Sajjad, H., Durrani, N., Dalvi, F., and Glass, J. Identifying and controlling important neurons in neural machine translation. In International Conference on Learning Representations , 2019. URL https://openreview.net/forum?id=H1z-PsR5KX .

Belinkov, Y. and Glass, J. Analysis methods in neural language processing: A survey. Transactions of the Association for Computational Linguistics , 7:49-72, March 2019. doi: 10.1162/tacl_a_00254. URL https://www.aclweb.org/anthology/Q19-1004 .

Belinkov, Y., Durrani, N., Dalvi, F., Sajjad, H., and Glass, J. What do Neural Machine Translation Models Learn about Morphology? In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL) , Vancouver, July 2017. Association for Computational Linguistics. URL https://aclanthology.coli.uni-saarland.de/pdf/P/P17/P17-1080.pdf .

Bills, S., Cammarata, N., Mossing, D., Tillman, H., Gao, L., Goh, G., Sutskever, I., Leike, J., Wu, J., and Saunders, W. Language models can explain neurons in language models. https: //openaipublic.blob.core.windows.net/neuron-explainer/paper/index.html , 2023.

Bricken, T., Templeton, A., Batson, J., Chen, B., Jermyn, A., Conerly, T., Turner, N., Anil, C., Denison, C., Askell, A., Lasenby, R., Wu, Y., Kravec, S., Schiefer, N., Maxwell, T., Joseph, N., HatfieldDodds, Z., Tamkin, A., Nguyen, K., McLean, B., Burke, J. E., Hume, T., Carter, S., Henighan, T., and Olah, C. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread , 2023. https://transformer-circuits.pub/2023/monosemanticfeatures/index.html.

Colombo, P., Noiry, N., Irurozki, E., and CLEMENCON, S. What are the best systems? new perspectives on NLP benchmarking. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processing Systems , 2022. URL https://openreview.net/ forum?id=kvtVrzQPvgb .

Conneau, A., Kruszewski, G., Lample, G., Barrault, L., and Baroni, M. What you can cram into a single vector: Probing sentence embeddings for linguistic properties. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL) , July 2018.

Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzmán, F., Grave, E., Ott, M., Zettlemoyer, L., and Stoyanov, V. Unsupervised cross-lingual representation learning at scale. CoRR , abs/1911.02116, 2019. URL http://arxiv.org/abs/1911.02116 .

Cooper, D. and Zillante, A. A comparison of cumulative voting and generalized plurality voting. In Public Choice , 2012. doi: 10.1007/s11127-010-9707-5.

Dai, D., Dong, L., Hao, Y., Sui, Z., Chang, B., and Wei, F. Knowledge neurons in pretrained transformers. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pp. 8493-8502, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.acl-long.581. URL https://aclanthology. org/2022.acl-long.581 .

Dalvi, F., Durrani, N., Sajjad, H., Belinkov, Y ., Bau, D. A., and Glass, J. What is one grain of sand in the desert? analyzing individual neurons in deep nlp models. In Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence (AAAI, Oral presentation) , January 2019.

Dalvi, F., Sajjad, H., Durrani, N., and Belinkov, Y . Analyzing redundancy in pretrained transformer models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP-2020) , Online, November 2020.

Dalvi, F., Khan, A. R., Alam, F., Durrani, N., Xu, J., and Sajjad, H. Discovering latent concepts learned in BERT. In International Conference on Learning Representations , 2022. URL https: //openreview.net/forum?id=POTMtpYI1xH .

Dalvi, F., Durrani, N., and Sajjad, H. Neurox library for neuron analysis of deep nlp models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics: System Demonstrations , pp. 75-83, Toronto, Canada, July 2023. Association for Computational Linguistics.

Daniel, W. Applied Nonparametric Statistics . Duxbury advanced series in statistics and decision sciences. PWS-KENT Pub., 1990. ISBN 9780534919764. URL https://books.google.ca/ books?id=0hPvAAAAMAAJ .

Danilevsky, M., Qian, K., Aharonov, R., Katsis, Y., Kawas, B., and Sen, P. A survey of the state of explainable AI for natural language processing. In Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing , pp. 447-459, Suzhou, China, December 2020. Association for Computational Linguistics. URL https://www.aclweb.org/anthology/2020. aacl-main.46 .

Denil, M., Demiraj, A., and de Freitas, N. Extraction of salient sentences from labelled documents. CoRR , abs/1412.6815, 2014. URL http://arxiv.org/abs/1412.6815 .

Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , Minneapolis, Minnesota, 2019. Association for Computational Linguistics.

Dhamdhere, K., Sundararajan, M., and Yan, Q. How important is a neuron? CoRR , abs/1805.12233, 2018. URL http://arxiv.org/abs/1805.12233 .

Durrani, N., Dalvi, F., Sajjad, H., Belinkov, Y., and Nakov, P. One size does not fit all: Comparing NMT representations of different granularities. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pp. 1504-1516, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1154. URL https: //www.aclweb.org/anthology/N19-1154 .

Durrani, N., Sajjad, H., Dalvi, F., and Belinkov, Y. Analyzing individual neurons in pretrained language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pp. 4865-4880, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.emnlp-main.395. URL https: //www.aclweb.org/anthology/2020.emnlp-main.395 .

Durrani, N., Dalvi, F., and Sajjad, H. Linguistic correlation analysis: Discovering salient neurons in deepnlp models, 2022. URL https://arxiv.org/abs/2206.13288 .

Elhage, N., Nanda, N., Olsson, C., Henighan, T., Joseph, N., Mann, B., Askell, A., Bai, Y., Chen, A., Conerly, T., DasSarma, N., Drain, D., Ganguli, D., Hatfield-Dodds, Z., Hernandez, D., Jones, A., Kernion, J., Lovitt, L., Ndousse, K., Amodei, D., Brown, T., Clark, J., Kaplan, J., McCandlish, S., and Olah, C. A mathematical framework for transformer circuits. Transformer Circuits Thread , 2021. https://transformer-circuits.pub/2021/framework/index.html.

Elhage, N., Hume, T., Olsson, C., Nanda, N., Henighan, T., Johnston, S., ElShowk, S., Joseph, N., DasSarma, N., Mann, B., Hernandez, D., Askell, A., Ndousse, K., Drain, D., Chen, A., Bai, Y., Ganguli, D., Lovitt, L., Hatfield-Dodds, Z., Kernion, J., Conerly, T., Kravec, S., Fort, S., Kadavath, S., Jacobson, J., Tran-Johnson, E., Kaplan, J., Clark, J., Brown, T., McCandlish, S., Amodei, D., and Olah, C. Softmax linear units. Transformer Circuits Thread , 2022. https://transformercircuits.pub/2022/solu/index.html.

Ethayarajh, K. How contextual are contextualized word representations? comparing the geometry of BERT, ELMo, and GPT-2 embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , pp. 55-65, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19-1006. URL https://www.aclweb.org/anthology/D19-1006 .

Foote, A., Nanda, N., Kran, E., Konstas, I., Cohen, S., and Barez, F. Neuron to graph: Interpreting language model neurons at scale, 2023.

Geva, M., Schuster, R., Berant, J., and Levy, O. Transformer feed-forward layers are key-value memories. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pp. 5484-5495, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.emnlp-main.446. URL https: //aclanthology.org/2021.emnlp-main.446 .

Hennigen, L. T., Williams, A., and Cotterell, R. Intrinsic probing through dimension selection. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pp. 197-216, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/ v1/2020.emnlp-main.15. URL https://www.aclweb.org/anthology/2020.emnlp-main.15 .

Hewitt, J. and Liang, P. Designing and interpreting probes with control tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , pp. 2733-2743, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19-1275. URL https://aclanthology.org/D19-1275 .

Hupkes, D., Veldhoen, S., and Zuidema, W. Visualisation and 'diagnostic classifiers' reveal how recurrent and recursive neural networks process hierarchical structure, 2018.

Kádár, Á., Chrupała, G., and Alishahi, A. Representation of linguistic form and function in recurrent neural networks. Computational Linguistics , 43(4):761-780, December 2017a. doi: 10.1162/ COLI_a_00300. URL https://www.aclweb.org/anthology/J17-4003 .

Kádár, A., Chrupała, G., and Alishahi, A. Representation of linguistic form and function in recurrent neural networks. Computational Linguistics , 43(4):761-780, 2017b.

Karpathy, A., Johnson, J., and Fei-Fei, L. Visualizing and understanding recurrent networks. arXiv preprint arXiv:1506.02078 , 2015.

Kendall, M. A new measure of rank correlation. Biometrika , 1938.

Lakretz, Y., Kruszewski, G., Desbordes, T., Hupkes, D., Dehaene, S., and Baroni, M. The emergence of number and syntax units in LSTM language models. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pp. 11-20, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1002. URL https: //www.aclweb.org/anthology/N19-1002 .

Lapata, M. Automatic evaluation of information ordering: Kendall's tau. Computational Linguistics , 32(4):471-484, 2006. doi: 10.1162/coli.2006.32.4.471. URL https://aclanthology.org/ J06-4002 .

Li, J., Chen, X., Hovy, E., and Jurafsky, D. Visualizing and understanding neural models in NLP. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pp. 681-691, San Diego, California, June 2016a. Association for Computational Linguistics. doi: 10.18653/v1/N16-1082. URL https://www.aclweb.org/anthology/N16-1082 .

Li, J., Monroe, W., and Jurafsky, D. Understanding neural networks through representation erasure. CoRR , abs/1612.08220, 2016b. URL http://arxiv.org/abs/1612.08220 .

Lin, Z., Li, Y., and Guo, X. Consensus of rankings. CoRR , abs/1704.08464, 2017. URL http: //arxiv.org/abs/1704.08464 .

Lippman, D. Voting Theory, Math in Society . Pierce College Ft Steilacoom, 2012.

Liu, N. F., Gardner, M., Belinkov, Y., Peters, M. E., and Smith, N. A. Linguistic knowledge and transferability of contextual representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pp. 1073-1094, Minneapolis, Minnesota, June 2019a. Association for Computational Linguistics. URL https://www.aclweb.org/anthology/ N19-1112 .

Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. Roberta: A robustly optimized BERT pretraining approach. CoRR , abs/1907.11692, 2019b. URL http://arxiv.org/abs/1907.11692 .

Lundberg, S. M. and Lee, S.-I. A unified approach to interpreting model predictions. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 30 , pp. 4765-4774. Curran Associates, Inc., 2017. URL http://papers.nips.cc/paper/ 7062-a-unified-approach-to-interpreting-model-predictions.pdf .

Marcus, M. P., Santorini, B., and Marcinkiewicz, M. A. Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics , 19(2):313-330, 1993. URL https: //www.aclweb.org/anthology/J93-2004 .

Martins, A. and Astudillo, R. From softmax to sparsemax: A sparse model of attention and multi-label classification. In Balcan, M. F. and Weinberger, K. Q. (eds.), Proceedings of The 33rd International Conference on Machine Learning , volume 48 of Proceedings of Machine Learning Research , pp. 1614-1623, New York, New York, USA, 20-22 Jun 2016. PMLR. URL https://proceedings.mlr.press/v48/martins16.html .

Mousi, B., Durrani, N., and Dalvi, F. Can llms facilitate interpretation of pre-trained language models? In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP) , Singapore, dec 2023. Association for Computational Linguistics.

Mu, J. and Andreas, J. Compositional explanations of neurons. CoRR , abs/2006.14032, 2020. URL https://arxiv.org/abs/2006.14032 .

Na, S., Choe, Y. J., Lee, D., and Kim, G. Discovery of natural language concepts in individual units of CNNs. CoRR , abs/1902.07249, 2019. URL http://arxiv.org/abs/1902.07249 .

Nanda, N., Chan, L., Lieberum, T., Smith, J., and Steinhardt, J. Progress measures for grokking via mechanistic interpretability. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=9XFSbDPmdW .

O'Connor, J. and Robertson, E. The mactutor history of mathematics archive, 2003.

Olsson, C., Elhage, N., Nanda, N., Joseph, N., DasSarma, N., Henighan, T., Mann, B., Askell, A., Bai, Y., Chen, A., Conerly, T., Drain, D., Ganguli, D., Hatfield-Dodds, Z., Hernandez, D., Johnston, S., Jones, A., Kernion, J., Lovitt, L., Ndousse, K., Amodei, D., Brown, T., Clark, J., Kaplan, J., McCandlish, S., and Olah, C. In-context learning and induction heads. Transformer Circuits Thread , 2022. https://transformer-circuits.pub/2022/in-context-learning-and-induction-heads/index.html.

Patrão Neves, M. Consensus , pp. 19-29. Encyclopedia of Global Bioethics, Springer, 06 2016. doi: 10.1007/978-3-319-05544-2_119-1.

Poerner, N., Roth, B., and Schütze, H. Interpretable textual neuron representations for NLP. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP , pp. 325-327, Brussels, Belgium, November 2018. Association for Computational Linguistics. doi: 10.18653/v1/W18-5437. URL https://www.aclweb.org/anthology/ W18-5437 .

Prasanna, S., Rogers, A., and Rumshisky, A. When BERT Plays the Lottery, All Tickets Are Winning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pp. 3208-3229, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.emnlp-main.259. URL https://aclanthology.org/2020.emnlp-main. 259 .

Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. OpenAI blog , 1(8):9, 2019.

Ribeiro, M. T., Singh, S., and Guestrin, C. Anchors: High-precision model-agnostic explanations. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence , AAAI'18/IAAI'18/EAAI'18. AAAI Press, 2018. ISBN 978-1-57735-800-8.

Sajjad, H., Alam, F., Dalvi, F., and Durrani, N. Effect of post-processing on contextualized word representations. In Proceedings of the 29th International Conference on Computational Linguistics , pp. 3127-3142, Gyeongju, Republic of Korea, October 2022a. International Committee on Computational Linguistics. URL https://aclanthology.org/2022.coling-1.277 .

Sajjad, H., Durrani, N., and Dalvi, F. Neuron-level Interpretation of Deep NLP Models: A Survey. Transactions of the Association for Computational Linguistics , 2022b.

Suau, X., Zappella, L., and Apostoloff, N. Finding experts in transformer models. CoRR , abs/2005.07647, 2020. URL https://arxiv.org/abs/2005.07647 .

Sundararajan, M., Taly, A., and Yan, Q. Axiomatic attribution for deep networks. In Proceedings of the 34th International Conference on Machine Learning - Volume 70 , ICML'17, pp. 3319-3328. JMLR.org, 2017.

Tenney, I., Das, D., and Pavlick, E. BERT rediscovers the classical NLP pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pp. 4593-4601, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19-1452. URL https://www.aclweb.org/anthology/P19-1452 .

Tjong Kim Sang, E. F. and Buchholz, S. Introduction to the CoNLL-2000 shared task chunking. In Fourth Conference on Computational Natural Language Learning and the Second Learning Language in Logic Workshop , 2000. URL https://www.aclweb.org/anthology/W00-0726 .

Vig, J. Visualizing attention in transformer-based language representation models. CoRR , abs/1904.02679, 2019. URL http://arxiv.org/abs/1904.02679 .

Voita, E., Talbot, D., Moiseev, F., Sennrich, R., and Titov, I. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pp. 5797-5808, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19-1580. URL https://www.aclweb.org/anthology/P19-1580 .

Wang, K., Variengien, A., Conmy, A., Shlegeris, B., and Steinhardt, J. Interpretability in the wild: a circuit for indirect object identification in gpt-2 small, 2022a.

Wang, X., Wen, K., Zhang, Z., Hou, L., Liu, Z., and Li, J. Finding skill neurons in pre-trained transformer-based language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , pp. 11132-11152, Abu Dhabi, United Arab Emirates, December 2022b. Association for Computational Linguistics. doi: 10.18653/v1/2022.emnlp-main.765. URL https://aclanthology.org/2022.emnlp-main.765 .

Yilmaz, E., Aslam, J. A., and Robertson, S. A new rank correlation coefficient for information retrieval. In Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval , SIGIR '08, pp. 587-594, New York, NY, USA, 2008. Association for Computing Machinery. ISBN 9781605581644. doi: 10.1145/1390334.1390435. URL https://doi.org/10.1145/1390334.1390435 .

Zhang, K. and Bowman, S. Language modeling teaches you more than translation does: Lessons learned through auxiliary syntactic task analysis. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP , pp. 359-361, Brussels, Belgium, November 2018. Association for Computational Linguistics. doi: 10.18653/v1/W18-5448. URL https://www.aclweb.org/anthology/W18-5448 .

Zou, H. and Hastie, T. Regularization and variable selection via the elastic net. Journal of the Royal Statistical Society, Series B , 67:301-320, 2005.

7 Appendix

(a) BERT

(b) RoBERTa

(c) XLMR

Layer 1 -

0.09

0.46

0.41

0.23

0.66

0.26

Layer 1 -

0.08

0.35

0.31

0.28

0.46

0.3

Layer 1

0.02

0.28

0.25

0.22

0.44

0.26

layer6-

0.12

0.49

0.42

0.28

0.62

0.32

layer6-

0.12

0.34

0.31

0.35

0.51

0.29

layer6-

0.02

0.36

0.33

0.31

0.49

0.27

layer 12

0.12

0.28

0.24

0.22

0.54

0.32

layer 12-

0.12

0.19

0.17

0.31

0.34

0.28

layer 12

0.02

0.21

0.19

0.26

0.29

0.25

Probeless

Probeless

Probeless

7.1 Case Study

Mean Select: We propose a new corpus-based method, MeanSelect , for neuron ranking as a case-study to illustrate how researchers can use our proposed methodology. Kádár et al. (2017b) generated explanations for neurons by extracting top-N 5-gram context for each neuron based on the magnitude of their activations, followed by human annotation. Na et al. (2019) removed the human-in-the-loop by extracting concepts of various granularities from a parsed tree and aligned highly activating neurons to the concepts. Inspired by these works, we propose a novel method, MeanSelect , that generates a ranking of neurons with respect to a concept. The intuition is that a neuron learning a concept will have consistently high activations across different contexts where that concept appears. However, there may be a neuron that always activates with high value irrespective of the concept. A difference in the mean value of the neuron activating the concept and all other concepts will provide the true behavior of the neuron for the concept. Following our notation in Section 2, the score of a given neuron n is defined as follows:

where µ ( C ) is the average, n max is the max and n min is the min of activations z ( n, w ) where w ∈ C , and µ ( ˆ C ) is the average of activations over the random concept set.

Compatibility Metrics Table 4 shows the compatibility score of MeanSelect using the representation of 3 different layers and compared it with the Random selection of neurons. The high compatibility scores show that MeanSelect discovers neurons that are endorsed by other neuron interpretation methods. This serves as a measure of confidence in the proposed method. Moreover, one may observe that the method has a relatively lower score for the last layer compared to other layers, giving insight into potential improvements that can be made to the behavior of this method.

Pairwise Comparison The pairwise comparison of methods further provides insights into how the new method relates to other methods in terms of the resulting neurons. Figure 2 shows the heatmaps of three pre-trained models. MeanSelect has the highest overlap with the Probeless method and except Gaussian, it shows an overlap of at least 0.23 points with other methods. While the high overlap of MeanSelect with Probeless is not surprising given both are based on similar intuitions, the overlap with LCA and L1 shows that the method is selecting a diverse set of neurons captured by a variety of methods. Similar to the discussion on compatibility score, here we observe a substantial overlap drop in the last layer and this highlights the potential vulnerability of the method to certain representations.

7.2 Concept Datasets

We consider concepts from three linguistic tasks: parts of speech tags (POS, Marcus et al., 1993), semantic tags (SEM, Abzianidze et al., 2017) and syntactic chunking (Chunking) using CoNLL 2000 shared task dataset (Tjong Kim Sang & Buchholz, 2000). For the POS dataset, we used 20 concepts

which have a total dataset size of 40137. These concepts include VBG (777), VBZ (908), NNPS (204), DT (4015), TO (1177), CD (1935), JJ (2836), PRP (801), MD (463), RB (1348), VBP (534), VB (1244), NNS (3021), VBN (1082), POS (433), IN (5039), NN (6660), CC (1220), NNP (4698), and VBD (1742).

For the SEM dataset, we used three concepts which have a total dataset size of 120941. These concepts include IST (72240), NOW (24137) and EXS (24564). We used 10 concepts from Chunking which have a total dataset size of 220606. These concepts include B-ADJP (2493), B-ADVP (5081), B-NP (67285), B-PP (26005), B-VP (26078), I-ADJP (805), I-ADVP (532), I-NP (77368) I-PP (339) and I-VP (14620). For all the datasets used in the experiments, we use train/valid/test split 70%, 15% and 15%.

7.3 Results

7.3.1 Semantic Tagging Concepts

For the SEM dataset, we sample 20000 sentences for experimental validation with train/valid/test split 70%, 15% and 15%. We select three tags: IST (intersective), NOW (present tense) and EXS (untensed simple event). Table 5 presents the average NeuronVote scores across three models. We observed identical trends to that of POS i.e. Probeless is the most consistent method, LCA and Lasso are second best methods but they suffer on the last layers.

7.4 Chunking Concepts

Figures 3 and 4 show layer-wise results for the two voting methods proposed in the paper, across the three understudied models (BERT, RoBERTa and XLM-R). The results show that voting methods consistently rank the Probeless method as the most compatible in terms of neuron rankings across the layers. We are including detailed results in Tables 5-10 give detailed results with exact numbers.

(a) BERT

(b) RoBERTa

(c) XLMR

0.35

0.30

0.4

0.4

AvgUverlap

0.25

AvgOverlap

0.3

AvgOverlap

0.3

0.20

0.15

Random

0.2

Random

0.2

Random

0.10

LCA

LCA

LCA

0.1

0.1

LasSO

0.05

Probeless

Probeless

Probeless

一

loU

0.0

loU

0.0

loU

0

2

4

6

8

10

12

0

2

4

6

8

10

12

0

2

4

6

8

10

12

Layer

Layer

Layer

(a) BERT

(b) RoBERTa

(c) XLMR

0.6

0.6

0.6

0.5

0.5

0.5

NeuronVote

0.4

NeuronVote

0.4

Random

NeuronVote

0.4

Random

LCA

LCA

0.3

0.3

0.3

0.2

Random

0.2

Probeless

loU

0.2

Probeless

loU

LCA

0.1

0.1

0.1

Probeless

0.0

loU

0.0

0.0

o

2

4

6

8

10

12

0

2

4

6

8

10

12

0

2

4

6

8

10

12

Layer

Layer

Layer

(a) Layer 1

(b) Layer 2

(c) Layer 3

(d) Layer 4

(e) Layer 5

(f) Layer 6

(g) Layer 7

(h) Layer 8

(i) Layer 9

(j) Layer 10

(k) Layer 11

(l) Layer 12

Random

1.0

0.02

0.02

0.02

0.02

0.02

0.02

Random

1.0

0.02

0.02

0.02

0.02

0.02

0.02

Random

1.0

0.01

0.02

0.02

0.02

0.02

0.02

0.02

0.06

0.06

0.11

0.05

0.05

Gaussian-

0.02

1.0

01

0.1

0.12

0.09

0.07

0.01

1.0

0.1

0.1

0.12

0.1

0.07

LCA

0.02

0.06

0T

0.58

0.26

0.66

0.24

0.02

0.1

1.0

0.59

0.29

0.6

0.26

0.02

0.1

0.57

0.3

0.6

0.26

0.02

0.06

0.58

0.26

0.55

0.2

0.02

0.1

0.59

0.26

0.5

0.22

0.02

0.1

0.57

0.27

0.5

0.22

0.02

0.11

0.26

0.26

0.23

0.17

0.02

0.12

0.29

0.26

0.29

0.16

0.02

0.12

0.3

0.27

EO

0.17

Probeless

0.02

0.05

0.66

0.55

0.23

1.0

0.25

Probeless

0.02

0.09

0.6

0.5

0.29

0.28

Probeless

0.02

0.1

0.6

0.5

0.3

0

0.3

IOU

0.02

0.05

0.24

0.2

0.17

0.25

1.0

IOU-

0.02

0.07

0.26

0.22

0.16

0.28

1.0

IOU-

0.02

0.07

0.26

0.22

0.17

0.3

1.0

CA

CA

CA

Random

Probeless

Random

Probeless

Random

Probeless

Random

10

0.02

0.02

0.02

0.02

0.02

0.02

Random

1.0

0.02

0.02

0.02

0.03

0.02

0.02

Random

1.0

0.03

0.02

0.02

0.02

0.02

0.02

0.02

0.11

0.11

0.13

0.11

Gaussian-

0.02

1.0

0.1

0.1

0.13

0.1

0.08

0.03

1.0

0.1

0.09

0.1

0.1

0.09

LCA

0.02

0.11

1.0

0.59

0.33

0.59

0.27

0.02

0.1

1.0

0.6

0.33

0.6

0.26

LCA

0.02

0.1

OT

0.64

0.29

0.62

0.24

0.02

0.11

0.59

1.0

0.3

0.51

0.24

0.02

0.1

0.6

1号

0.32

0.52

0.24

0.02

0.09

0.64

0.28

0.52

0.21

0.02

0.13

0.33

0.3

0.33

0.19

0.03

0.13

0.33

0.32

0.32

0.18

0.02

0.1

0.29

0.28

0.29

0.16

Probeless

0.02

0.11

0.59

0.51

0.33

19

0.3

Probeless

0.02

0.1

0.6

0.52

0.32

0.29

Probeless

0.02

0.1

0.62

0.52

0.28

IOU-

0.02

0.07

0.27

0.24

0.19

0.3

1.0

IOU-

0.02

0.08

0.26

0.24

0.18

0.29

Q1

IOU-

0.02

0.09

0.24

0.21

0.16

0.28

1.0

CA

.CA

Random

Probeless

Random

Probeless

Random

Probeless

Random

1.0

0.03

0.02

0.02

0.02

0.02

0.02

Random

1.0

0.03

0.02

0.01

0.01

0.02

0.02

Random

1.0

0.02

0.02

0.02

0.02

0.02

0.02

0.03

0.13

0.13

0.14

0.13

0.08

00

OT

0.12

0.13

0.16

0.13

0.07

0.02

1.0

0.12

0.11

0.16

0.13

0.09

LCA

0.02

0.13

1.0

0.6

0.3

0.58

0.25

0.02

0.12

1.0

0.55

0.3

0.56

0.25

0.02

0.12

1.0

0.51

0.26

0.51

0.22

0.02

0.13

0.6

1.0

0.3

0.52

0.23

0.01

0.13

0.55

0.28

0.46

0.22

0.02

0.11

0.51

0.24

0.43

0.18

0.02

0.14

0.3

0.3

0.3

0.17

0.01

0.16

0.3

0.28

0.3

0.16

0.02

0.16

0.26

0.24

0.28

0.16

Probeless

0.02

0.13

0.58

0.52

0.3

0.31

Probeless

0.02

0.13

0.56

0.46

0.3

0.3

Probeless

0.02

0.13

0.51

0.43

0.28

0.29

IOU

0.02

0.08

0.25

0.23

0.17

0.31

1.0

IOU-

0.02

0.07

0.25

0.22

0.16

0.3

Q1

IOU-

0.02

0.09

0.22

0.18

0.16

0.29

CA

Random

Probeless

Random

Probeless

Random

Probeless

Random

1.0

0.02

0.02

0.02

0.02

0.02

0.02

Random

1.0

0.01

0.02

0.02

0.02

0.02

0.02

Random

1.0

0.02

0.02

0.02

0.02

0.02

0.02

0.02

0.1

0.1

0.14

0.11

0.09

0.01

OT

0.11

0.1

0.16

0.11

0.08

0.02

1.0

0.09

0.08

0.15

0.09

0.07

LCA

0.02

0.1

1.0

0.54

0.22

0.51

0.24

0.02

0.11

1.0

0.51

0.21

0.49

0.23

0.02

0.09

1.0

0.43

0.2

0.51

0.21

0.02

0.1

0.54

1.0

0.22

0.44

0.2

0.02

0.1

0.51

1.0

0.2

0.43

0.2

0.02

0.08

0.43

0.22

0.38

0.16

0.02

0.14

0.22

0.22

0.23

0.16

0.02

0.16

0.21

0.2

0.23

0.17

0.02

0.15

0.2

0.22

0.24

0.13

Probeless

0.02

0.11

0.51

0.44

0.23

0.3

Probeless

0.02

0.11

0.49

0.43

0.23

0.29

Probeless

0.02

0.09

0.51

0.38

0.24

IOU

0.02

0.09

0.24

0.2

0.16

0.3

1.0

IOU-

0.02

0.08

0.23

0.2

0.17

0.29

Q1

IOU-

0.02

0.07

0.21

0.16

0.13

0.3

OT

CA

LCA

Jor

Random

Probeless

Random

Probeless

Random

Probeless

(a) Layer 1

(b) Layer 2

(c) Layer 3

(d) Layer 4

(e) Layer 5

(f) Layer 6

(g) Layer 7

(h) Layer 8

(i) Layer 9

(j) Layer 10

(k) Layer 11

(l) Layer 12

Random

10

0.02

0.02

0.02

0.02

0.02

0.02

Random

1.0

0.02

0.02

0.02

0.01

0.02

0.02

Random

1.0

0.02

0.02

0.02

0.03

0.02

0.02

0.02

0.08

0.08

0.13

0.09

0.06

Gaussian-

0.02

1.0

0.11

0.1

0.15

0.11

0.07

0.02

1.0

0.12

0.1

0.15

0.13

0.08

LCA

0.02

0.08

0T

0.51

0.34

0.56

0.22

0.02

0.11

1.0

0.52

0.36

0.56

0.2

0.02

0.12

0.5

0.36

0.53

0.2

0.02

0.08

0.51

0.29

0.47

0.19

0.02

0.1

0.52

0.31

0.48

0.19

0.02

0.1

0.5

0.3

0.47

0.17

0.02

0.13

0.34

0.29

0.38

0.18

0.01

0.15

0.36

0.31

0.43

0.2

0.03

0.15

0.36

0.3

0.44

0.22

Probeless

0.02

0.09

0.56

0.47

0.38

1.0

0.27

Probeless

0.02

0.11

0.56

0.48

0.43

OT

0.26

Probeless

0.02

0.13

0.53

0.47

0.44

0.27

IOU

0.02

900

0.22

0.19

0.18

0.27

1.0

IOU-

0.02

0.07

0.2

0.19

0.2

0.26

1.0

IOU-

0.02

0.08

0.2

0.17

0.22

0.27

1.0

LCA

LCA

LCA

Random

Probeless

Random

Probeless

Random

Probeless

Random

10

0.02

0.01

0.02

0.01

0.01

0.02

Random

1.0

0.02

0.01

0.02

0.02

0.02

0.02

Random

1.0

0.02

0.02

0.02

0.02

0.02

0.02

0.02

0.11

0.1

0.15

0.12

Gaussian-

0.02

1.0

0.11

0.1

0.16

0.12

0.07

0.02

1.0

0.11

0.1

0.17

0.12

0.07

LCA

0.01

0.11

1.0

0.51

0.35

0.51

0.19

LCA

0.01

0.11

1.0

0.5

0.34

0.51

0.18

0.02

0.11

OT

0.5

0.32

0.5

0.19

0.02

0.1

0.51

1.0

0.3

0.45

0.18

0.02

0.1

0.5

1.0

0.3

0.45

0.17

0.02

0.1

0.5

0.29

0.45

0.17

0.01

0.15

0.35

0.3

0.45

0.22

0.02

0.16

0.34

E'0

0.43

0.21

0.02

0.17

0.32

0.29

0.42

0.21

Probeless

0.01

0.12

0.51

0.45

0.45

0.27

Probeless

0.02

0.12

0.51

0.45

0.43

0.28

Probeless

0.02

0.12

0.5

0.45

0.42

0.27

IOU-

0.02

0.07

0.19

0.18

0.22

0.27

1.0

IOU-

0.02

0.07

0.18

0.17

0.21

0.28

Q1

IOU-

0.02

0.07

0.19

0.17

0.21

0.27

1.0

CA

LCA

LCA

Random

Probeless

Random

Probeless

Random

Probeless

Random

1.0

0.02

0.02

0.02

0.02

0.02

0.02

Random

1.0

0.02

0.02

00

0.02

0.02

0.03

Random

1.0

0.01

0.01

0.02

0.01

0.01

0.01

0.02

0.11

0.1

0.16

0.12

0.09

0.02

OT

0.11

0.11

0.15

0.12

0.08

0.01

1.0

0.11

0.1

0.17

0.12

0.09

0.02

0.11

1.0

0.52

0.3

0.51

0.2

0.02

0.11

1.0

0.52

0.31

0.53

0.19

0.01

0.11

1.0

0.54

0.33

0.53

0.17

0.02

0.1

0.52

1.0

0.28

0.46

0.17

0.03

0.11

0.52

1.0

0.28

0.46

0.17

0.02

0.1

0.54

0.31

0.47

0.16

0.02

0.16

0.3

0.28

0.39

0.2

0.02

0.15

0.31

0.28

0.39

0.2

0.01

0.17

0.33

0.31

0.21

Probeless

0.02

0.12

0.51

0.46

0.39

0.27

Probeless

0.02

0.12

0.53

0.46

0.39

0.26

Probeless

0.01

0.12

0.53

0.47

0.4

0.25

IOU

0.02

0.09

0.2

0.17

0.2

0.27

1.0

IOU-

0.03

0.08

0.19

0.17

0.2

0.26

Q1

IOU-

0.01

0.09

0.17

0.16

0.21

0.25

LCA

Random

Probeless

Random

Probeless

Random

Probeless

Random

1.0

0.02

0.02

0.02

0.02

0.02

0.02

Random

1.0

0.02

0.03

00

0.02

0.02

0.03

Random

1.0

0.02

0.02

0.02

0.02

0.02

0.02

0.02

16

0.1

0.1

0.18

0.11

0.09

0.02

OT

0.09

0.1

0.16

0.11

0.09

0.02

1.0

0.08

0.08

0.15

0.12

0.09

LCA

0.02

0.1

1.0

0.53

0.3

0.52

0.17

LCA

0.09

1.0

0.52

0.29

0.49

0.15

0.02

0.08

1.0

0.32

0.25

0.37

0.11

0.02

0.1

0.53

1.0

0.27

0.45

0.15

0.03

0.1

0.52

1.0

0.27

0.46

0.14

0.02

0.08

0.32

0.24

0.34

0.1

0.02

0.18

0.3

0.27

0.37

0.19

0.02

0.16

0.29

0.27

0.38

0.19

0.02

0.15

0.25

0.24

0.49

0.2

Probeless

0.02

0.11

0.52

0.45

0.37

0.24

Probeless

0.02

0.11

0.49

0.46

0.38

0.22

Probeless

0.02

0.12

0.37

0.34

0.49

0.19

IOU

0.02

0.09

0.17

0.15

0.19

0.24

1.0

IOU-

0.03

0.09

0.15

0.14

0.19

0.22

Q1

IOU-

0.02

0.09

0.11

0.1

0.2

0.19

LCA

LCA

Random

Probeless

Random

Probeless

Random

Probeless

(a) Layer 1

(b) Layer 2

(c) Layer 3

(d) Layer 4

(e) Layer 5

(f) Layer 6

(g) Layer 7

(h) Layer 8

(i) Layer 9

(j) Layer 10

(k) Layer 11

(l) Layer 12

Random

1.0

0.02

0.02

0.02

0.03

0.02

0.02

Random

1.0

0.02

0.02

0.02

0.02

0.02

0.02

Random

1.0

0.02

0.02

0.02

0.02

0.02

0.02

0.02

0.02

0.03

0.02

0.02

0.02

Gaussian-

0.02

1.0

0.03

0.03

0.02

0.03

0.02

0.02

1.0

0.02

0.02

0.02

0.02

0.02

LCA

0.02

0.02

0

BEO

0.21

0.46

0.17

0.02

0.03

10

0.52

0.27

0.56

0.17

0.02

0.02

0.56

0.3

0.56

0.17

0.02

0.03

0.38

10

0.2

0.38

0.15

0.02

0.03

0.52

0.24

0.48

0.15

0.02

0.02

0.56

0.27

0.47

0.14

0.03

0.02

0.21

0.2

0.29

0.15

0.02

0.02

0.27

0.24

0.3

0.13

0.02

0.02

0.3

0.27

0.33

0.15

Probeless

0.02

0.02

0.46

BEO

0.29

1.0

0.24

Probeless

0.02

0.03

0.56

0.48

0.3

0

0.21

Probeless

0.02

0.02

0.56

0.47

0.33

0.21

IOU

0.02

0.02

0.17

0.15

0.15

0.24

1.0

IOU-

0.02

0.02

0.17

0.15

0.13

0.21

1.0

IOU-

0.02

0.02

0.17

0.14

0.15

0.21

1.0

CA

CA

LCA

JOL

Random

Probeless

Random

Probeless

Random

Probeless

Random

1.0

0.02

0.02

0.02

0.01

0.02

0.02

Random

1.0

0.02

0.02

0.02

0.02

0.02

0.02

Random

1.0

0.02

0.02

0.02

0.02

0.02

0.02

0.02

0.02

0.02

0.02

0.02

0.02

Gaussian-

0.02

1.0

0.02

0.02

0.02

0.02

0.02

0.02

1.0

0.02

0.02

0.02

0.02

0.02

LCA

0.02

0.02

1.0

0.56

0.33

0.57

0.17

0.02

0.02

1.0

0.57

0.32

0.57

0.18

LCA

0.02

0.02

1.0

0.57

0.31

0.55

0.19

0.02

0.02

0.56

0.28

0.47

0.14

0.02

0.02

0.57

1号

0.28

0.48

0.16

0.02

0.02

0.57

0.29

0.48

0.17

0.01

0.02

0.33

0.28

0.36

0.16

0.02

0.02

0.32

0.28

0.36

0.16

0.02

0.02

0.31

0.29

0.36

0.18

Probeless

0.02

0.02

0.57

0.47

0.36

0.22

Probeless

0.02

0.02

0.57

0.48

0.36

0.23

Probeless

0.02

0.02

0.55

0.48

0.36

0.24

IOU-

0.02

0.02

0.17

0.14

0.16

0.22

1.0

IOU-

0.02

0.02

0.18

0.16

0.16

0.23

IOU-

0.02

0.02

0.19

0.17

0.18

0.24

1.0

LCA

LCA

1O

Random

Probeless

Random

Probeless

Random

Probeless

Random

1.0

0.02

0.02

0.02

0.02

0.02

0.02

Random

1.0

0.02

0.02

0.02

0.02

0.02

0.02

Random

1.0

0.02

0.02

0.02

0.02

0.02

0.02

0.02

0.02

0.02

0.02

0.02

0.02

0.02

OT

0.02

0.02

0.02

0.02

0.02

0.02

1.0

0.02

0.02

0.02

0.02

0.02

LCA

0.02

0.02

1.0

0.54

0.32

0.53

0.18

LCA

0.02

0.02

1.0

0.55

0.33

0.54

0.19

0.02

0.02

1.0

0.55

0.3

0.54

0.18

0.02

0.02

0.54

1.0

0.31

0.47

0.18

0.02

0.02

0.55

1.0

0.3

0.46

0.16

0.02

0.02

0.55

0.26

0.47

0.16

0.02

0.02

0.32

0.31

0.37

0.2

0.02

0.02

0.33

0.3

0.38

0.2

0.02

0.02

0.3

0.26

0.36

0.19

Probeless

0.02

0.02

0.53

0.47

0.37

0.24

Probeless

0.02

0.02

0.54

0.46

0.38

0.25

Probeless

0.02

0.02

0.54

0.47

0.36

0.24

IOU

0.02

0.02

0.18

0.18

0.2

0.24

1.0

IOU-

0.02

0.02

0.19

0.16

0.2

0.25

Q1

IOU-

0.02

0.02

0.18

0.16

0.19

0.24

.CA

CA

LCA

Random

Probeless

Random

Probeless

Random

Probeless

Random

1.0

0.02

0.02

0.02

0.02

0.02

0.02

Random

1.0

0.02

0.02

0.02

0.02

0.02

0.02

Random

1.0

0.02

0.02

0.02

0.03

0.02

0.02

0.02

0.02

0.02

0.02

0.02

0.02

0.02

OT

0.02

0.01

0.02

0.02

0.02

0.02

1.0

0.02

0.02

0.02

0.02

0.02

LCA

0.02

0.02

1.0

0.55

0.3

0.53

0.18

LCA

0.02

0.02

1.0

0.51

0.27

0.49

0.15

0.02

0.02

1.0

0.13

0.09

0.15

0.05

0.02

0.02

0.55

1.0

0.27

0.46

0.16

0.02

0.01

0.51

1.0

0.24

0.44

0.14

0.02

0.02

0.13

0.09

0.14

0.05

0.02

0.02

0.3

0.27

0.35

0.18

0.02

0.02

0.27

0.24

0.33

0.18

0.03

0.02

0.09

0.09

0.53

0.22

Probeless

0.02

0.02

0.53

0.46

0.35

0.23

Probeless

0.02

0.02

0.49

0.44

EEO

0.22

Probeless

0.02

0.02

0.15

0.14

0.53

0.24

IOU

0.02

0.02

0.18

0.16

0.18

0.23

1.0

IOU-

0.02

0.02

0.15

0.14

0.18

0.22

Q1

IOU-

0.02

0.02

0.05

0.05

0.22

0.24

CA

Random

Probeless

Random

Probeless

Random

Probeless

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work investigates interpretation methods in NLP that identify which neurons in a neural network are most related to particular concepts (e.g. a specific part of speech). The key idea is to compare how consistent one method's ranking of neurons is (w.r.t. a specific concept) with that of all other considered methods. The assumption is that the method that produces the most consistent rankings is the one that is closest to the true ranking of important neurons. The authors consider a number of interpretation methods and pretrained language models and show that across several settings, the method that is most consistent is Probeless (Antverg and Belinkov, 2021).\n\nSTRENGTHS:\n- The work considers the problem of interpreting the neuron-level representations of neural networks, which seems to be a recently emerging subarea of interpretability within NLP. I'm not fully convinced by its utility but there are indeed several recent works in this area so perhaps there would be interest in this work\n- The paper is mostly well written and motivated\n\nWEAKNESSES:\nThe main weakness is the assumption that consensus with existing interpretability methods is desirable. This consensus of course depends on the other considered interpretability methods, and it's not clear that the number of total interpretability methods (6) is sufficient to lead to reliable results that are based on consensus.\n\nQUESTIONS:\nHow do the insights about which interpretability methods is the most consistent change according to the considered interpretability methods? For example, can the authors remove 1,..n-1 interpretability methods from the consensus calculation and compare the results with those from the full set of n interpretability methods? \n\nMinor questions / notes:\n\nL100: are the examples supposed to be country names? The provided examples are cities.\n\nThere are a lot more \"random\"concepts in a corpus than target concepts. Are the random concepts that are used for evaluation sampled such that they are equal in number to the target concepts? How consistent is this evaluation over different random samples?\n\nL185: why does the performance of a probe using random 100 neurons being higher than other neurons mean that the probe is memorizing? As the authors point out earlier in the manuscript, the concept knowledge is distributed so it makes sense that as one increases the number of random neurons that are being considered, the prediction performance would improve. If the authors were looking at randomly initialized models rather than pretrained models, then I would agree with their conclusion, but in the current setup, there seem to be alternate explanations for their results.\n\nLIMITATIONS:\nSee under Weaknesses", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

comments: SUMMARY:
This work evaluates six different interpretation methods from a unified perspective. The authors focus on two challenges in this field: the absence of standard metrics and the lack of benchmarks. They propose two voting-based metrics to evaluate the compatibility among these six methods. Probeless consistently achieves the highest compatibility across all models based on their evaluation methodology. Further analytics experiments provide insight to the research community.

STRENGTHS:
1. The findings of the interpretation methods mentioned in this paper may foster research on the topic.

2. The idea of this paper is slightly novel.

WEAKNESSES:
1. The selected six methods need to be more novel. More recent advancements in this field may exist rather than relying on L1 & L2 regularization.

2. The validation of the superiority of Probeless needs to be more comprehensive.

3. The mentioned problems (metrics and benchmarks) were not effectively addressed, making this method difficult to be reproduced in future applications.

QUESTIONS:
1. Are there additional performance comparison experiments to practically demonstrate the effectiveness of this method? For example, try to deactivate the most important 10% of neurons calculated by each interpretation method, and then observe whether Probeless exhibits the most significant performance decline.

2. To assess the efficacy of an interpretation method across various fields, is it essential to replicate multiple interpretation methods within those fields before applying the voting metrics? If so, this process can be quite challenging and should be acknowledged as a limitation in this work.

LIMITATIONS:
There is no potential negative societal impact.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides a comparative analysis of six neuron interpretation methods utilizing diverse concepts across three distinct pre-trained models and introduces an evaluation framework predicated on voting theory. Importantly, it offers the first comprehensive examination of multiple neuron interpretation methods and strives to mitigate the challenges in this field.  The authors note similarities among the most proficient techniques within layers of neurons, with these resemblances remaining regardless of methodological deviations. It is thereby suggested, existing neuronal interpretation methodologies might have focused on a reciprocal group of top-performing neurons. \n\nThe paper also discusses the lack of no recognised evaluative metrics along with the lack of gold annotations by suggesting an evaluation strategy that consists of two compatible metrics with a pairwise comparison. This approach could facilitate the creation of a means to evaluate new neuron interpretation methods.\n\nSTRENGTHS:\nThe major strength of the paper lies in its originality; being the first work to create a ranking metric using voting theory for comparing the various neuron interpretation methods available. This methos should allow for autumatic comparison between existing and new neuron actiavtion methos. Furthermore, it embarked on a comprehensive comparative analysis via this new evaluation framework\n\nWEAKNESSES:\nThe primary weakness of the paper lies in its disregard for significant, influential studies previously published in the domain, thus lacking a comprehensive deliberation on an extensive range of neuron activation methods studied by the mechanistic interpretability community. In addition, the paper grievously overlooks the complexities associated with polysemantic, superposition and neuroplastic neuron behaviors encountered in neuron activation.\n\nWork worth considering are:\n\n1- Elhage, et al - https://transformer-circuits.pub/2021/framework/index.html\n\n2 - Olsson et al - https://transformer-circuits.pub/2022/in-context-learning-and-induction-heads/index.html\n\n3- Elhage, et al - https://transformer-circuits.pub/2022/in-context-learning-and-induction-heads/index.html\n\n4- Henighan et al - https://transformer-circuits.pub/2023/toy-double-descent/index.html\n\n5 - Foote et al - https://arxiv.org/pdf/2305.19911.pdf\n\n6 - Bills et al - https://openaipublic.blob.core.windows.net/neuron-explainer/paper/index.html \n\n7 - wang et al - arXiv preprint arXiv:2211.00593\n\n8 - Chan et al - https://www.alignmentforum.org/posts/JvZhhzycHu2Yd57RN/causal-scrubbing-a-method-for-rigorously-testing\n\n9 - Tenney et al - https://arxiv.org/pdf/2008.05122.pdf\n\nQUESTIONS:\nDo I understand it right that if the methods under study are sub-optimal, the voting results can also be sub-optimal? In other words, the voting method does not provide any indications of how well a method may perform. Rather, it simply allows for a comparison between them.\n\nConsidering that we are in the early phases of understanding the black-box nature of ML models, is it appropriate to start discussing benchmarks now? I have my reservations due to our lack of comprehensive hypotheses regarding their inner mechanisms. Hence, it presents a dilemma because it's as if we might not be in the position to start comparing or benchmarking.\n\nLIMITATIONS:\nThe authors provide some insightful limitations of thier work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a standardized evaluation metric and benchmark for comparing various neuron interpretation methods, based on ideas from majority voting. The benchmark is based on the hypothesis that \"neurons that are commonly discovered by different interpretation methods are more informative than others\", and uses this hypothesis to rank & score methods by their alignment with the majority (in terms of the neurons they assign behaviors to). I discuss some thoughts/limitations of this hypothesis below (and the authors discuss it as well in their limitations section). The authors demonstrate empirically why the current commonly-used evaluation metric (fitting a classifier to the neurons to predict the concept) favors certain methods over others and suffers from the same common issues as probing classifiers in general. Then then propose two \"compatibility\"-based metrics, one set-based and one rank-based, for comparing the top-k neurons deemed informative for explaining a concept by each method, and provide in-depth analysis using the metrics to compare 6 different existing neuron interpretation methods (3 corpus-based, and 3 classifier-based).\n\nOverall, I believe this paper is sound, well-executed, thorough, and clear, and will serve a useful role in the neural model interpretability sub-community. Based on the authors' response to my below questions, I may be inclined to raise my score further.\n\nEdit: I read the authors' rebuttal and it addressed my concerns. I did not change my score as I originally insinuated above, in part due to finding validity in the other reviewers' critiques. I still think the paper should be accepted.\n\nSTRENGTHS:\nOriginality & Significance:\n- The paper serves an important role in the mechanistic interpretability sub-community, and is likely to be of good value to this community.\n- I am not aware of any other attempts to organize methods in this manner, or propose a standardized benchmark.\n\nQuality & Clarity:\n- The experimental section is sound. The lack of ground-truth in makes evaluating explanation evaluation metric \"faithfulness\" difficult, but the authors have nonetheless provided a convincing set of results.\n- The paper is well-organized and well-written. The paper uses proper mathematical notation throughout.\n- The authors have promised to release the code upon publication.\n\nWEAKNESSES:\nAgreement with hypothesis underlying the benchmark:\n- I am not sure I am totally convinced by the claim that the most informative neurons will be discovered by the most methods, given that neurons that are easily discoverable can often be those which are perhaps more simplistic in what they encode (e.g., only serve a singular function rather than multiple, concept is encoded only in that singular neuron rather than spread over multiple, etc.) However, the granularity and assumptions underlying neuron interpretation methods is a nascent research question in the interpretability community overall, so I will not fault the authors for their hypothesis (i.e., I think the paper is appropriately scoped and the hypothesis is reasonable in the context of existing/popular neuron interpretation methods). It would be nice for the authors to discuss limitations of the hypothesis (or neuron interpretation methods in general) in the limitations section.\n \nMore minor:\n- The paper is lacking a related works section. While the methods studied in the paper are described in detail in Section 2, it would be nice to have some sort of summary paragraph of the field of neuron-level interpretation as a whole after the introduction to situate the work (and potentially mention other methods that are not tested in the paper). \n- There is some minor conflation it seems between the decision to treat each neuron interpretation method as producing a ranked list vs. performing binary set membership classification. This is particularly confusing with the classifier methods, which are directly trained to do the latter (4.1) but tested on the former via the NeuronVote method. However, the inclusion of the set-based metric, AvgOverlap, mitigates this issue to some extent. I think it would be good for the authors to clarify early in the paper that each method *can* be viewed as producing a ranking, even though this is not always how the methods have been proposed or used in practice, and that the inclusion/comparison of *both* evaluation metrics is designed to provide a view that does not unnecessarily favor one method over the other, by considering both the set membership and the rank views.\n\nQUESTIONS:\n- L12: colon instead of semicolon?\n- L20: ungrammatical sentence\n- Would be good to introduce the term \"probe\" somewhere in l.37-43 of the introduction (or earlier). You mention the \"Probeless\" classifier multiple times in the intro/abstract, but don't define this term anywhere.\n- L102: represents --> represent\n- L150: extra \"i\"\n- L205-206 describe AvgOverlap as set-based (not taking ranking into account); this contradicts lines 222-223, where I think the word \"ranking\" is meant as \"set\". Can you clarify?\n- I don't understand the role of Section 3.3 and its associated equation. Why wouldn't you just use Eqns. 7 and 8 with $|\\mathcal{M}| = 2$?\n- L255: the choice of hyperparameter values here seems like it could have a big impact on the results; did you do any search?\n- Table 2 is never referred to in the text\n\nLIMITATIONS:\nYes- see one suggestion above.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}