SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We consider three 12-layered pre-trained models: BERT-base-cased (BERT, Devlin et al., 2019), RoBERTa-base-cased (RoBERTa, Liu et al., 2019b) and XLM-Roberta-base (XLMR, Conneau et al., 2019). Each layer consists of 768 neurons. We consider concepts from three linguistic tasks: parts of speech tags (POS, Marcus et al., 1993), semantic tags (SEM, Abzianidze et al., 2017) and syntactic chunking (Chunking) using CoNLL 2000 shared task dataset (Tjong Kim Sang & Buchholz, 2000). Dataset details are provided in Appendix 7.2. We consider every tag as a concept and identify neurons with respect to the concept. In the case of classifier-based methods, we use a binary classification setup where the contextualized representation of words belonging to the concept serves as positive
Table 3: Task: POS, Average NeuronVote compatibility scores across concepts when selecting the top 10, 30, and 50 neurons from layers 1, 6 and 12. Bold numbers, underline numbers, and dashed numbers show the first, second, and third best scores respectively
Layers	BERT	RoBERTa	XLMR
1	6	12	1	6	12	1	6	12
Random	0.019	0.021	0.023	0.020	0.019	0.017	0.020	0.017	0.023
Gaussian	0.147	0.177	0.183	0.140	0.179	0.179	0.118	0.135	0.153
LCA	0.501	0.544	0.391	0.550	0.495	0.380	0.450	0.531	0.373
Lasso	0.496	0.545	0.395	0.475	0.453	0.350	0.410	0.471	0.267
Ridge	0.332	0.405	0.360	0.452	0.491	0.510	0.358	0.446	0.455
Probeless	0.497	0.550	0.515	0.590	0.589	0.580	0.534	0.590	0.497
IoU	0.343	0.380	0.348	0.315	0.338	0.275	0.334	0.320	0.344
examples. We randomly select the negative class words from the rest of the data, equal in size to the positive class examples. We drop the concepts that have less than 200 examples to ensure stable training. We split the binary classification dataset into train/dev/test splits of 70/15/15 percent. We use  $\lambda_{1} = 0.01$  and  $\lambda_{2} = 0.01$  for regularization-based methods. We present the results of POS only in the main paper and provide the rest in Appx. 7.3.
Number of top neurons: We did not optimize the number of neurons  $s$  selected by each method as this would result in a varying number of neurons and will make the compatibility score incompatible across methods. We consider  $s = 10, 30, 50$  which covers a diverse range to generalize the findings.
Baseline: We generate a random ranking of neurons to compare the behavior of a neuron interpretation method with a random selection of neurons. We refer to it as Random.
Procedure: We extract layer-wise contextualized representations of words in the dataset. For every layer representation, we discover the top neurons with respect to a concept using each neuron interpretation method. We follow a leave-one-out strategy to calculate the compatibility score of each method with respect to other methods. More specifically, we consider one neuron interpretation method as a test method and select the rest as the database of the sets of top neurons. We then calculate the AvgOverlap and NeuronVote scores of the test method.
4.2 Compatibility Scores
Table 2 presents the average compatibility scores across all concepts and when selecting the top 10, 30, and 50 neurons from layers 1, 6, and 12.
AvgOverlap and NeuronVote show a similar pattern: While the scale of the compatibility score is different between both metrics, they are consistent in highlighting the top interpretation methods i.e. Lasso, LCA, and Probeless. In the rest of the paper, we mainly present the results of NeuronVote, and discuss the AvgOverlap only when it is necessary to show a different trend.
Overall comparison across methods: Table 3 presents the NeuronVote scores averaged over all POS concepts and when selecting the top 10, 30, and 50 neurons. The bold, underline, and dashed numbers represent the first, second, and third best compatibility scores respectively. Probeless consistently achieves the highest compatibility across all layers and models. In other words, the neurons selected by Probeless have the highest overlap with the neurons selected by other methods. We did not observe a similar trend for IoU which is also a corpus-based method.
LCA and Lasso achieve the second-best compatibility score (see underlined and dashed numbers in Table 3). Ridge, although methodologically similar to LCA and Lasso, did not consistently show high compatibility across models. We discuss these methods later in Section 4.3. Gaussian achieved the worst compatibility score which is lower by 0.354 points from the best NeuronVote score on layer 1 and is only 0.128 better than Random for BERT. The result of Gaussian presents an interesting scenario where the compatibility score suggests that the neurons identified by Gaussian are least similar to all other methods. However, the difference with Random, although small, suggests that the selection of neurons is not random. The low results of Gaussian are inline with Antverg & Belinkov (2021) where they found that the Gaussian method memorizes the probing task and it may not provide the most faithful ranking of neurons. These scores, therefore, suggest that further investigation may be required to confirm the efficacy of the Gaussian method.
A notable point about the top 3 methods (Probeless, LCA, and Lasso) is their methodological diversity, despite of which they selected similar neurons to other methods. On the other hand, methodologically similar techniques such as IoU and Probeless did not result in similar compatibility scores. This shows that the compatibility metric is not biased by methodological similarity, which was one of the primary concerns with the commonly used evaluation metrics.
Layer-wise trend: Ethayarajh (2019) showed that each layer is different in terms of representation geometry. Sajjad et al. (2022a) revealed that due to the geometry of the representation space, particularly in the last layers, knowledge of a concept may not be readily available. We hypothesize that the performance of certain neuron interpretation methods may vary based on the nature of the representation space. Our compatibility methods facilitate quantitative evidence to support this hypothesis by showing that certain neuron analysis methods suffer in selecting the most compatible neurons from the higher layers. LCA and Lasso both showed a substantial drop in their compatibility scores for the 12th layer (for example, Table 3: NeuronVote LCA - 0.544 for layer 6 and 0.391 for layer 12 for BERT). In contrast, Probeless did not show any substantial difference in the scores of the last layers and earlier layers (NeuronVote: Probeless - 0.550 vs 0.515 for layer 6 and 12 respectively for BERT). This further support Probeless as the most reliable method and encourages further investigation into the robustness of Lasso and LCA with varying representational space.
Compatibility across models: The overall trend of top-performing neuron interpretation methods is similar across BERT, RoBERTa, and XLMR. One notable difference is the compatibility score of Ridge for the last layer of the RoBERTa (0.510) and XLM-R (0.439) models which is substantially higher compared to the BERT model (0.360). Moreover, it achieved the second best score for XLM-R after Probeless and is substantially better than LCA and Lasso. Ridge prefers correlated features in contrast to Lasso, which prefers a few spiky features highly predictive of the concept. Based on the results, we hypothesize that the last layers of RoBERTa and XLM-R consist of highly correlated neurons and Ridge is effective in discovering salient neurons of correlated nature. LCA balances between spiky and correlated neurons, and results in higher scores than Lasso in these cases.
(a) Layer 1
(b) Layer 6
Figure 1: Task: POS, Model: BERT, Comparing average overlap of top 10-50 neurons
(c) Layer 12
Random selection of neurons: The compatibility score of Random serves as a baseline that provides a perspective of the correctness of a newly proposed neuron interpretation method. It gave substantially lower scores than all neuron interpretation methods (Table 3). Even though the Gaussian showed the lowest compatibility score among interpretation methods, the difference with Random adds a factor of confidence to its ranking. Moreover, a low compatibility score for Random also implies that each of the other methods are bringing in important neurons to some degree.
4.3 Pairwise Comparison
The compatibility metrics provide a single score to understand an interpretation method in the context of a large set of other interpretation methods. We further extend the analysis to pair-wise comparison and provide insights into the relationship between different methods. Figure 1 presents the heatmaps comparing the average overlap of the top  $10 - 50$  neurons across methods using BERT. The light color in the heatmap shows higher matches while the dark color shows lower matches. We added Random - a random selection of neurons as a baseline. It showed an average overlap of less than  $3\%$  only (see the first row in heatmaps).
Comparing Regularization-based Methods: LCA and Lasso showed a high overlap of up to  $64\%$  in the discovered neurons (see Figure 1(a), 1(b)). However, Ridge results in a lower overlap of at most
29% compared to other methods. Note that the Ridge regularization does not force the coefficients of neurons to zero, thus considers most of the neurons during the training phase. The resulting top neurons may represent correlated neurons capturing similar and redundant knowledge (Dalvi et al., 2020) and may not be a representative of the neurons with the most predictive power with respect to the understudied concept. Lasso prefers individual neurons most predictive of the concept and ElasticNet (LCA) provides a good balance between selecting individual neurons and correlated neurons highly predictive of the concept.
Corpus-based vs. Regularization-based Methods: Probeless and IoU both directly rely on the activation value of a neuron. However, we did not observe a consistent overlap between their discovered neurons (at most  $30\%$  in Figure1(c)). In comparison to other classes of methods, Probeless showed a better and consistent overlap with LCA and Lasso. For example, in Figure 1(a) we observed an overlap of  $66\%$  between Probeless and LCA, and  $55\%$  between Probeless and Lasso. The relatively high overlap of Probeless with methods using Lasso points towards the selection of identical focused neurons by these methods despite their methodological differences.
Outliers: Both Ridge and IoU show a smaller but consistent overlap with most of the methods. As discussed earlier, Ridge works well for correlated features which may not be highly predictive of the concept. IoU relies on activation values and it may select neurons that consistently activate higher than the threshold irrespective of the concept. Gaussian did not show substantial overlap with any methods or with any combination of methods. Further investigation is needed to evaluate its' efficacy.
The area of interpreting deep learning models constitutes a broad expanse of research. This section provides a synthesized overview of diverse interpretability subareas within deep learning models applied to Natural Language Processing (NLP), while also outlining the scope of our study.
Feature importance and attribution methods identifies the contribution of input features to predictions. These methodologies predominantly rely on the gradient of the output concerning the input feature and determine input feature importance by evaluating the magnitude of gradient values (Denil et al., 2014; Sundararajan et al., 2017; Danilevsky et al., 2020).
Counterfactual Intervention revolves around an intricate analysis of the interplay between input features and predictions. This approach involves manipulating inputs and quantifying resulting output alterations. Diverse intervention strategies, including erasing input words and substituting words with different meanings have been used (Li et al., 2016b; Ribeiro et al., 2018).
Attention Weights: Numerous investigations have been directed towards interpreting components of deep learning models at varying levels of granularity. For instance, attention weights have emerged as a viable metric to gauge the interrelation between input instances and model outputs (Martins & Astudillo, 2016; Vig, 2019). Along these lines, Geva et al. (2021) delved into the analysis of feedforward neural network components within the transformer model, revealing their functionality as key-value memories. Additionally, Voita et al. (2019) demonstrated that pruning many attention heads has minimal impact on performance.
Mechanistic Interpretability puts a focus on reverse engineering of network weights to comprehend their behavior (Nanda et al., 2023). Building upon the Distill Circuits thread, Elhage et al. (2021) investigated two-layered transformer models with attention blocks, identifying attention heads contributing to in-context learning. This understanding was further extended to larger transformer-based language models by Olsson et al. (2022). To enhance neuron interpretability, Elhage et al. (2022) introduced a Softmax Linear unit as an activation function replacement. Wang et al. (2022a) attempted to bridge mechanistic interpretability findings in small networks to large ones, particularly GPT-2 small. Their approach involved iteratively tracing influential model components from predictions using causal intervention. They showcased the potential of mechanistic interpretability in understanding extensive models, while also highlighting associated challenges. Similarly, Bricken et al. (2023) uses a sparse autoencoder to disentangle polysemantic neurons.
Representation Analysis involves probing network representations concerning predefined concepts, to quantify the extent of knowledge captured in these representations (Conneau et al., 2018; Liu et al., 2019a; Tenney et al., 2019; Durrani et al., 2019; Arps et al., 2022). This is often realized through
training diagnostic classifiers for linguistic concepts, wherein classifier accuracy serves as an indicator of concept knowledge within representations. See Belinkov & Glass (2019) for a comprehensive survey.
Neuron Interpretation A more intricate form of representation analysis, termed neuron interpretation, delves into how knowledge is structured within the network (Sajjad et al., 2022b). This approach establishes connections between neurons and predefined concepts, offering insights into where and how specific concept knowledge is assimilated. Work done on neuron analysis can be broadly classified into three groups: Neuron visualization involves manual identification of patterns across a set of sentences (Li et al., 2016a; Karpathy et al., 2015). More recently Foote et al. (2023) proposed an automated approach to enhance interpretability of Large Language Models (LLMs) by extracting and visualizing individual neuron behaviors as interpretable graphs. Corpus-based Methods explore the role of a neuron through techniques such as ranking sentences in a corpus (Kádár et al., 2017b), generating synthetic sentences (Poerner et al., 2018) that maximize its activation, or computing neuron-level statistics over a corpus (Mu & Andreas, 2020; Suau et al., 2020; Antverg & Belinkov, 2022). Bills et al. (2023); Mousi et al. (2023) proposed the use of LLM to interpret neurons. Probing Methods identify salient neurons for a concept by training a classifier using neuron activations as features (Radford et al., 2019; Lakretz et al., 2019; Durrani et al., 2022) or fitting a multivariate Gaussian over all neurons and then extracting individual probes for single neurons (Hennigen et al., 2020).
A number of works identify neurons with respect to the output class (Wang et al., 2022b; Dai et al., 2022). They are effective in finding neurons that play a role in the prediction. In this paper, we focus on the neuron interpretation methods that take a concept as input and find neurons with respect to the concept. We propose an evaluation framework to formalize the evaluation and comparison of results across methods. Moreover, we propose a novel method, MeanSelect and present a case study of using the evaluation framework.
We provided a thorough comparative analysis of neuron interpretation methods in NLP. We overcame the challenge of the lack of a standard evaluation metric and gold annotations by proposing an evaluation strategy consisting of two compatibility metrics and a pair-wise comparison. In addition to developing a capability to evaluate a new neuron analysis method, we presented various insights into existing neuron interpretation methods. For example, due to the correlated nature of last layer representations, they are most challenging for interpretation methods. The selection of top neurons overlaps substantially irrespective of the methodological differences among techniques. We made the evaluation framework available to the research community.
**Limitations:** Our methodology relied on consensus and thus inherits the limitations of that paradigm. The consensus may mislead the evaluation under certain settings e.g. if the majority of the voters form a lobby then they can skew the results of the consensus. This scenario can happen if a set of methods always produce close to identical ranking, they will cause a decrease in the compatibility score of a new method that produces a ranking different from theirs. Another possible issue is if a new method discovers an entirely different set of neurons than the ones discovered by other methods. The compatibility score of such a method will be low. We intend to mitigate these issues by including a large number of diverse neuron interpretation methods that are theoretically different from each other. We believe that under these settings, the scenario of an entirely different set of discovered neurons is more theoretical than practical. We further present a pair-wise analysis that provides an analysis at a more granular level and bring insight into how a method compares with other methods. The pairwise comparison heatmap will highlight if a set of methods form a lobby.
Another limitation of current neuron interpretation methods is that they do not explicitly target the discovery of neurons of diverse nature such as polysemantic, and superposition. Theoretically, only the ElasticNet regularization is capable of discovering neurons learning a singular function and multiple functions. The other methods such as Probeless are incapable of discovering multifunction neurons. None of the neuron interpretation methods explicitly identify superposition neurons. Explicit modeling of neurons of different nature in a neuron interpretation method may result in discovering novel sets of neurons. The evaluation methods including our proposed framework do not explicitly compare neurons of different types.
Wang, K., Variengien, A., Conmy, A., Shlegeris, B., and Steinhardt, J. Interpretability in the wild: a circuit for indirect object identification in gpt-2 small, 2022a.
Wang, X., Wen, K., Zhang, Z., Hou, L., Liu, Z., and Li, J. Finding skill neurons in pre-trained transformer-based language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 11132-11152, Abu Dhabi, United Arab Emirates, December 2022b. Association for Computational Linguistics. doi: 10.18653/v1/2022.emnlp-main.765. URL https://aclanthology.org/2022.emnlp-main.765.
Yilmaz, E., Aslam, J. A., and Robertson, S. A new rank correlation coefficient for information retrieval. In Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '08, pp. 587-594, New York, NY, USA, 2008. Association for Computing Machinery. ISBN 9781605581644. doi: 10.1145/1390334.1390435. URL https://doi.org/10.1145/1390334.1390435.
Zhang, K. and Bowman, S. Language modeling teaches you more than translation does: Lessons learned through auxiliary syntactic task analysis. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pp. 359-361, Brussels, Belgium, November 2018. Association for Computational Linguistics. doi: 10.18653/v1/W18-5448. URL https://www.aclweb.org/anthology/W18-5448.
Zou, H. and Hastie, T. Regularization and variable selection via the elastic net. Journal of the Royal Statistical Society, Series B, 67:301-320, 2005.
(a) BERT
(b) RoBERTa
Figure 2: Task: POS, Pairwise comparison of MeanSelect with other methods using 10-50 neurons
(c) XLMR
7.1 Case Study
Mean Select: We propose a new corpus-based method, MeanSelect, for neuron ranking as a case-study to illustrate how researchers can use our proposed methodology. Kádár et al. (2017b) generated explanations for neurons by extracting top-N 5-gram context for each neuron based on the magnitude of their activations, followed by human annotation. Na et al. (2019) removed the human-in-the-loop by extracting concepts of various granularities from a parsed tree and aligned highly activating neurons to the concepts. Inspired by these works, we propose a novel method, MeanSelect, that generates a ranking of neurons with respect to a concept. The intuition is that a neuron learning a concept will have consistently high activations across different contexts where that concept appears. However, there may be a neuron that always activates with high value irrespective of the concept. A difference in the mean value of the neuron activating the concept and all other concepts will provide the true behavior of the neuron for the concept. Following our notation in Section 2, the score of a given neuron  $n$  is defined as follows:
$$
R (n, \mathcal {C}) = \frac {\mu (\mathcal {C}) - \mu (\hat {\mathcal {C}})}{n _ {\max} - n _ {\min}} \tag {10}
$$
where  $\mu (\mathcal{C})$  is the average,  $n_{max}$  is the max and  $n_{min}$  is the min of activations  $z(n,w)$  where  $w\in \mathcal{C}$ , and  $\mu (\hat{\mathcal{C}})$  is the average of activations over the random concept set.
Compatibility Metrics Table 4 shows the compatibility score of MeanSelect using the representation of 3 different layers and compared it with the Random selection of neurons. The high compatibility scores show that MeanSelect discovers neurons that are endorsed by other neuron interpretation methods. This serves as a measure of confidence in the proposed method. Moreover, one may observe that the method has a relatively lower score for the last layer compared to other layers, giving insight into potential improvements that can be made to the behavior of this method.
Pairwise Comparison The pairwise comparison of methods further provides insights into how the new method relates to other methods in terms of the resulting neurons. Figure 2 shows the heatmaps of three pre-trained models. MeanSelect has the highest overlap with the Probeless method and except Gaussian, it shows an overlap of at least 0.23 points with other methods. While the high overlap of MeanSelect with Probeless is not surprising given both are based on similar intuitions, the overlap with LCA and L1 shows that the method is selecting a diverse set of neurons captured by a variety of methods. Similar to the discussion on compatibility score, here we observe a substantial overlap drop in the last layer and this highlights the potential vulnerability of the method to certain representations.
7.2 Concept Datasets
We consider concepts from three linguistic tasks: parts of speech tags (POS, Marcus et al., 1993), semantic tags (SEM, Abzianidze et al., 2017) and syntactic chunking (Chunking) using CoNLL 2000 shared task dataset (Tjong Kim Sang & Buchholz, 2000). For the POS dataset, we used 20 concepts
Table 4: Task: POS, Model: BERT, Average NeuronVote score of MeanSelect using 10-50 neurons
Layers	BERT	RoBERTa	XLMR
1	6	12	1	6	12	1	6	12
Random	0.019	0.021	0.023	0.020	0.019	0.017	0.019	0.017	0.022
MeanSelect	0.464	0.476	0.402	0.392	0.451	0.328	0.368	0.408	0.253