SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yimin Fan $^{\text{♥}}$  Fahim Dalvi $^{\text{♣}}$  Nadir Durrani $^{\text{♣}}$  Hassan Sajjad $^{\text{♣}}$
The Chinese University of Hong Kong, Hong Kong, China
$\diamond$ Qatar Computing Research Institute, HBKU, Qatar
\*Faculty of Computer Science, Dalhousie University, Canada
Neuron interpretation offers valuable insights into how knowledge is structured within a deep neural network model. While a number of neuron interpretation methods have been proposed in the literature, the field lacks a comprehensive comparison among these methods. This gap hampers progress due to the absence of standardized metrics and benchmarks. The commonly used evaluation metric has limitations, and creating ground truth annotations for neurons is impractical. Addressing these challenges, we propose an evaluation framework based on voting theory. Our hypothesis posits that neurons consistently identified by different methods carry more significant information. We rigorously assess our framework across a diverse array of neuron interpretation methods. Notable findings include: i) despite the theoretical differences among the methods, neuron ranking methods share over  $60\%$  of their rankings when identifying salient neurons, ii) the neuron interpretation methods are most sensitive to the last layer representations, iii) Probeless neuron ranking emerges as the most consistent method.
The advent of deep neural network (DNN) models and their success in Natural Language Processing (NLP) has opened a new research frontier to interpret the representations learned within these models. Representation analysis provides a holistic view of the knowledge learned within the representation. Whereas neuron analysis gives insight into how knowledge is structured within the representation. To this end, an ample amount of work has been done to understand the knowledge captured within the learned representations (Belinkov et al., 2017; Liu et al., 2019a; Tenney et al., 2019; Dalvi et al., 2022) and individual neurons (Karpathy et al., 2015; Kádár et al., 2017a; Dalvi et al., 2019).
Neuron analysis provides a fine-grained interpretation of the representation. It discovers salient neurons of the network w.r.t to a concept such as morphological or syntactic concepts (Lakretz et al., 2019; Durrani et al., 2020), and provides insight into how they may be used during inference (Lundberg & Lee, 2017; Dhamdhere et al., 2018). Beyond understanding the inner dynamics of the DNN models, neuron analysis has various potential applications such as network manipulation (Bau et al., 2019), domain adaptation (Antverg et al., 2022), model distillation (Dalvi et al., 2020) and architectural search (Prasanna et al., 2020).
A number of neuron analysis methods have been proposed in the literature to interpret deep NLP models. While these methods have shown promising results, there is a notable absence of systematic comparisons among them in existing studies. Conducting a fair comparison of neuron interpretation methods is crucial for comprehending the merits and limitations of each technique. Such comparisons
<https://github.com/fdalvi/neuron-comparative-analysis
serve as a guiding light for the community, steering efforts towards the development of improved and more effective interpretation methodologies.
In this work, we study a variety of neuron interpretation methods under a controlled setting and provide a thorough comparative analysis. There are two main challenges in comparing neuron interpretation methods: i) absence of a standard evaluation metric, ii) lack of an evaluation benchmark (Sajjad et al., 2022b). The most common evaluation metric used in the neuron analysis literature is to train a classifier using the discovered neurons as features. The performance of the classifier in predicting the concept of interest serves as evidence of the quality of the discovered neurons, in other words, the correctness of the neuron interpretation method.
We argue that the classifier as an evaluation metric is suboptimal for two reasons: i) training a classifier provides an opportunity to learn the task and it may result in good performance irrespective of the quality of the neurons used as features (Hewitt & Liang, 2019), ii) the classifier as an evaluation method may favor the neuron interpretation methods that are methodologically similar (for instance, methods that use a classifier to discover salient neurons). Antverg & Belinkov (2021) shed light on the latter issue and showed that the methodological similarity between the neuron interpretation methods and the evaluation methods may result in unreasonably high accuracy scores. We empirically demonstrate the issue in Section 3. Another bottleneck in comparing neuron interpretation methods is the absence of evaluation benchmark. Neurons are distributive in nature and exhibit polysemy (capturing multiple concepts at the same time). Moreover, there can be different sets of neurons learning a concept, and there exists no single correct answer. It is therefore intricately challenging to create the ground truth annotations and infeasible to scale to a large set of models and concepts.
Given the lack of a standard evaluation metric and the infeasibility of creating ground truth annotations, how can we compare neuron interpretation methods? We rely on the voting theory (O'Connor & Robertson, 2003) that performs a systematic aggregation of results in order to achieve a consensus (Patrao Neves, 2016). It is effectively used in political, social, and machine-learning communities for cases wherein there are distinct preferences among voters (models). Examples of specific applications of voting theory include ranking search results obtained from various algorithms in information retrieval (Lin et al., 2017; Yilmaz et al., 2008), finding the best machine translation system using pair-wise ranking (Lapata, 2006) and aggregating results of several NLP tasks (Colombo et al., 2022).
We hypothesize that neurons that are commonly discovered by different interpretation methods are more informative than others, and may serve as a signal of correctness for the selected neurons. Motivated by this, we devise two voting-based compatibility metrics. Our first metric, Avg0ver1ap, is based on plurality voting (Cooper & Zillante, 2012). Avg0ver1ap computes an average overlap of the discovered neurons across all voters. Our second method, NeuronVote, inspired by Borda Count (Lippman, 2012), takes into account the ranking of neurons of each method while calculating the average pair-wise intersection over union with all the other methods. The compatibility metrics provide a single score for each method, where a higher score refers to a neuron interpretation method whose discovered neurons are more compatible with the other methods. We further extend the evaluation of neuron interpretation methods to pair-wise comparison, providing insights into how any two given methods relate in terms of their discovered neurons.
We conduct a comparative analysis of six neuron interpretation methods using diverse concepts consisting of morphological, semantic, and syntactic properties and using three pre-trained models. The analysis suggests that the best performing methods, despite their methodological differences, share more than  $60\%$  of the top neurons. Probeless ranking Antverg & Belinkov (2022) consistently exhibited the highest overlap with other methods. Because neurons in the last layer representations are highly correlated, they are the most challenging to interpret. Finally, we present a case study that demonstrates the usefulness of our evaluation methodology for any new method proposed in the future (Appendix 7.1). Our contributions are as follows:
- We conduct the first thorough comparative analysis of a large set of neuron interpretation methods of NLP across 3 pre-trained models and using a diverse set of linguistic concepts.
- Our proposed evaluation methodology consisting of two compatibility metrics and a pairwise comparison targets one of the critical limitations of the neuron interpretation studies.
- We provide the evaluation methodology as a framework to facilitate future studies and comparison of methods.
2 Neuron Interpretation Methods
The goal of neuron interpretation methods is to rank  $\mathcal{N}$  neurons with respect to some concept  $\mathcal{C}$ . These methods can be broadly classified into two classes: corpus-based methods and probing-based methods (Sajjad et al., 2022b). The former discovers the role of neurons by aggregating statistics over neuron activations and the latter trains classifiers to achieve the same.
Let  $\mathcal{D}$  be a dataset of sentences (represented as a list of tokens), where each token  $w$  appears in a specific context/sentence and has a contextual representation associated with it. Let  $z(n, w)$  be the activation of the  $n^{th}$  neuron and the  $w^{th}$  token in  $\mathcal{D}$ . The neuron  $n \in \mathcal{N}$  can be from any component of the original model, such as a specific layer. Also note that the activation value  $z(n, w)$  in context-dependent, so the same  $w$  can result in different  $z(n, w)$  values depending on context.
A concept  $\mathcal{C}$  represents a property we want to discover neurons for and  $\hat{\mathcal{C}}$  represents a random concept, i.e. containing words that are not associated with the concept  $\mathcal{C}$ . For example,  $\mathcal{C}_{\text {country }} = \{\text { Rome, Paris, London, New York, } \ldots \}$  represent a concept of country names and  $\hat{\mathcal{C}}$  would consist of all other words in the corpus that are not country names. More formally,  $\hat{\mathcal{C}} = \mathcal{D} \setminus \mathcal{C}$ .
Let  $R(n, \mathcal{C})$  represents the score of a neuron  $n$  with respect to a concept  $C$ . Each neuron interpretation method implements  $R$ , and provides a ranked list of neurons with respect to a concept. In the following, we present six interpretation methods<sup>2</sup> studied in this paper.
2.1 Corpus-based Methods
Corpus-based methods align neurons with a concept by accumulating co-occurrence patterns between neuron activations and the presence of the concept of interest. We use two corpus-based methods in this work presented as follows.
2.1.1 Probeless
The Probeless method (Antverg & Belinkov, 2021) obtains neuron rankings based on an accumulative strategy. The score of a given neuron  $n$  is defined as follows:
$$
R (n, \mathcal {C}) = \mu (\mathcal {C}) - \mu (\hat {\mathcal {C}}) \tag {1}
$$
where  $\mu(\mathcal{C})$  is the average of activations  $z(n, w)$ ,  $w \in \mathcal{C}$ .  $\mu(\hat{\mathcal{C}})$  is the average of activations over the random concept  $\hat{\mathcal{C}}$ . Note that the ranking for each neuron  $n$  is computed independently.
2.1.2 IoU Method
Mu & Andreas (2020) proposed IoU to generate compositional explanations for neurons. For each token in the dataset, they create i) a binary mask of a neuron by thresholding activations above a percentile and ii) a binary mask of a concept by checking its presence in the token. They compute Intersection over Union (IoU) over the two masks:
$$
R (n, \mathcal {C}) = \frac {\sum_ {w} \mathbb {1} (z (n , w) > \delta) \wedge \mathbb {1} (w \in \mathcal {C})}{\sum_ {w} \mathbb {1} (z (n , w) > \delta) \vee \mathbb {1} (w \in \mathcal {C})} \tag {2}
$$
where  $\mathbb{1}$  is the indicator function.  $z(n,w) > \delta$  is a binary mask over neuron activations based on the threshold  $\delta$ .  $w \in \mathcal{C}$  is a binary mask, created by checking if the current word represents the concept  $\mathcal{C}$  of interest. Mu & Andreas (2020) used IoU scores to generate alternative explanations for neurons on the task of image classification. Here we apply their method in the NLP domain to generate neuron ranking for any concept of interest. Like Probeless, the ranking for every neuron is computed independently of other neurons.
2.2 Probing Methods
Probing methods train a classifier using neuron activations as features for the task of predicting the concept of interest (Belinkov et al., 2017; Hupkes et al., 2018). The internals of the trained classifier (e.g. its weights) are used to rank the features (neurons).
Available in the NeuroX toolkit Dalvi et al. (2023).
2.2.1 Lasso Regularizer (L1)
Radford et al. (2019) trained a linear classifier with L1 regularization and used the weights of the classifier as a proxy for the importance of neurons for the given concept. The loss function of the classifier is as follows:
$$
\mathcal {L} (\theta) = - \sum_ {w} \log P _ {\theta} (c | z _ {\mathcal {N}} (w)) + \lambda_ {1} \| \theta \| _ {1} \tag {3}
$$
where,  $c \in \{\mathcal{C}, \hat{\mathcal{C}}\}$  is the correct class for  $w$ ,  $z_{\mathcal{N}}(w)$  is the vector of all neuron activations for word  $w$ , i.e.  $[z(0,w), z(1,w), \dots, z(n,w)]$ , and  $P_{\theta}(c|z_{\mathcal{N}}(w))$  is the probability that word  $w$  belongs to the class  $c$ . The learned weights  $\theta$  serve as the ranking of neurons. Specifically,  $R(n,\mathcal{C})$  is defined as the absolute weights for the class i.e.  $|\theta_{\mathcal{C}}(n)|$ .
L1 adds the "absolute value of magnitude" of the coefficient as a penalty term to the  $L(\theta)$ , which shrinks the less important weights to zero, leading to sparse weight distribution. Radford et al. (2019) used this regularization to force the classifier to learn spiky weights, indicating the selection of a few specialized neurons learning a concept, while setting the majority of neurons' weight to zero. Such an assumption is useful in discovering focused neurons that learned one particular concept only.
2.2.2 Ridge Regularizer (L2)
Lakretz et al. (2019) used L2 regularization to train the linear classifier. The loss is as follows:
$$
\mathcal {L} (\theta) = - \sum_ {w} \log P _ {\theta} (c | z _ {\mathcal {N}} (w)) + \lambda_ {2} \| \theta \| _ {2} ^ {2} \tag {4}
$$
L2 forces weights to be close to zero (but not zero). It is useful to deal with multicollinearity (neurons that are highly correlated) scenarios, through constricting the coefficient while keeping all the features. Intuitively, this encourages grouping of features, thus discovering group neurons that jointly learn a concept. The score  $R(n,\mathcal{C})$  is computed in a similar fashion as the Lasso Regularizer method.
2.2.3 ElasticNet Regularizier (LCA)
Both L1 and L2 regularizations capture properties that are desirable when selecting the most important neurons. L1 facilitates sparsity, identifying focused neurons while L2 encourages identifying groups of highly correlated features into account. Dalvi et al. (2019) used ElasticNet regularization (Zou & Hastie, 2005) that balances the trade-off between them.
$$
\mathcal {L} (\theta) = - \sum_ {w} \log P _ {\theta} (c | z _ {\mathcal {N}} (w)) + \lambda_ {1} \| \theta \| _ {1} + \lambda_ {2} \| \theta \| _ {2} ^ {2} \tag {5}
$$
$\lambda_{1}$  and  $\lambda_{2}$  are hyperparameters that are tuned to optimize the effect of L1 and L2 regularization.
2.2.4 Gaussian Classifier
Hennigen et al. (2020) assumes that neurons exhibit a Gaussian distribution. They fit a multivariate Gaussian, say  $\mathcal{P}$  over all neurons. Since multivariate Gaussian is decomposable by nature, they are able to extract individual probes for any subset of input features without additional work. Formally, they are able to extract  $\mathcal{P}_{\mathcal{F}}$ , where  $\mathcal{F} \subset \mathcal{N}$ . The classifier itself is trained using the Bayesian framework, specifically the maximum a posteriori estimate to compute the parameters  $\theta$ . Once a multivariate Gaussian is trained, the neuron selection is performed in a greedy fashion:
$$
F = (), F = F \oplus \arg \max  _ {n} \mathcal {P} (\mathcal {C} | F \oplus \{n \}) \quad \text {f o r} i \text {i n} 1.. | \mathcal {N} |
$$
$$
R (n, \mathcal {C}) = | \mathcal {N} | - i n d e x (F, n) \tag {6}
$$
In essence, probes for individual neurons are first extracted, and the neuron probe with the highest log-likelihood is considered the most important neuron. Next, probes are extracted for pairs of neurons, with one of them being the selected neuron. The pair with the highest log-likelihood then contributes the second most important neuron to the ranking. The full ranking is compared iteratively, with each step being a greedy selection of the next best neuron to add based on the log-likelihood. This method makes two fundamental assumptions: i) neuron activations follow a Gaussian distribution and ii) neuron ranking can be done in a greedy fashion.
3 Evaluation Methodology
3.1 Classifier Accuracy
The commonly used evaluation metric i.e. classification accuracy of the selected neurons is inadequate: i) because it favors the neuron ranking methods that are based on the probing framework (Antverg & Belinkov, 2021), ii) it is not clear whether the classifier performance is a reflection of the knowledge learned in the discovered neurons or it is due to the capacity of the classifier to learn and memorize the task (Hewitt & Liang, 2019; Zhang & Bowman, 2018).
We demonstrate these issues empirically, by using a linear classifier without regularization as an evaluation metric to evaluate various neuron interpretation methods. Given  $s$  discovered neurons with respect to a concept  $\mathcal{C}$ , we train the classifier using these discovered neurons as features. The performance of the classifier serves as a measure of the correctness of  $s$ . Table 1 presents the average accuracy across all parts of speech (POS) concepts and across all layers of BERT for  $s = 30, 50, 70, 100$ . Random refers to a random selection of neurons. The details of the dataset and the experimental setup are provided in Section 4. A few notable observations are: the classifier accuracy is higher for the probing methods (Lasso and Ridge) than the corpus-based methods (Probeless and IoU), ii) the classifier has the capacity to memorize as a Random selection of neurons performs reasonably well when more than a certain number of neurons are used in the evaluation. These results complement the issues raised earlier i.e. the classifier may result in high performance irrespective of the correctness of the neurons, and it favors the interpretation methods that are methodologically similar to itself.
Table 1: Task: POS, Model: BERT, Accuracy scores using classifier accuracy as an evaluation metric. Bold and underline are the first and second best results.
Neurons	30	50	70	100
Probeless IoU	0.92	0.94	0.95	0.96
0.91	0.93	0.94	0.95
Lasso Ridge	0.93	0.95	0.95	0.96
0.95	0.97	0.98	0.98
Random	0.81	0.86	0.89	0.92
Table 2: Task: POS, Model: BERT, Average compatibility scores AvgOverlap and NeuronVote when selecting the top 10, 30, 50 neurons from layers 1, 6, 12. Bold and underline are the first and second best scores.
AvgOverlap	NeuronVote
Random	0.021	0.021
Gaussian	0.086	0.169
LCA	0.258	0.514
Lasso	0.240	0.473
Ridge	0.177	0.362
Probeless	0.269	0.532
IoU	0.156	0.365
3.2 Compatibility Metrics
Given the absence of a good evaluation metric and gold annotations, we rely on consensus-based voting theory. The consensus serves as a tool to measure the relation between rankings. In NLP, it has been widely used in information retrieval to combine search results of various algorithms (Lin et al., 2017; Yilmaz et al., 2008), in machine translation to compare different systems' output using pair-wise ranking (Lapata, 2006) and to aggregate results of NLP tasks (Colombo et al., 2022).
One way to compare rankings is to perform pairwise comparison using rank correlation or similarity functions such as Kendall's  $\tau$  (Kendall, 1938) and Spearman correlation (Daniel, 1990). These methods assume conjoint rankings where all rankings contain identical elements. In the case of neuron rankings, for every neuron interpretation method, we only consider the  $s$  top-ranked neurons with respect to a concept. The rationale behind this setting is to minimize the noise and randomness in rankings that may occur for neurons that are not learning the concept. This results in disjoint sets and the rank correlation methods can not be directly applied in such cases (Lin et al., 2017).
Motivated by the voting theory, we devise two metrics to compare disjoint rankings. Our first metric, AvgOverlap, is based on plurality voting (Cooper & Zillante, 2012). AvgOverlap computes the pair-wise average neuron overlap across methods. It is a set-based method and it does not consider the internal ranking of neurons into consideration. For example, the first top neuron and fifth top neuron in a ranking will get equal weight when compared with another ranking. Our second method, NeuronVote, inspired by Borda Count (Lippman, 2012), considers the order of neurons into consideration in comparing rankings. We call our methods compatibility metrics, which when given
a neuron ranking, provide its compatibility score with respect to other methods. The metrics give a high score to a method that is most aligned with the rankings of the other neuron interpretation methods. To build a deeper understanding of how any two methods relate in terms of the resulting neurons, we also present a pair-wise comparison of the discovered neurons across methods.
Formally, let  $S_{m} = n_{1}, n_{2}, \ldots, n_{s}$  represent the  $s$  top-ranked neurons for a method  $m$ , where  $s$  is a hyperparameter that can be adjusted to vary the number of top neurons selected for evaluation. We describe the compatibility metrics and the pair-wise comparison method as follows.
3.2.1 Average Overlap
AvgOverlap scores an interpretation method based on its average neuron overlap with other methods. We define the overlap  $o(m_1, m_2)$  between two methods as the intersection over union of their respective top-neuron sets:
$$
o \left(m _ {1}, m _ {2}\right) = \frac {\left| S _ {m _ {1}} \wedge S _ {m _ {2}} \right|}{\left| S _ {m _ {1}} \vee S _ {m _ {2}} \right|} \tag {7}
$$
AvgOverlap is defined as:
$$
\operatorname {A v g O v e r l a p} _ {m _ {i}} = \frac {1}{\mathcal {M} - 1} \sum_ {j = 1, j \neq i} ^ {\mathcal {M}} o \left(m _ {i}, m _ {j}\right) \tag {8}
$$
where  $\mathcal{M}$  is the set of all methods. A large value of AvgOverlap means that the ranking of that method is well aligned with the ranking of other neuron interpretation methods.
3.2.2 NeuronVote
AvgOverlap treats each neuron equally in a set of discovered neurons and ignores the ranking assigned by the methods. However, intuitively a neuron interpretation method whose top choice is endorsed by all other methods should get a higher score than an interpretation method whose 10th top choice is endorsed by all other methods. To take this into account, we follow the Borda count strategy (Lippman, 2012) and propose NeuronVote. It considers each neuron's ranking in the voting process. We first create  $S_{best}$ , an aggregated ranking based on the position of individual neurons in the rankings of various methods.
$$
S _ {b e s t} = \operatorname {a r g s o r t} \left(\sum_ {m} ^ {\mathcal {M}} s - \operatorname {i n d e x} \left(S _ {m}, n\right) \quad \text {f o r} n \in \mathcal {N}\right) \tag {9}
$$
where  $\text{argsort}$  is an ascending sort function and  $\text{index}$  gives the position of neuron  $n$  in the ranked list. The accumulated  $S_{best}$  represents an aggregation on weighted votes, with neurons appearing at the top of various methods' ranked lists getting a higher weight than others. Following the notation in AvgOverlap, we define NeuronVote $_{mi} = o(S_{mi}, S_{best})$ . The method with high NeuronVote implies that its ranking of neurons is most endorsed by other interpretation methods.
3.3 Pairwise Comparison
The compatibility metrics provide a holistic evaluation of a neuron interpretation method with respect to other methods. We further extend the analysis to pair-wise comparison to understand how the discovered neurons of a method related to another method. The pair-wise comparison is calculated as an intersection between the output of two methods i.e.  $o(m_1, m_2) = S_{m_1} \wedge S_{m_2}$
4.1 Settings