On the Pitfalls of Analyzing Individual Neurons in Language Models

1 INTRODUCTION . Many studies attempt to interpret language models by predicting different linguistic properties from word representations , an approach called probing classifiers ( Adi et al. , 2017 ; Conneau et al. , 2018 , inter alia ) . A growing body of work focuses on individual neurons within the representation , attempting to show in which neurons some information is encoded , and whether it is localized ( concentrated in a small set of neurons ) or dispersed . Such knowledge may allow us to control the model ’ s output ( Bau et al. , 2019 ) , to reduce the number of parameters in the model ( Voita et al. , 2019 ; Sajjad et al. , 2020 ) , and to gain a general scientific knowledge of the model . The common methodology is to train a probe to predict some linguistic attribute from a representation , and to use it , in different ways , to rank the neurons of the representation according to their importance for the attribute in question . The same probe is then used to predict the attribute , but using only the k-highest ranked neurons from the obtained ranking , and the probe ’ s accuracy in this scenario is considered as a measure of the ranking ’ s quality ( Dalvi et al. , 2019 ; Torroba Hennigen et al. , 2020 ; Durrani et al. , 2020 ) . We see this framework as exhibiting Pitfall I : Two distinct factors are conflated—the probe ’ s classification quality and the quality of the ranking it produces . A good classifier may provide good results even if its ranking is bad , and an optimal ranking may cause an average classifier to provide better results than a good classifier that is given a bad ranking . Another shortcoming of the current methodology , which we mark as Pitfall II , is the focus on encoded information , regardless of whether it is actually used by the model in its language modeling task . A few studies ( Elazar et al. , 2021 ; Feder et al. , 2020 ) have considered this question , and shown that encoded information is not necessarily being used for language modeling , but these do not look at individual neurons . We argue that in order to evaluate a ranking , one should also examine if , and how , the k-highest ranked neurons are used by the model for the attribute in question , meaning that modifying them would change the model ’ s prediction—but with respect to that attribute only . This would allow some control over the model ’ s output , and grant us parameter-level explanations of the model ’ s decisions . In this work , we analyze three neuron ranking methods . Since the ranking space is too large ( 768 ! in BERT ’ s case ) , these methods provide approximations to the problem and are non-optimal . Two of these methods—LINEAR ( Dalvi et al. , 2019 ) and GAUSSIAN ( Torroba Hennigen et al. , 2020 ) —rely on an external probe to obtain a ranking : the first makes use of the internal weights of a linear probe , while the second considers the performance of a decomposable generative probe . The third is a simple ranking method we propose , PROBELESS , which ranks neurons according to the difference in their values across labels , and thus can be derived directly from the data , with no probing involved . 1Our code is available at : < ANONYMIZED > We experiment with disentangling probe quality and ranking quality , by using a probe from one method with a ranking from another method , and comparing the different probe–ranking combinations . We expose the problematic nature of the current methodology ( Pitfall I ) , by showing that in some cases , a suitable probe which is given an intentionally bad ranking , or a random one , provides higher accuracy than another which is given its allegedly optimal ranking . We find that while the GAUSSIAN method generally provides higher accuracy , its probe ’ s selectivity ( Hewitt & Liang , 2019 ) is lower , implying that it performs the probing task by memorizing , which improves probing quality but not necessarily ranking quality . We further find that GAUSSIAN provides the best ranking for small sets of neurons , while LINEAR provides a better ranking for large sets . We then turn to analyzing which ranking selects neurons that are used by the model , by applying interventions on the representation : we modify subsets of neurons from each ranking and measure— using a novel metric we introduce—the effect on language modeling w.r.t to the property in question . We highlight the need to focus on used information ( Pitfall II ) : even though PROBELESS does not excel in the probing scenario , it selects neurons that are used by the model , more so than the two probing-based rankings . We find that there is an overlap between encoded information and used information , but they are not the same , and argue that more attention should be given to the latter . We primarily experiment with the M-BERT model ( Devlin et al. , 2019 ) on 9 languages and 13 morphological attributes , from the Universal Dependencies dataset ( Zeman et al. , 2020 ) . We also experiment with XLM-R ( Conneau et al. , 2020 ) , and find that most of our results are similar between the models , with a few differences which we discuss . Our experiments reveal the following insights : • We show the need to separate between probing quality and ranking quality , via cases where intentionally poor rankings provide better accuracy than good rankings , due to probing weaknesses . • We present a new ranking method that is free of any probes , and tends to prefer neurons that are being used by the model , more so than existing probing-based rankings . • We show that there is an overlap between encoded information and used information , but they are not the same . 2 NEURON RANKINGS AND DATA . We begin by introducing some notation . Denote the word representation space as H ⊆ Rd and an auxiliary task as a function F : H → Z , for some task labels Z ( e.g. , part-of-speech labels ) . Given a word representation h ∈ H and some subset of neurons S ⊆ { 1 , ... , d } , we use hS to denote the subvector of h in dimensions S. For some auxiliary task F and k ∈ N , we search for an optimal subset S∗ such that |S∗| = k and hS∗ contains more information regarding F than any other subvector hS′ , |S′| = k. For the search task , we define neuron-ranking as a permutation Π ( d ) on { 1 , ... , d } and consider the subset Π ( d ) [ k ] = { Π ( d ) 1 , ... , Π ( d ) k } . One may wish to find an optimal ranking Π∗ ( d ) such that ∀k , Π∗ ( d ) [ k ] is the optimal subset with respect to F . However , finding an optimal ranking , or even an optimal subset , is NP-hard ( Binshtok et al. , 2007 ) . Thus , we focus on several methods to produce rankings , which provide approximations to the problem , and compare them . 2.1 RANKINGS . The ranking methods we compare include two rankings obtained from prior probing-based neuronranking methods , and a novel ranking we propose , based on data statistics rather than probing . LINEAR The first method , henceforth LINEAR ( named linguistic correlation analysis in Dalvi et al . 2019 ) ,2 trains a linear classifier on the representations to learn the task F . Then , it uses the trained classifier ’ s weights to rank the neurons according to their importance for F . Intuitively , neurons with a higher magnitude of absolute weights should be more important , or contain more relevant information , for solving the task . Dalvi et al . ( 2019 ) showed that their method identifies important neurons through probing and ablation studies , and found that while the information is distributed across neurons , the distribution is not uniform , meaning it is skewed towards the top-ranked neurons . In this work , we use a slightly modified version of the suggested approach ( Appendix A.1 ) . GAUSSIAN The second method , henceforth GAUSSIAN ( Torroba Hennigen et al. , 2020 ) , trains a generative classifier on the task F , based on the assumption that each dimension in { 1 , ... , d } 2A small enhancement to the algorithm was presented in Durrani et al . ( 2020 ) . is Gaussian-distributed . Then , it makes use of the decomposability of the multivariate Gaussian distribution to greedily select the most informative neuron , according to the classifier ’ s performance , at every iteration . This way we obtain a full neuron ranking after training only once , while applying this greedy method to LINEAR would require retraining the probe d ! times , which is clearly infeasible . Torroba Hennigen et al . ( 2020 ) found that most of the tasks can be solved using a low number of neurons , but also noted that their classifier is limited due to the Gaussian distribution assumption . PROBELESS The third neuron-ranking method we experiment with is based purely on the representations , with no probing involved , making it free of probing limitations ( Belinkov , 2021 ) that might affect ranking quality . For every attribute label z ∈ Z , we calculate q ( z ) , the mean vector of all representations of words that possess the attribute and the value z . Then , we calculate the element-wise difference between the mean vectors , r = ∑ z , z′∈Z |q ( z ) − q ( z′ ) | , r ∈ Rd ( 1 ) and obtain a ranking by arg-sorting r , i.e. , the first neuron in the ranking corresponds to the highest value in r. For binary-labeled attributes , this is simply the difference in means . In the general case , PROBELESS assigns high values to neurons that are most sensitive to a given attribute . 2.2 DATA AND MODELS . Throughout our work , we follow the experimental setting of Torroba Hennigen et al . ( 2020 ) : we map the UD treebanks ( Zeman et al. , 2020 ) to the UniMorph schema ( Kirov et al. , 2018 ) using the mapping by McCarthy et al . ( 2018 ) . We select a subset of the languages used by Torroba Hennigen et al . ( 2020 ) : Arabic , Bulgarian , English , Finnish , French , Hindi , Russian , Spanish and Turkish , to keep linguistic diversity . The tasks we experiment with are predictions of morphological attributes from these languages . Full data details are provided in Torroba Hennigen et al . ( 2020 ) and further data preparation steps are detailed in Appendix A.2 . We process each sentence in pre-trained M-BERT and XLM-R ( unless stated otherwise , all results are with M-BERT ) , and take word representations from layers 2 , 7 and 12 of each model , to see if there are different patterns in the beginning , middle and end of the models . We end up with a total of 156 different configs ( language × attribute × layer ) to test for each model . For words that are split during tokenization , we define their final representation to be the average over their sub-token representations . Thus , each word has one representation for each layer , of dimension d = 768 . We do not mask any words throughout our work .

Summary: This paper responds to several recent works focused on identifying important individual neurons for particular classifying tasks. They consider 2 existing methods which rely on an external probe to rank the neurons in a network. They also introduce a method that does not rely on a probe, instead ranking neurons according to the difference between their values across labels.  The primary focus in this paper is on two claimed flaws of the existing neuron ranking methods. The first is that the authors feel evaluation of the neuron rankings is unfair because a high quality probe can have higher performance on a worse ranking, simply because it is better able to take advantage of the data that is offered. The second is that not all rankings actually point to neurons that are specialized for a particular task or label, instead just indicating highly informative neurons.  In analyzing these two flaws, the authors develop evaluation metrics for the rankings of these neurons. They consider a ranking better if the neurons encode information that is specific to the label while maintaining the particular lemma being encoded. They also consider the ranking to be better if removing the features indicated damages the performance of the language model. They test two ways of modifying the network to remove important neurons, first by ablating the neurons and then by learning a geometric translation that moves a word towards a different attribute label.  They also consider the ranking better if a ranking from top to bottom outperforms a random ranking, which outperforms a reversed ranking. This seems like a fairly weak standard to hold a ranking to, but some rankings fail to adhere to it.