On the Pitfalls of Analyzing Individual Neurons in Language Models
Authors: Omer Antverg
Yonatan Belinkov
Venue: ICLR 2022 Poster (2022)
Keywords: NLP
interpretability
multilingual
individual neurons

Abstract: While many studies have shown that linguistic information is encoded in hidden word representations, few have studied individual neurons, to show how and in which neurons it is encoded.
Among these, the common approach is to use an external probe to rank neurons according to their relevance to some linguistic attribute, and to evaluate the obtained ranking using the same probe that produced it.
We show two pitfalls in this methodology:
    1. It confounds distinct factors: probe quality and ranking quality.
    We separate them and draw conclusions on each.
    2. It focuses on encoded information, rather than information that is used by the model.
    We show that these are not the same.
We compare two recent ranking methods and a simple one we introduce, and evaluate them with regard to both of these aspects.

ON THE PITFALLS OF ANALYZING INDIVIDUAL NEU- RONS IN LANGUAGE MODELS
Omer Antverg
Yonatan Belinkov
Technion - Israel Institute of Technology omer - verrgee wtwprrrrr........ii.oeeeee technion..nn.occ ac.il
Technion - Israel Institute of Technology be 1 i iwoo@tecccnnn w............................... ac.il
ABSTRACT
While many studies have shown that linguistic information is encoded in hidden word representations, few have studied individual neurons, to show how and in which neurons is encoded. Among these, the common approach is to use an external probe rank neurons according to their relevance to some linguistic attribute, and to evaluate the obtained ranking using the same probe that produced it. We show two pitfalls in this methodology: 1. It confounds distinct factors: probe quality and ranking quality. We separate them and draw conclusions on each. 2. It focuses encoded information, rather than information that is used by the model. We show that these are not the same. We compare two recent ranking methods and simple introduce, and evaluate them with regard to both of these aspects
INTRODUCTION
Many studies attempt to interpret language models by predicting different linguistic properties from word representations, an approach called probing classifiers (Adi et al.) [017]] Conneau et al.) 2018 inter alia). A growing body of work focuses on individual neurons within the representation, attempting to show in which neurons some information is encoded, and whether it is localized (concentrated in a small set of neurons) or dispersed. Such knowledge may allow us to control the model's output (Bau et al., 2019), to reduce the number of parameters in the model (Voita al., 2019; Sajjad et al. (020). and to gain a general scientific knowledge of the model. The common methodology is train probe to predict some linguistic attribute from a representation, and to use it, in different ways, to rank the neurons of the representation according to their importance for the attribute in question. The same probe is then used to predict the attribute, but using only the k-highest ranked neurons from the obtained ranking, and the probe's b accuracy in this scenario is considered as measure of the ranking's s quality (Dalvi al., 2019 Torroba Hennigen et al. 2020 Durrani et al., 2020). We see this framework as exhibiting Pitfall I: Two distinct factors are conflated the probe's 's classification quality and the quality of the ranking produces. good classifier may provide good results even if its ranking is bad, and an optimal ranking may cause average classifier to provide better results than a good classifier that given a bad ranking.
Another shortcoming of the current methodology, which we mark as Pitfall II, is the focus on encoded information, regardless of whether is actually used by the model in its language modeling task. A few studies (Elazar et al. 2021 Feder et al. 2020) have considered this question, and shown that encoded information not necessarlly being used for language modeling, but these do not look at individual neurons. We argue that in order evaluate ranking,
should also examine if, and how, the k-highest ranked neurons are used by the model for the attribute in question, meaning that modifying them would change the model's prediction- but with respect to that attribute only. This would allow some control over the model's output, and grant us parameter-level explanations of the model's 's decisions.
In this work, we analyze three neuron ranking methods. Since the ranking space is too large (768! in BERT's case), these methods provide approximations the problem and are non-optimal. Two of these methods LINEAR (Dalvi et al. 2019) and GAUSSIAN (Torroba Hennigen et al.) 2020)- rely on an external probe obtain a ranking: the first makes use of the internal weights of linear probe, while the second considers the performance of decomposable generative probe. The third is
DINEAR
code is available at: http:///tt////tttttc////e/eeeeee
Indi dual Neurons
Pitfalls

---

simple ranking method propose, PROBELESS, which ranks neurons according to the difference in their values across labels, and thus can be derived directly from the data, with no probing involved.
We experiment with disentangling probe quality and ranking quality, by using a probe from one method with a ranking from another method, and comparing the different proki-ranking combinations. We expose the problematic nature of the current methodology (Pitfall D). by showing that in some cases, a suitable probe which is given an intentionally bad ranking, or a random one, provides higher accuracy than another which is given its allegedly optimal ranking. We find that while the GUSSSIAN method generally provides higher accuracy, its probe's selectivity (Hewitt & Liang 2019) is lower, implying that performs the probing task by memorizing, which improves probing quality but not necessarily ranking quality. We further find that GAUSSIAN provides the best ranking for small sets of neurons, while LINEAR provides a better ranking for large sets.
We then turn to analyzing which ranking selects neurons that are used by the model, by applying interventions on the representation: we modify subsets of neurons from each ranking and measure- using novel metric we introduce- effect on language modeling w.r.t to the property in question. We highlight the need to focus on used information (Pitfall II): even though PROBELESS does not excel in the probing scenario, it selects neurons that are used by the model, more so than the two probing- based rankings. We find that there is an overlap between encoded information and used information, but they not the same, and argue that more attention should be given to the latter.
We primarily experiment with the M-BERT model (Devlin et al. 2019) on 9 languages and 13 morphological attributes, from the Universal Dependencies dataset (Zeman al., 2020). We also experiment with XLM-R (Conneau et al., 2020), and find that most of our results are similar between the models, with a few differences which we discuss. Our experiments reveal the following insights:
We show the need to separate between probing quality and ranking quality, via cases where inten- tionally poor rankings provide better accuracy than good rankings, due to probing weaknesses.
We present a new ranking method that is free of any probes, and tends to prefer neurons that are being used by the model, more so than existing probing-based rankings.
We show that there is overlap between encoded information and used information, but they are not the same.
NEURON RANKINGS AND DATA
We begin by introducing some notation. Denote the word representation space as H C IRd and an auxiliary task as a function F: : H + Z, for some task labels Z (e.g.. part-of-speech labels). Given a word representation h E H and some subset of neurons 5 C {1...., d), we use hs to denote the subvector of in dimensions S. For some auxiliary task and E N, we search for an optimal subset S* such that |S*| = k and his contains more information regarding F than any other subvector hss, |S-| = k. For the search task, we define neeurnnrranking as a permutation I[(d) on {1...., d} and consider the subset [I)
{Ii(d),, .... [[(d)x). One may wish to find an optimal ranking I1 (d) such that Vk, IT' (d) is the optimal subset with respect to F. However, finding an optimal ranking. or even an optimal subset, is ha-hard (Binshtok et al. 2007). Thus, we focus on several methods to produce rankings, which provide approximations to the problem, and compare them.
2.1 RANKINGS
The ranking methods compare include two rankings obtained from prior probing- prosing-based neuron- ranking methods, and a novel ranking we propose, based on data statistics rather than probing.
The first method, henceforth LINEAR (named linguistic correlation analysis in Dalvi et al. 2019 trains linear classifier the representations learn the task F. Then, it uses the trained classifier's weights to rank the neurons according to their importance for F. Intuitively, neurons with a higher magnitude of absolute weights should be more important, or contain more relevant information, for solving the task. Dalvi et al. (2019) showed that their method identifies important neurons through probing and ablation studies, and found that while the information distributed across neurons, the distribution is not uniform, meaning it is skewed towards the top-ranked neurons. In this work, we use a slightly modified version of the suggested approach (Appendix A.1).
LINEAR
A small enhancement the algorithm was presented in Durrani al. 2020
2

---

The second method, henceforth GAUSSIAN (Torroba Hennigen al. 2020), trains a generative classifier on the task F. based on the assumption that each dimension in {1.... d} is Gaussian-distributee
GAUSSIAN
Then, it makes use of the decomposability of the multivariate Gaussian distribution to greedily select the most informative neuron, according to the classifier's performance, every iteration. This way obtain full neuron ranking after training only once, while applying this greedy method to LINEAR would require retraining the probe d! times, which is clearly infeasible. Torroba Hennigen et al. (020)) found that most of the tasks can be solved using low number of neurons, but also noted that their classifier is limited due to the Gaussian distribution assumption.
The third neuron-ranking method we experiment with is based purely on the repre- sentations, with no probing involved, making it free of probing limitations (Belinkov) (002I) that might affect ranking quality. For every attribute label 2 E Z, we calculate g(z). the mean vector of all representations of words that possess the attribute and the value 2. Then, we calculate the element-win difference between the mean vectors,
PROBELESS
Σ 1l(x))=(<E)) Z
E d
(I)
and obtain ranking by arg-sorting i.e., the first neuron in the ranking corresponds to the highest value in r. For binary-labeled attributes, this is simply the difference in means. In the general case, PROBELESS assigns high values to neurons that are most sensitive to given attribute. We note that PROBELESS is very fast to use, as we are only limited by averaging and sorting, as opposed to training a classifier in LINEAR or the expensive greedy algorithm of GAUSSIAN.
2.2 DATA AND MODELS
Throughout our work, we follow the experimental setting of Torroba Hennigen al. (2020):
map the UD treebanks (Zeman et al. 2020) to the UniMorph schema (Kirov et al. 20188) using the mapping by McCarthy al. (2018). We select a subset of the languages used by Torroba Hennigen et al. 2020): Arabic, Bulgarian, English, Finnish, French, Hindi, Russian, Spanish and Turkish, to keep linguistic diversity. The tasks we experiment with are predictions of morphological attributes from these languages. Full data details are provided in Torroba Hennigen et al. (2020) and further data preparation steps are detailed in Appendix A.2 We process each sentence in pre-trained M-BERT and XLM-R (unless stated otherwise, all results are with M-BERT), and take word representations from layers 2. 7 and 12 of each model, to see if there are different patterns in the beginning, middle and end of the models. We end up with a total of 156 different configs (language X attribute layer) to test for each model. For words that are split during tokenization, we define their final representation to be the average over their sub-token representations. Thus, each word has one representation for each layer, of dimension d = 768. We do not mask any words throughout our work.
2.3 OVERLAPS
Before evaluating our rankings in different scenarios, we first characterize them by looking at the 100-highest ranked neurons (out of 768) from different rankings, across different configs.
Some neurons are important for an attribute across languages Since we work with multilingual models, we expect to overlap in the selected neurons for one attribute across different languages. Fig. [ shows that for PROBELESS this indeed the case, as some attributes share a large number of important neurons across languages. For example, number in Spanish and number in French share 70 of their 100 most important neurons, where the expected number for overlap of two random selections of neurons only 13 (Appendix (A.)). Compared the other two rankings (Appendix A.4). PROBELESS is the most consistent across languages, while GAUSSIAN rarely shows consistency. which may be a weakness.
Some neurons are unanimously important By looking at the overlaps between important neurons selected by different rankings for the same config. we observe that for all configs, the overlap between all three rankings surpasses the expected number (which is ~ 1.69; Appendix (A.3). meaning there neurons that are recognized by all three rankings as important.
We further see that in most cases, the greatest overlap is between LINEAR and PROBELESS. We find it reasonable, both of them aim select neurons that separate classes the best-one by classifier
3

---

and the other by data statisties while GAUSSIAN takes different approach, assuming a Gaussian distribution and selecting neurons only by performance. Examples from configs are shown in Fig.
Aiper
50.0
Doussian, Lncer
Doussion Probcccs
twrawe
Lincar, rrobicas
40.0
Number
90
30.0
Overlee
20.0
Tansa
10.0
Number
N2
rang.3
Figure 1: Layer 7 neurons overlap. using PROBELESS ranking. Blue squares are above the expected overlap between 2 rankings (Ap- pendix A.3), red are below. Major ticks are at- tributes, minor are languages.
Figure 2: Layer neurons overlap between ev- ery pair rankings, from 8 randomly selected configs. Gray and black dashed lines show the expected overlap between 2 and random rank- ings, respectively (Appendix A.3
There are more overlaps in XLM-R Performing the same analysis across languages on XLM-R (Appendix A.4). the overlap size is at least as the expected one between all config pairs, and usually greater. It may imply that XLM-R's representations can be pruned more easily than M-BERT's, since some neurons encode multiple attributes, and there is greater redundancy among the others.
PITFALL I: CLASSIFIERS VS. RANKINGS
We now turn to evaluating the rankings, and present the pitfalls in doing so. Given some ranking (d)). we would like to evaluate how well it sorts the neurons for the task F. Our first ranking-evaluatto approach is the standard probing approach from previous work (Dalvi et al. 2019 Torroba Hennigen cct al. 2020), where we expose the classifier to a subvector of the representation and evaluate how well it predicts the task. However, while previous work conflated rankings and classifiers -aitierss--iiall I-we are more careful: we separate the two, and pair each ranking with two classifiers, meaning that at least one of them is completely unrelated to the ranking.
Formally, for an increasing & E N, we train a classifier f H2 + Z to predict the task label, F(h), solely from hT(())))))) (the subvector of the representation in the top k neurons in ranking IT), ignoring the rest of the neurons] The assumption is that the better f performs, the more task-relevant information is encoded in hrT(d) Yet, the behaviour of f itself might affect results and conclusions about the ranking. Thus, we take classifier capabilities into consideration when analyzing results, and also measure selectivity (13.1.)). The process is further illustrated in Appendix A.5
3.1 EXPERIMENTAL SETUP
As classifiers, we experiment with both classifiers used by the first two ranking methods (LINEAR and GAUSSIAN). We use the hyperparameters reported in Durrani et al. (2220) Torroba Hennigen et al. (020)) for training the classifiers. As rankings, we experiment with the ranking methods described in 2 For each, to as top-to-bottom and bottom-to-top, respectively. To those we add random ranking baseline, resulting in different rankings overall. We compare all classifier-ranking combinations for each k.
the original ranking it produces and its reversed version, referred
Since both of the first two ranking methods are inherently tied to the classifier that was used to generate them, and the third ranking a classifier-neutral ranking, it can be used for a fair comparison between the classifiers.
We only probe into representations of words that possess the attribute, e.g., if the attribute is gender we do prohe into the representation of the word "pizza".
4

---

3.1.1 METRICS
Accuracy First, measure the accuracy of the probe's predictions. Since experiment with many different configs, we use the Wilcoxon signed-rank test (Wilcoxon) 1992) as a statistical
significance test determine whether certain combination of classifier and ranking is statistically significantly better than another combination.
Selectivity We also evaluate our probes by selectivity (Hewitt & Liang 2019). defined as the difference between the classifier's accuracy on the actual probing task and its accuracy on predicting random labels assigned to word types, called control task. Low selectivity implies that the probe can memorize the word-type-labe pair, and so high accuracy in the probing task does not necessarily en- tail the presence of the linguistic attribute. Thus, we prefer probes that both accurate and selective.
3.2 RESULTS
Across the 156 configs we experiment with, we observe three different accuracy patterns, demon- In these figures, each color represents combination of a classifier and ranking, where a solid line is used for the top-to-bottom version of the ranking and a dotted line is for the bottom-too of the configs follow the Standard pattern (Fig. Say. in which all top-to- boooootoom rankings are always better than the random ranking, which is always better than all bottom-to-top rankings. The other half consists of two surprising patterns, that demonstrate the inherent flaws in this ranking-evaluation approach. In the G>L pattern (Fig. (b). the GAUSSIAN classifier performs exceptionally well, pro- viding higher accuracy (after certain point) using a random or even a bottom-to- ranking, than the LINEAR classifier using its top-to-bottons ranking. the L>G pattern (Fig. 3c), the GAUSSIAN classifier fails quickly, and thus the LINEAR classifier provides higher accuracy using random or bottom-to-top ranking than the GAUSSIAN classifier using its top-to-bottom ranking.
strated in Figs. 3a 3c
version of it, and a dashed line is used for the random ranking. Almost half
Fig. 3d shows t-SNE (van der Maaten & Hinton] 2008) projection after performing K-means clustering on our 156 accuracy results, where each point represents accuracy results from one config (details on clustering procedure are in Appendix A.6). It shows three clusters of configs, that correspond to the three distinct patterns. On XLM-R R we see very similar results, and most configs follow the same pattern in each model (Appendix A.8 We now turn to analyze these results.
3.2.1 RANKING METHODS ARE INHERENTLY CONSISTENT
In most configs, each classifier provides better accuracy using a top-to-bottom ranking (solid lines) compared to the bottom-to-top version of the same ranking (same color, dotted line), and the random ranking (dashed lines) is in between. This is also seen in our statistical significance tests (Appendix A.7). We conclude that even if they are not optimal, all ranking methods we consider generally rank iask-informative neurons higher than non-informative ones.
3.2.2 WHICH CLASSIFIER IS BETTER?
In the Standard and G>L patterns (Figs. 3a 3b). GAUSSIAN achieves better accuracy than LINEAR when both of them use the same ranking (including top-to-bottom LINEAR), especially when using small sets of neurons. Our statistical significance tests (Appendix(A.. A.7) show that GAUSSIAN performs significantly better than LINEAR with 6 out of the 7 rankings we tried when using 10 neurons, with 5 rankings when using 50 neurons, and with 5 rankings when using 150 neurons. We now turn to analyze what makes GAUSSIAN more successful, and show some exceptions.
GAUSSIAN is memorizing Across all configs, LINEAR provides higher selectivity than GAUSSIAN using any ranking, after certain point (Appendix A.8). This means that GAUSSIAN tends to memorize the word-type-labe pair when solving the task. While this is apparent all configs, we note that specifically in the part-of-speech attribute there is large portion of function words, i.e., closed set labels (e.g., pronouns, determiners), meaning that memorization can significantly help solve the task. Thus, most configs involving part of speech belong to the G>L pattern. Since memorization is a trait of the classifier and not of the ranking, this pattern demonstrates the problematic nature of the current umking-evaluattin approach, as the results are highly dependent on the probe.
LINEAR is more stable On the other hand, pattern L>G shows that there are certain configs where GAUSSIAN is struggling model the distribution, resulting in mediocre accuracy soults-whian
5

---

Linear by Linear
Linear by Gaussian
Linear by random
Gaussian by Gaussian
Gaussian by random
0.95
0.9
0.90
0.8
accuracy
0.85
accuracy
0.7
0.80
0.6
0.75
0.5
0.70
25
100
125
150
100
125
150
neurons
neurans
(a) Bulgarian definiteness layer 7 (Standard patter)..
(b) Hindi part of speech layer 12 (G>L pattern..
Standard
0.85
G
L>G
accuracy
0.80
0.75
0.70
25
50
75
100
125
150
neurons
(c) Russian animacy layer (L>G pattern..
(d) I-SNE projection clustered probing results.
Clustering of the three different patterns 3d). and example of each of the patterns Bcp. Solid lines are top-to-bottom rankings; dashed are random rankings; dotted are bottom- to-top rankings. "X by Y" means classifier X using ranking Y. Some lines omitted for clarity; complementing figures can found Appendix A.8
Figure 3:
even start decreasing at some point-sometimes even below majority baseline, as seen in Fig. 3c This has also been mentioned in Torroba Hennigen et al. (2020), where was shown that in those configs, there are only a few (or no) dimensions that are informative for the attribute and are Gaussian- distributed. Thus, the GAUSSIAN classifier tries to model these distributions with the wrong tools, and fails. Poor modeling then leads to wrong predictions and low accuracy. In general, LINEAR behaves similarly across configs, making it more stable.
3.2.3 WHICH RANKING IS BETTER?
When looking at rankings, we would like to compare performance of the same classifier, using different rankings. We would expect that each classifier would perform best when using the ranking has generated. However, this is not always the case. As we can see in all patterns in Fig. 3, for small sets of neurons, LINEAR actually achieves better accuracy when using GAUSSIAN'S ranking (solid green) than its own ranking (solid orange). As the number of neurons increases, at some point its accuracy with its own ranking becomes higher than with GAUSSIAN'S 's ranking.
We suggest two explanations for this phenomenon: First, due to its greediness, the GAUSSIAN ranking not guaranteed to provide the optimal subset. For subset of size but as the size grows there are more subsets that are not taken into consideration in the algorithm, so it is more likely to miss the best sets. Second, GAUSSIAN assumes the embedding distribution to be Gaussian. On dimensions which are not of the contribution each neuron. So, a neuron is informative towards the attribute but is not coussian-distribun its addition the selected neurons set unlikely to improve performance. and thus it is not selected. This is a problem with a performance- maseed selection criterion, where the selection of neurons depends on the performance of the probe.
goes over all possibilities,
it makes less accurate evaluation

---

To summarize, it seems that GAUSSIAN is good selecting specific informative neurons, but misses the rest. While LINEAR'S 's ranking is not optimal (it is definitely worse then GAUSSIAN'S 's on small sets), it does seem to be more stable different sizes. PROBELESS provides decent performance (and is inherently consistent), but is usually behind the other two.
PITFALL II: ENCODED INFORMATION VS. USED INFORMATION
The variance of results in our probing experiments can mostly be attributed probing limitations (He- witt & Liang 2019 Belinkov 202T). and emphasizes the need to distinguish between two properties: the probe's 's classification quality, and the neuron-ranking quality. To isolate the latter, and to shed light on which ranking prefers neurons that are actually used by the model for the attribute in question (which is ignored by previous work-Pitfall II), we take a second ranking-evaluution approach: we intervene by modifying the representation in the neurons selected by the ranking, and observe if, and how, our intervention affects the language model output. This approach is more of causal one, inspired by similar prior work (Giulianelli et al. 2018 Elazar et al., 2021 Feder al. 2020 Lovering et al.; 2021 Rayfogel et al.] (021). We note that in this section, we use only the ranking itself, detaching it from any probes, thus removing classification quality from ranking comparisons.
Formally, for a representation h E H. ranking (Td) (corresponding to an attribute F) and an increasing & N, we intervene by modifying h only in the [T()) A] neurons, and observe the effect our intervention had on the model's output- the word prediction (given the modified representation) For vocabulary V. we divide the model to two components: E V + H and D H + V, such that for interventions in layer i,
composed of all of the layers of the model up to (including) i, and D is composed of all of the rest of the layers, including the classification head. After receiving a representation h = E(w) for word w E V, we modify to get a new representation h D(h) # D(h)), then D is using the modified information. The process is illustrated in Appendix A.9
However, knowing that the information is being used is not enough; we would like to know to what purpose it is being used, and to verify that it only affects the specific attribute we are interested in. Thus, we perform a finer-grainee analysis, and check if D(h') is similar, to some extent, to D(h). For that, we define lemmatizer T. analyzer A V + Z, which maps words to their task labels. Our goal is to intervene such that I(D(h))= ((D())), but A(D(h)) # A(D())) For example, if we intervene for tense, we would like the word "sleeps" to become "slept". : If this is the case, it implies that we have successfully identified where the task-relevant information that D uses is encoded, and how is being used.
V, which maps words their lemmas, and an
4.1 INTERVENTION METHODS
We consider two methods for modifying (d))) and compare them.
Ablation A common modification method is trying to remove the information by ablating some neurons (Morcos et al. 2018 Bau et al. [019 Lakretz al. 2019, meaning we set hz(d),. By that we amm to erase the information encoded in Nm(d))
Translation For a word w E V with attribute label = E Z, we attempt to translate its representation (in the geometric sense) to produce a word with attribute label z' Z, z * z by taking a step the direction of z', where bigger steps are applied to neurons that are marked as more important for the attribute. Formally, we apply the following protocol:
1. We calculate ((z) and ((²') as in eq.
2. We set
hn(a)s + (xg(((=)))()))) q(=)
where a nd is a log-scaled coefficients vector in the range [0, ,8. such that the coefficient of the highest-ranked neuron is 3 and that of the lowest-ranked neuron is 0, and 33 is hyperparameter.
Note that the rest of the neurons-those not in - unaffected. Using this protocol, we give each neuron its own special treatment- an approach that was not applied before (as far we know). This can be seen as a generalization of Gonen et al. (2020).
We apply the intervention on representations of all words that possess the attribute in the sentence.
7

---

4.2 EXPERIMENTAL SETUP
We handle the data the same way as in our probing experiments. However, since we analyze the model's predictions-which may be different from the original input-we do not have gold morphology labels anymore. Thus, for morphologically analyzing the model's predictions (L and A), we use spaCy (Honnibal et al. 2020). Out of the languages we used in our probing experiments, in this section we use only those that are supported by spaCy (English, Spanish and French). We calculate g==) based on the entire training set, and perform our interventions on the test set. We compare the same rankings we used in our probing experiments ($ 3.1
4.3 METRICS
Error rate For our intervention experiments, we first measure the error rate of the language model. We want error rate to be high, since high error rate means we modified parts of the representation that have been used by the model in its prediction.
Correct Lemma, Wrong Value (CLWV) While inspecting predictions that are wrong after inter- vening (D(N) f w, where is the true word), we categorize them by ((D()) and A(D())). If our intervention were successful, meaning we changed only the word's specific attribute, and not other information, then we expect to see L(D(h)) = ((D())) and A(D(h)) & A(D())); that is, correct lemma but wrong value (CLWV). For example, if the word "makes" becomes "made" when intervening for tense, then it is considered as a correct type of error, but if becomes "make" or "prepared" it does not. Thus, we define CLWV the portion of those errors out of all predictions.
4.4 RESULTS
4.4.1
ABLATION IS NOT EFFECTIVE
Across most configs, about 400 neurons from layer 2 and 200-300 neurons from layers 7 and 12 can be ablated without any implications on the output, meaning error rate remains the same; an example is shown in Appendix 10 Moreover, when error rate does grow, CLWV is very low. By qualitatively analyzing those errors we saw that most predicted words are common words, e.g., "and", "if" in English. After ablating 600-700 (80%-90% neurons from the representations, errors, but most of them are because the word is predicted as nonsensical punctuation. Another major concern is that in some configs, ablating by bottom-to-top ranking provides better results than by the top-to-bottom version of the same ranking. In general, there are no distinct differences between the rankings. Thus, from here on we focus on translation rather than ablation.
observe a lot of
4.4.2 TRANSLATION IS EFFECTIVE
Across all translation experiments (Fig. 4 shows one example, more are in Appendix A.II CLWV increases until a certain saturation point, after which it remains constant or drops a little This means that we reached neurons that are not relevant for the attribute, and modifying them can result in loss of other information- error rate grows while CLWV does not. Thus, we are interested in the CLWV value at the saturation point (higher is better), and in the number of neurons modified the saturation point (lower is better). We would also like the difference between the error rate and CLWV at the saturation point to be as small as possible. All terms considered, we perform a sweep search on the values of 3 in the range [1, 12] on a dev set. We find that low 33 values provide low CLWV, while high values provide higher CLWV but also widen the gap between error rate and CLWV. We find B = 8 to be a balanced point, and thus report test results with / = 8 in three configs in Table 1 and the rest of the configs in Appendix AI The results for XLM-R are given Appendix A.12
Compared to ablation, translating a relatively small number of neurons results in higher error rate, and these errors are closer to what we would expect. For example, translating only 50 neurons selected by PROBELESS in Spanish gender layer 2 results in 37% CLWV and 49% error rate, while ablating 50 neurons from the same config and ranking gives 0% CLWV errors and only 1% of error rate. We further note that unlike in ablation experiments, here our rankings are inherently consistent: across all configs, all top-to-bottom rankings perform better than the rest, while random rankings'
'We define "saturation point" the first point from which there are two consecutive points where the value increase by factor lower than 1.05.
8

---

error rate sometimes increases a little, and bottom-to-top rankings do not manage to affect the model's is one example, more are in Appendix A.III.
output at all (Fig. 4
by to Linear
by rendom
by bet Probeless Caussion
Table I: CLWV value at saturation point and number of neurons modified at the saturation point, using the translation method, with 3 = 8. In each cell, the three lines refer layers 2, 7 and 12 respectively.
It Probeless
Linear
Ita Caussian
0.6
traction predict.ons
seturation paint
0.5-
0.4-
LINEAR
GAUSSIAN
PROBELESS
0.3-
0.39, 60
0.26, 150
0.38, 30
0.2
English tense
0.37, 50
0,34, 70
0.34, 30
011
0.51, 60
0.41, 120
0.46, 30
0.28, 110
0.19, 100
0.35,
50
100
150
200
250
300
Spanish
neurons
0.23, 150 number
0.26, 50
0.20, 40
0.25, 30
0.16, 140
0.40, 80
Figure 4: Spanish gender layer 2, translation results with / 8. Solid lines are error rates, dashed are CsWVs. ttb and btt stand for top- to-bottom and bottom-to-top, respectively.
0,29, 50
0.25, 80
0.37, 50
Spanish gender
0.29, 50
0.31, 50
0.33, 30
0.26, 130
0.16, 110
0.35, 60
4.4.3 PROBELESS IS THE MOST EFFECTIVE RANKING FOR INTERVENTIONS
clear trend from our M-BERT results (Table 1 Fig. f and Appendix I1. is that in most cases, PROBELESS achieves higher CLWV values, and does using smaller number of neurons, than the other two rankings. Furthermore, its error rate is significantly higher than the other two. This implies that PROBELESS tends to select neurons that are being used by the model, more so than the other rankings. However, while it does select neurons that are relevant for the attribute in question (CLWV is relatively high), it also tends select neurons that are used by the model for other kinds of attributes (the difference between error rate and CLWV is relatively high). Among LINEAR and GAUSSIAN, LINEAR seems to have the upper hand, with higher CLWV values in most configs. This provides another evidence that the superiority of GAUSSIAN in the probing experiments may be due to the quality of its classifier, and specifically its memorization ability, rather than the quality of the ranking it produces, here only the ranking affects results. In XLM-R, it seems that LINEAR and PROBELES ELESS both have the lead, with GAUSSIAN falling behind (Appendix 12. We perform additional experiments monolingual models (Appendix A.13) where PROBELESS is again superior.
5 DISCUSSION AND CONCLUSION
In this work, we show two pitfalls with the common approach for ranking neurons according to their importance for morphological attribute, and compare different ranking methods that follow this approach. We show that to evaluate a ranking in a probing scenario, one should separate between the ranking itself and the quality of the classifier that is using the ranking- concentrated on encoded information Pitfall II II-we show that it is not the same information used by a model, by showing that GAUSSIAN is inferior in the interventions scenario, in contrast to our probing results. This implies that high probing accuracy does not necessarily entail that the information is actually important for the model. This conclusion is also present in prior work (Elazar et al. 2021 Feder et al. 2020 Ravfogel al. 2021). but it has been largely neglected in studies of individual neurons via probes. We propose a new, fast-to-use ranking method that relies solely on the data, without training any auxiliary classifier, and show that is valid, and prefers neurons that are being used by the model, more so than other ranking methods. We also propose a method for intervening within the model's representations such that it transforms the output a desired way.
I. While previous work
In our intervention experiments, modifying too many neurons results in more errors that are not related the true word. This proves the importance of looking into individual neurons, especially when trying to intervene in the inner workings of the model. For example, Gonen et al. (2020) try to change the language of a word by intervening with the representation, using the same translation method we use, but with the same coefficient for every neuron, and on the entire representation. Our results imply that they may get better results by using our finer-grained method.

---

Ethics statement Our work contributes to the effort of improving the interpretability of language models, and more generally of neural networks. Better explanations and controls over a model's outputs can ameliorate its fairesss, for example in the case of gender bias: intervention method can guide users on how to reduce such biases, by pointing to model components (neurons) responsible for gender and offering intervention methods control model behavior w.r.t partieular property. On the other hand, malicious actors could use such capability to increase discrimination. Exposing the capabilities may also help develop defense mechanisms.
Reproducibility statement All our results are reproducible using the code repository we will release. All experimental details, including hyperparameters, are reported in $3.1 4.1 and 4.2 As
language models, we used the implementation of the transformers library (Wolf al. 2020). We performed our experiments NVIDIA RTX 2080 Ti GPU. All data preparation details are reported $2. and Appendix A.2
ACKNOWLEDGMENTS
We thank Lucas Torroba Hennigen for his helpful comments. This research was supported by the ISRAEL SCIENCE FOUNDATION (grant No. 448/20) and by an Azrieli Foundation Early Career Faculty Fellowship. YB is supported by the Viterbi Fellowship in the Center for Computer Engineering at the Technion.
REFERENCES
Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, and Yoav Goldberg. Fine- Fineggraieed analysis of sentence embeddings using auxiliary prediction tasks. 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track P Proceedings. OpenReview.net, 2017. URL https: /ttp::::/ppeeevvieeee ew. net. ftott//formm///mmmmo d=
Anthony Bau, Yonatan Belinkov, Hassan Sajjad, Nadir Durrani, Fahim Dalvi, and James R. Glass. Identifying and controlling important neurons in neural machine translation. In 7th International Conference Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
wwwp:/wwwwwee.teeeeeeeeeeee wicpummmmmmiii1a d= d-H12-PaR5kx
Yonatan Belinkov. Probing classifiers: Promises, shortcomings, and alternatives. abs/2102.12452, 2021. URL
CoRR,
Maxim Binshtok, Ronen Brafman, Solomon Eyal Shimony, Ajay Martin, and Craig Boutilier. Computing optimal subsets. In AAAI, pp. 1231-1236, 2007.
Alexis Conneau, German Kruszewski, Guillaume Lample, Loic Barrault, and Marco Baroni. What you can cram into single $&.#* vector: Probing sentence embeddings for linguistic properties. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2126-21 136, Melbourne, Australia, 2018. Association for Computational Linguistics. doi: 10.18653/w1/P18- 1198. URL https: www aclweb. org/ wwt./an......ooo. anthol P18-1198
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Un- supervised cross-lingual representation learning at scale. In Proceedings of the 58th An- nual Meeting of the Association for Computational Linguistics, pp. 8440-8451, Online. July 2020. Association for Computational Linguistics. doi: 10. 1/20220///222.acll acl-main, 747. URL https:
a0t.:/2222...cmm...... 747
Fahim Dalvi, Nadir Durrani, Hassan Sajjad, Yonatan Belinkov, Anthony Bau, and James R. Glass. What is one grain of sand in the desert? analyzing individual neurons deep NLP models. The Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu. Hawaii, USA, January 27 February 1, 2019, pp. 6000-6333333 6317, AAAI Press, 2019. doi: 10.1609/aaai, wtw.::i..3311113111163 URL 0. 609 /a00////aa..... wwtaaa.vv333.
Thirty-Third AAAI Conference
https: / org
v 03331111.3063300
10

---

Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-train Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume / (Long and Short Papers), pp. 4171-4186, Minneapolis, Minnesota, 2019. Association for Computational Linguisties. doi: 10.1 18553/ 1423. URL hotptttss /www
aclweb. org/ wttg//nnhhloogy//IL
1423
Nadir Durrani, Hassan Sajjad, Fahim Dalvi, and Yonatan Belinkov. Analyzing individual neu- rons in pre-trained language models. In Proceedings of the 2020 Conference Empirical
Methods in Natural Language Processing (EMNLP), pp. 4865-4880, Online, 2020. Associa- tion for Computational Linguistics. doi: 10. (0.18663//1120000emmpm-mmm 7 1/www.acl ac web. org wwwp:/www..o.r/toooooooooooooeee
URL https
emn 1p pw-mpppmm...... metai..9955
Yanai Elazar, Shauli Ravfogel, Alon Jacovi, and Yoav Goldberg. Amnesic probing: Behavioral explanation with amnesic counterfactuals. Transactions of the Association for Computational Linguistics, 9:160-175, 2021.
Amir Feder, Nadav Oved, Uri Shalit, and Roi Reichart. Causalm: Causal model explanation through counterfactual language models. Computational Linguistics, pp. 1 1-52, 2020.
Mario Giulianelli, Jack Harding, Florian Mohnert, Dieuwke Hupkes, and Willem Zuidema. Under the hood: Using diagnostic classifiers investigate and improve how language models track agreement information. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pp. 240-248. Brussels, Belgium, 2018. Association for Computational Linguistics. doi: 10. 1118653vv1//IT 8 5426.
www ac web...ebb... org/
Bachology/Win 5426
Hila Gonen, Shauli Ravfogel, Yanai Elazar, and Yoav Goldberg. It's not Greek to mBERT: Inducing word-level translations from multilingual BERT. In Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pp. 45-56, Online, 2020. Association for Computational Linguistics. doi: 10. 11166///1/2200baccooolll 1.5. URL https!
//www.aclweet o http:////....aaoa////////...aaaa
John Hewitt and Percy Liang. Designing and interpreting probes with control tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 2733- 2743, Hong Kong, China, 2019. Association for Computational Linguisties, doi: 10.18653/v1/D19 1275. URL https:: /wwp///ww..... a web. wtt.///anooooog////1--1111111111
Matthew Honnibal, Ines Montani, Sofie Van Landeghem, and Adriane Boyd. spaCy: Industrial- strength Natural Language Processing in Python, 2020. httptttts///oi.rrggg/// 5281/
zenodo. 1212303
Christo Kirov, Ryan Cotterell, John Sylak Glassman, Géraldine Walther, Ekaterina Vylomova, Patrick Xia, Manaal Faruqui, Sabrina J. Mielke, Arya McCarthy, Sandra Kübler, David Yarowsky, Jason Eisner, and Mans Hulden. UniMorph 2.0: Universal Morphology. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan, 2018 European Language Resources Association (ELRA). URL https
http:/wwww...//tthooogggoooo 1299
Yair Lakretz, German Kruszewski, Thco Desbordes, Dieuwke Hupkes, Stanislas Dehacne, and Marco Baroni. The emergence of number and syntax units in LSTM language models. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 11-20, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 18.13653/v1/ N19-1002. bt pattts::/aclannthooogy. wwg.:y......///. www.NN1--N/02222
Charles Lovering, Rohan Jha, Tal Linzen, and Ellie Pavlick. Predicting inductive biases of fine- tuned models. In International Conference on Learning Representations, 2021. URL htthps
//openrevies net. (enpaarreieee.nettfoooooooooooo tttee//orrrmmiimmttaaaaaaa

---

Arya D. McCarthy, Miikka Silfverberg, Ryan Cotterell, Mans Hulden, and David Yarowsky. Marrying Universal Dependencies and Universal Morphology. In Proceedings of the Second Workshop Universal Dependencies (UDW 2018), pp. 91-101, Brussels, Belgium, 2018. Association for Computational Linguistics. doi: 10. 10.18653/v1/W1 11-11111111111111
URL htt.h:tp::/// www. ac web. org/
anthollogy/Wi1B66011
Ari Morcos, David GT Barrett, Neil C Rabinowitz, and Matthew Botvinick. On the importance of single directions for generalization. arXiv preprint arXhv:1803.06959, 2018.
Shauli Ravfogel, Grusha Prasad, Tal Linzen, and Yoav Goldberg. Counterfactual interventions reveal the causal effect of relative clause representations on agreement prediction. In Proceedings of the 25th Conference on Computational Natural Language Learning, pp. 194-209, Online, November 2021. Association for Computational Linguistics. URL htt..httss:///... //acc antho ogy org/2021 201./2221..co111 conll 1.15
Hassan Sajjad. Fahim Dalvi, Nadir Durrani, and Preslav Nakov. Poor man's bert: Smaller and faster transformer models. ArXiv, abs/2004.03844, 2020.
Lucas Torroba Henmigen, Adina Williams, and Ryan Cotterell. Intrinsic probing through dimension selection. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 197-216, Online, 2020. Association for Computational Linguistics. doi:
10.18653/v1/2020.co 20tp633ww12222.emmmmmmmmaaa emnlp-mainnnppp URL https: htop:://wwwwt www.ww.wwwwb.com org/ anthoooyyoooo22
emr1p main. 15
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Ma-
chine Learning Research, 9(86):257-2666 2008. http://ttt//mmt / http://mmm.orrppppp/////////////
vandermaaten html
Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 5797-5808, Florence, Italy, 2019. Association for Computational Linguistics. doi:
URL httthhtp::// / www. www.eb. org.
Frank Wilcoxon. Individual comparisons by ranking methods. In Breakthroughs in statistics, pp. 196-202. Springer, 1992.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gug- ger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38-45, Online, October 2020. Association for Computational Linguistics. doi: 10. 00111855/////0/ 12055///110000000mmmmmmm... URL www.emmnwwsssseseeeeemmmmmeemmee
ffate-of-the-ad
Daniel Zeman, Joakim Nivre, Mitchell Abrams, Elia Ackermann, Noëmi Aepli, Hamid Aghaei, Zeljko Agié, Amir Ahmadi, Lars Ahrenberg, Chika Kennedy Ajede, Gabriele Aleksandrawiččite, Ika Al- fina, Lene Antonsen, Katya Aplonova, Angelina Aquino, Carolina Aragon, Maria Jesus Aranzabe, Hórunn Amnrdóóttr, Gashaw Arutie, Jessica Naraiswari Arwidarasti, Masayuki Asahara, Luma Ateyah, Furkan Atmaca, Mohammed Attia, Aitziber Atutxa, Liesbeth Augustinus, Elena Badmaeva, Keerthana Balasubramani, Miguel Ballesteros, Esha Banerjee, Sebastian Bank, Verginica Barbu Mi- titelu, Victoria Basmov, Colin Batchelor, John Bauer, Seyyit Talha Bedir, Kepa Bengoetxea, Gözde Berk, Yevgeni Berzak, Irshad Ahmad Bhat, Riyaz Ahmad Bhat, Erica Biagetti, Eckhard Bick, Agné Bielinskiené, Kristín Bjarnadóttir, Rogier Blokland, Victoria Bobicev, Loic Boizou, Emanuel Borges Völker, Carl Börstell, Cristina Bosco, Gosse Bouma, Sam Bowman, Adriane Boyd, Kristina Brokaité, Aljoscha Burchardt, Marie Candito, Bernard Caron, Gauthier Caron, Tatiana Cavalcanti, Gülzen Cebiroglu Eryiğit, Flavio Massimiliano Cecchini, Giuseppe G. A. Celano, Slavomír Čéplö, Savas Cetin, Özlem Çetinoğlu, Fabricio Chalub, Ethan Chi, Yongseok Cho, Jinho Choi, Jayeol Chun, Alessandra T. Cignarella, Silvie Cinková, Aurélie Collomb, Çağn Çöltekii, Miriam Connor, Marine Courtin, Elizabeth Davidson, Marie-Catherine de Marneffe, Valeria de Paiva, Mehmet Oguz
12

---

Derin, Elvis de Souza, Arantza Diaz de Ilarraza, Carly Dickerson, Arawinda Dinakaramani, Bamba Dione, Peter Dirix, Kaja Dobrovoljc, Timothy Dozat, Kira Droganova, Puncet Dwivedi, Hanne Eckhoff, Marhaba Eli, Ali Elkahky, Binyam Ephrem, Olga Erina. Tomaž Erjavec, Aline Etienne, Wograine Evelyn, Sidney Facundes, Richárd Farkas, Marília Fernanda, Hector Fernandez. Alcalde, Jennifer Foster, Cláudia Freitas, Kazunori Fujita, Katarína Gajdošová, Daniel Galbraith, Marcos Garcia, Moa Gärdenfors, Sebastian Garza, Fabrício Ferraz Gerardi, Kim Gerdes, Filip Ginter, lakes Goenaga, Koldo Gojenola, Memduh Gökırmak, Yoav Goldberg, Xavier Gómez Guinovart, Berta González Saavedra, Bernadeta Griciüté, Matias Grioni, Loic Grobol, Normunds Grüzītis, Bruno Guillaume, Céline Baallote, Tunga Güngör, Nizar Habash, Hinrik Hafsteinsson, Jan Hajič, Jan Hajič jr., Mika Hämäläinen, Linh Hà Mý, Na-Rae Han, Muhammad Yudistira Hanifmuti, Sam Hardwick, Kim Harris, Dag Haug, Johannes Heinecke, Oliver Hellwig, Felix Hennig, Barbora Hladká, Jaroslava Hlaváčová, Florinel Hociung, Petter Hohle, Eva Huber, Jena Hwang. Takumi Ikeda, Anton Karl Ingason, Radu Ion, Elena Irimia, Qlájídé Ishola, Tomášs Jelínek, Anders Johannsen, Hildur Jónsdóttir, Fredrik Jørgensen, Markus Juutinen, Sarveswaran K, Hüner Kaşıkara, Andre Kaasen, Nadezhda Kabaeva, Sylvain Kahane, Hiroshi Kanayama, Jenna Kanerva, Boris Katz, Tolga Kayadelen, Jessica Kenney, Václava Kettnerová, Jesse Kirchner, Elena Klemen- tieva, Ame Köhn, Abdullatif Köksal, Kamil Kopacewicz. Timo Korkiakangas, Natalia Kotsyba, Jolanta Kovalevskaite, Simon Krek, Parameswari Krishnamurthy, Sookyoung Kwak, Veronika Laippala, Lucia Lam. Lorenzo Lambertino, Tatiana Lando, Septina Dian Larasati, Alexei Lavren- tiev, John Lee, Phùông L. Höng. Alessandro Lenci, Saran Lertpradit, Herman Leung, Maria I Levina, Cheuk Ying Li, Josie Li, Keying Li, Yuan Li, Kyung Tae Lim, Krister Lindén, Nikola Ljubešić, Olga Loginova, Andry Luthfi, Mikko Luukko, Olga Lyashevskaya, Teresa Lynn, Vivien Macketanz, Aibek Makazhanov, Michael Mandl, Christopher Manning, Ruli Manurung, Cătälina Märănduc, David Mareček, Katrin Marheinecke, Héctor Martínez Alonso, André Martins, Jan Mašek, Hiroshi Matsuda, Yuji Matsumoto, Ryan McDonald, Sarah McGuinness, Gustavo Mendonça, Niko Miekka, Karina Mischenkova, Margarita Misirpashayeva, Anna Missilä, Cätälin Mititelu, Maria Mitrofan, Yusuke Miyao, AmirHossein Mojiri Foroushani, Amirsaeid Moloodi, Simonetta Montemagni, Amir More, Laura Moreno Romero, Keiko Sophie Mori, Shinsuke Mori, Tomohiko Morioka, Shigeki Moro, Bjartur Mortensen, Bohdan Moskalevskyi. Kadri Muischnek, Robert Munro, Yugo Murawaki, Kaili Müürisep, Pinkey Nainwani, Mariam Nakhlé, Juan Ignacio Navarro Horñiacek, Anna Nedoluzhko, Gunta desppre-BBrzkalbe Luòng Nguyōn Thi, Huyên Nguyēn Thi Minh, Yoshihiro Nikaido, Vitaly Nikolaev, Rattima Nitisaroj, Alireza Nourian, Hanna Nurmi, Stina Ojala, Atul Kr. Ojha. Adédayò Olúökun, Mai Omura, Emeka Onwuegbuzia, Petya Osenova, Robert Östling, Lilja Øvrelid, Şaziye Betül Özates, Arzucan Özgür, Balkız Öztürk Başaran, Niko Parta- nen, Elena Pascual, Marco Passarotti, Agnieszka Patejuk, Guilherme Paulino-Passos, Angelika Peljak- Siyao Peng, Cenel-Augusto Perez, Natalia Perkova, Guy Perrier, Slav Petrov, Daria Petrova, Jason Phelan, Jussi Piitulainen, Tommi A Pirinen, Emily Pitler, Barbara Plank, Thierry Poibeau, Larisa Ponomareva, Martin Popel, Lauma Pretkalnipa, Sophie Prévost, Prokopis Prokopidis, Adam Przepiórkowski, Tina Puolakainen, Sampo Pyysalo. Peng Qi. Andriela Räabis, Alexandre Rademaker, Taraka Rama, Loganathan Ramasamy, Carlos Ramisch, Fam Rashel, Mo- hammad Sadegh Rasooli, Vinit Ravishankar, Livy Real, Petru Rebeja, Siva Reddy, Georg Rehm, Ivan Riabov, Michael Rießler, Erika Rimkuté, I Larissa Rinaldi, Laura Rituma, Luisa Rocha, Eiríkur Rögnvaldsson, Mykhailo Romanenko, Rudolf Rosa, Valentin Rosca, Davide Rovati, Olga Rudina, Jack Rueter, Kristján Rúnarsson, Shoval Sadde. Pegah Safari, Benoît Sagot, Alcksi Sahala. Shadi Saleh, Alessio Salomoni, Tanja Samardžić, Stephanie Samson, Manuela Sanguinetti, Dage Särg, Baiba Saulite, Yanin Sawanakunanon, Kevin Scannell, Salvatore Scarlata, Nathan Schneider, Sebastian Schuster, Djamé Seddah, Wolfgang Seeker, Mojgan Seraji, Mo Shen, Atsuko Shimada, Hiroyuki Shirasu, Muh Shohibussirri, Dmitry Sichinava, Einar Freyr Sigurosson, Aline Silveira, Natalia Silveira, Maria Simi, Radu Simionescu, Katalin Simk6, Mária Šimková, Kiril Simov, Maria Skachedubova, Aaron Smith, Isabela Antonio Stella, Milan Straka, Emmett Strickland, Jana Strnadová, Alane Suhr, Yogi Lesmana Sulestio, Umut Sulubacak, Shingo Suzuki, Zsolt Szántó, Dima Taji, Yuta Takahashi, Fabio Tam- burini, Mary Ann C. Tan, Takaaki Tanaka, Samson Tella, Isabelle Tellier, Guillaume Thomas, Lissi Torga, Marsida Toska, Trond Trosterud, Anna Trukhina, Reut Tsarfaty, Utku Türk, Francis Tyers, Sumire Uematsu, Roman Untilov, Zdeřka Urešová, Larraitz Uria, Hans Uszkoreit, Andrius Utka, Sowmya Vajjala, Daniel van Niekerk, Gertjan van Noord, Viktor Varga, Eric Villemonte de la Clergerie, Veronika Vincze, Aya Wakasa, Joel C. Wallenberg, Lars Wallin, Abigail Walsh, Jing Xian Wang, Jonathan North Washington, Maximilan Wendt, Paul Widmer, Scyi Williams,
Carolyn Spadine, Steinhór Steingrímsson,
13

---

Mats Wirén, Christian Wittern, Tsegay Woldemariam, Tak-sum Wong, Alina Wróblewska, Mary Yako, Kayo Yamashita, Naoki Yamazaki, Chunxiao Yan, Koichi Yasuoka, Marat M. Yavrumyan, Zhuoran Yu, Zdenčk Zabokrtsky, Shorouq Zahra, Amir Zeldes, Hanzhi Zhu, and Anna Zhuravl- eva. Universal dependencies 2.7, 2020. URLhttp: / / /hal. handle net net/11234/D 111//4214444444400 LINDAT/CLARIAII-CZ digital library at the Institute of Formal and Applied Linguistics (ÜFAL), Faculty of Mathematics and Physics, Charles University.
14

---

A APPENDIX
A.1 LINEAR METHOD MODIFICATION
In Dalvi al. 2019) after the probe has been trained, its weights are fed into neuron ranking algorithm. However, we observed that the original algorithm distributes the neurons equally among labels, meaning that each label would contribute the same number of neurons at each portion of the ranking, regardless of the amount of neurons that are actually important for this label. For example, for label A there are 10 important neurons and for label B there are only 2, then the first 10 neurons in the ranking would consist of 5 neurons for A and 5 for B. meaning that 3 non-important neurons are ranked higher than 5 important ones. Thus, we chose a different way to obtain the ranking: for each neuron, compute the mean absolute value of the |Z| weights associated with it, and sort the neurons by this value, from highest to lowest. In early experiments we found that this method empirically provides better results, and more adapted to large label sets.
A.2 DATA PREPARATION
We remove any sentences that would have a sub-token length greater than 512, the maximum allowed for M-BERT, the language model we use for generating representations. As in Torroba Hennigen et al. (020), we remove attribute labels that are associated with fewer than 100 word types in any of the data splits. This mostly removes function words, and we found makes it harder for probes to use memorization for solving the task. The morphological attributes experiment with include (in UniMorph annotations): Animacy, Aspect, Case, Definiteness, Gender and Noun Class, Mood, Number, Part of Speech, Person, Polarity, Possession, Tense and Voice.
A.3 EXPECTED OVERLAP BETWEEN RANDOM RANKINGS
Given i rankings, we calculate the expected size of overlap between the first M neurons across all rankings:
For selecting M neurons from the range {1. N}, let C E N be matrix such that in [n m. k] we keep the number of possibilities select neurons from the range [n] such that exactly & different neurons are selected by all i rankings, where k S S n and n, m. k >> 0. For calculating Ci(n, m, k] we first select the neurons from range [n] that are selected by all rankings, thus (x) possibilities. Then, for selecting the rest of the neurons, each ranking has select m
neurons from the remaining n k neurons, so there are (m=R) possibilities. From these, we want to substract the number of possibilities which there is at least one neuron that selected by all rankings, which is > C,(n k, m - j. Concluding, we compute C2(m, m, k) by:
)
n k
m- Ciln
G [n, m,
C Ciln-k,n k, mm k.
(3)
m
after initializing C(1, 1, Then, we calculate the expected number of overlapping neurons by:
m), Σ 122
k= k x C(n, m,
(4)
since Cln.m,k] is the probability to have exactly overlapping neurons. We thus get Ea 100) R (m))
13.02 and (768, 100) ~ 1.69.
A.4 OVERLAPS
Fig.1 shows overlaps between the 100 most important neurons chosen by LINEAR and GAUSSIAN for different configs. Both of them provide less overlaps then PROBELESS, with GAUSSIAN having almost no overlaps at all, showing its inconsistency across languages.
Fig. 6 presents the same analysis, but for XLM-R, with PROBELESS ranking (equivalent Fig. for M-BERT). We see far more overlaps in XLM-R, and no red squares (describing a lower overlap size than the expected one), implying that the information is more condensed in XLM-R than in M-BERT.
15

---

Spe
70
Tel 1a
90
90
Number
Number
POS
POS
-20
20
the A2 Ehee 120 SCOROO 22ee200225
Genstee
Manber
Ge-tter
Numbe
(a) LINEAR
(b) GAUSSIAN
Figure 5: Layer 7 neurons overlap using LINEAR and GAUSSIAN rankings.
Case
Gander
40
228a2 tettedee t tet seeeni ext
AMPIE
Case
Gender
Numbar
roS
Terse
Mike
Figure 6: XLM-R layer 7 neurons overlap, using PROBELESS. Blue squares are above expected value, red are below.
A.5 RANKING EVALUATION BY PROBING
Fig. fillustrates the process of evaluating rankings by probing. Two different actors affect the final accuracy: the ranking method, and the probe. In this work, we experiment with the 7 rankings described in 13.1aa as ranking methods, and the two probes used by LINEAR and GAUSSIAN probes. This methodology is problematic for evaluating rankings, as the choice probe significantly affects results.
A.6 CLUSTERING PROBING RESULTS
For each config out the 156 we experimented with, we have results of classifier-ranking combinations, each of length 150, the max & (number of neurons) we used. For clustering these results, we first remove all combinations involving a bottom-to-top ranking, as these add lot of noise to the clustering algorithm, making it focus irrelevant signals. Thus, shape [156, 8, 150]. We then reshape the matrix to shape [156, 8 x 150) and run K-means over it with K 3. Projecting the K-means output with t-SNE gives Figs. 3d and 9a
results matrix is of
A.7 STATISTICAL SIGNIFICANCE TESTS
Table shows the results of our statistical significance tests. The three rows in each cell correspond to using 10. 50 and 150 neurons. If there is in the [é cell, is means that the p-value under the null hypothesis that probe j is better than probe i lower than 0.05, when using the matching number of neurons. For example,
that there is an * in the first and second rows in the [0, 3) cell,
16

---

rankReene
LM
ranking method
Probe
(until
layer
Past Present
768
Figure 7: Ranking evaluation by probing: The language model creates word representation (e.g., of the word "was"), which is fed into neuron-ranking method, to rank its neurons according to their importance for some attribute (e.g., tense). The k-highest ranked neurons fed into probe, which is trained to predict the attribute.
Table 2: Statistical significance results. G, L, P and ttb are abbreviations for GAUSSIAN, LINEAR, PROBELESS and -o--to-bottomm respectively.
G by ttb
L by ttb
G by
L by ttb L
G by
L by ttb P
ttb
*
G by
#
L by ttb G
G by ttb L
I by itb L
G
L by ttb P
meaning we can confidently reject the hypothesis that LINEAR by LINEAR is better than GAUSSIAN by GAUSSIAN when using 10 50 neurons, but cannot do for 150 neurons. In fact, looking at the [3, 0] cell shows us that when using 150 neurons, GAUSSIAN by GAUSSIAN is not better than LINEAR by LINEAR.
While we do not show random and bottom-to- tooooomtttttop-ooo rankings in Table 2 for clarity, we asserted that each classifier statistically significantly better when using a top-to- top-to-bottom bottom ranking compared to random ranking, and when using random ranking compared to bottom-to-top ranking.
A.8 PROBING: ADDITIONAL RESULTS
Fig.
complements Fig.
including graph lines that missing in Fig. due its readability.
Fig. shows XLM-R probing results. XLM-R provides very similar results to M-BERT, apparent in Figs.
9a and 9h compared to Figs. 3dland and 3c respectively.
17

---

BAr000 gissnce
guennc Tear prosees
0.90
accJracy
accJracy
0.80
0.75
0.5
0.70
0.4
0.65
25
100
125
150
100
125
150
neurons
neurons
(a) Bulgarian definiteness layer
(b) Hindi part of speech layer 12.
0.90
0.85
accuracy
0.80
0.75
0.70
0.65
50
100
125
150
neurons
(c) Russian animacy layer 2.
Figure 8: Examples of each the patterns, with all graph lines (complementing Fig 3 Solid lines are top-to-bottom rankings; dashed are random rankings; dotted are bottom-to-top rankings. "X by Y" means classifier X using ranking Y.
Selectivity examples from both models are provided in Fig. 10 In all configs, both in M-BERT and XLM-R, LINEAR is significantly more selective than GAUSSIAN using any ranking.
A.9 RANKING EVALUATION INTERVENTIONS
Fig. II illustrates the process of evaluating rankings by probing. In this methodology, no external probes involved, and the rankings are evaluated with respect the degree of their selected neurons importance for the language model's output. In this work, we experiment with the 7 rankings described $3.1 as ranking methods, we intervene by ablations and translations $4.1
and evaluate
the model's outputs by the error rate and CLWV metrics ( (4.3)
A. ABLATION RESULTS
One ablation example is shown in Fig. 12 No matter the ranking, 400 neurons can be ablated with little impact on the output, and CLWV remains low. This behaviour generally consistent across all configs experimented with.
18

---

Linear by Linear
Linear by Gaussian
Linear by random
Gaussian by Gaussian
Gaussian by random
Standard
Foel
L>G
0.85
68/
accuracy
0.80
0.75
50
75
100
125
150
neurons
(a) t-SNE projection of clustered probing results, XLM-R.
(h) Russian animacy layer 2 accuracy, XLMMR.
Figure 9: Clustering of the three different patterns in XLM-R, R, and an example from one config. Solid lines are top-to-bottom rankings; dashed are random rankings; dotted are bottom-to-top rankings. Some lines omitted for clarity.
Linear by Linear
Linear by Gaussian
Linear by random
Gaussian by Gaussian
Gaussian by random
0.30
0.25
selectivity
selectivitt
0.4
0.20
0.15
0.3
0.10
0.2
0.05
25
50
100
125
150
neurons
neurons
(a) Russian animacy layer selectivity, XLM-R.
(b) Hindi part of speech layer 12 selectivity, M-BERT.
Figure 10: Two selectivity examples from M-BERT and XLM-R. Solid lines are top-to-bottom rankings, dashed are random rankings; dotted are bottom-to-top rankings. Some lines are omitted for clarity.
A.11 TRANSLATION RESULTS
All of M-BERT's translation results (complementing Table found in Table 3 and examples from two configs are in Fig. 13 As described $4.4.3 across most configs, PROBELESS achieves higher CLWV the saturation point, and gets there earlier (using less neurons), than the other two rankings- in contrast to probing results. Among the probing-base based rankings, LINEAR generally provides better results than GAUSSIAN.
We also note that there are certain attributes that seem harder to control for, e.g., English number and French tense.
19

---

Table 3: CLWV value at saturation point and number of neurons modified the saturation point, using the translation method on different configs, with B 8. In each cell, the three lines refer to layers 2, 7 and 12 respectively.
LINEAR
GAUSSIAN
PROBELESS
0.04,
0.02,
0.06, 90
English number
0.09, 50
0.07, 30
0.11,50
0.11, 130
0.04, 60
0.17, 110
0.39, 60
0.26, 150
0.38, 30
English
0.37,50 50
0.34, 70
0.34, 30
tense
0.51, 60
0.41, 120
0.46,
0.28, 110
0.19, 100
0.35, 60
Spanish
0.26,
0.20, 40
30
number
0.23, 150
0.16, 140
0.40,
0.20, 110
0.15, 140
0.27, 60
Spanish
0.16, 80
0.11, 70
0.20, 60
0.31, 130 tense
0.18,
0.33, 60
0.29,
0.25,
0.37, 50
Spanish
0.29, 50
0.31, 50
0,33, 30
gender
0.26, 130
0.16, 110
0.35, 60
0.19, 110
0.09, 150
0.25, 60
French
0.18, 50
0.17,
0.20, 30
number
0.07, 110
0.11, 150
0.33, 120
0.10, 110
0.01, 90
0.13, 70
French
0.10, 120
0.06, 100
0.08, 70
tense
0.14, 150
0.07, 110
0.15, 90
0.17, 80
0.17,80 80
0.22, 60
French gender
0.16, 40
0.16, 40
0.17, 30
0.14, 170
0.06, 140
0.20, 60
20

---

Bur
rankienee
Kwas
intervention
LM
ranking method
LM (from layer 1+1)
(until
layer
707.
was
Figure 11: Ranking evaluation by interventions: The language model creates a word representation (e.g.. of the word "was"), which fed into neuron-ranking method, rank neurons according to their importance for some attribute (c.g., tense). The k-highest ranked neurons are modified by an intervention (to a different color in the figure), and the new representation is fed into the rest of the language model's ' layers, to observe the final model's 's output.
by top avg
by bayes by random
by bottom avg
by werst
by top
by
fraction of all predictions
0.8
0.6
0.4
0.2
0.0
0
50
100
150
200
250
300
350
400
450
500
550
600
650
700
750
Figure 12: Spanish gender layer 2, ablation results. Solid lines are error rates, dashed are CLWVs.
A.12 XLM-R TRANSLATION RESULTS
XLM-R translation results (equivalent Table M-BERT) shown Table 4 The superiority of PROBELESS is not so clear in XLM-R compared to M-BERT, with LINEAR providing good competition. GAUSSIAN on the other hand, still falls behind.
We note that in XLM-R the CLWV values are somewhat lower compared to M-BERT. A possible explanation that could be the difference in tokenization between the models.
A.13 TRANSLATION MONOLINGUAL MODELS
As LINEAR and PROBELESS somewhat equal XLM-R, perform experiments additional models, to break the tie. We experiment with three monolingual models: bert-base-cased for English, http:///eeee--bas-aaan------eeee
for Spanish, and camembert-bon for French. The results reported in Table PROBELESS is superior in most of these experiments, both in terms of CLWV value and number of modified neurons when reaching the saturation point, with LINEAR coming second.
worth noting that in these models, saturation point values are generally higher, and achieved using fewer neurons, than in M-BERT and XLM-R. We believe it is due to their relative simplicity compared multilingual models, and their smaller vocabulary.
21

---

by
Linear by tib Probelinss
by Probeless by bitt Gaussian
by rib Probeless
by brt Linear
by ait Calaeiiin
by lent Linear
Caussiar
Caussian
0.6
0.8
fraction of all predictions
0.5
0.6
0.4-
0.3
0.4-
fraction
0.2
0.0
50
100
250
100
250
neurans
neurons
French number layer 12.
(b) English tense layer 12.
Figure 13: Translation results with _ = 8, from two different configs. Solid lines are error rates, dashed are CLWVs. ttb and btt stand for top-to-bottom and bottom-to-top, respectively.
Table 4: XLM-R CLWV value saturation point and number of neurons modified at the saturation point, using the translation method on different configs with 3 = 8. In each cell, the three lines refer layers 2, 7 and 12 respectively.
LINEAR
GAUSSIAN
PROBELESS
0.01, 130
0.00, 90
0.02, 80
English
0.03, 40
0.02, 30
0.04, 90
number
0.08, 90
0.06, 90
0.08, 90
0.22,
0.09, 190
0.21,30
English
0.34,
0.05, 50
0.09, 60
tense
0.35, 50
0,15, 130
50
0.29, 70
0.11, 80
0.34, 70
Spanish
0.23,30 30
0.20, 50
0.23, 20
number
0.33, 60
0.18, 120
0.33, 40
0.08, 90
0.00, 0
0.18, 70
Spanish tense
0.22,
0.07, 70
0.19, 40
0.16, 60
0.06, 150
0,24, 80
0.36, 60
0.17, 70
0.36, 20
Spanish
0.39, 70
0.30, 30
0.33, 30
gender
0.39, 60
0.30, 140
0.36, 20
0.14, 110
0.06, 100
0.24, 50
French
0.20, 80
0.14, 60
0.27, 80
number
0.30, 120
0.11, 130
0.31, 70
0.03, 120
0.01,
0.07, 30
French
0.10, 70
0.01, 40
0.06, 20
tense
0.10, 120
0.02, 110
0.06, 30
0.09, 130
0.02, 150
0.18, 50
French gender
0.16,
0.06,
3.
0.13,90
0.05, 100
0.19, 70
22

---

Table Monolingual models CLWV value saturation point and number of neurons modified at the saturation point, using the translation method on different configs with / = 8. In each cell, the three lines refer layers and 12 respectively.
LINEAR
GAUSSIAN
PROBELESS
0.09,
0.07,
0.09, 40
English
0.14,90 90
0.15, 90
0.17, 60
number
0.17, 110
0.08, 60
0.22, 90
0.49, 60
0.44, 70
0.47, 20
English
0.55, 70
0.43, 40
0.50, 20
tense
0.59, 40
0.55, 70
30
0.57, 30
0.57, 30
0.61,
Spanish
0.44, 20
0.40, 20
0.45, 20
number
0.49, 80
0.40, 110
0.54, 60
0.41, 60
0.35, 100
0.46, 40
Spanish tense
0.41, 90
0.28, 80
0.40, 60
0.47, 90
0.31, 110
0.57, 60
0.38, 30
0.35,
0.41,30
Spanish
0.35, 30
0.35, 30
0.37, 30
gender
0.34, 110
0.29, 100
0.40, 40
0.49, 10
0.50, 10
0.51, 10
French
0.46, 10
0.50, 10
0.46, 10
number
0.30, 10
0.31, 10
0.34, 10
0.36, 30
0.20, 50
0.13, 10
French
0.17,
0.26, 10
0.09, 10
tense
0.31, 50
0.13, 100
0.13, 50
0.28, 10
0.28, 20
0.29, 10
French gender
0.26, 10
0.26, 10
0.26, 10
0.17,90 90
0.14, 20
0.15, 40
23


Reviews:
[{"review_id": "mMYppXTyqH", "reviewer": "Reviewer_C78M", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n - The paper describes flaws in multiple recent high-profile NLP interpretability papers\n - The paper proposes an alternative mechanism (PROBELESS) which behaves differently from existing methods\n - Models studied are cross-lingual, unlike most probing work which has focused on English\n - Ranking and intervention experiments are generally convincing and well-executed \n\nWeaknesses:\n - Generally I found this paper was difficult to read. For example, the conflation between probe quality and ranking quality was not clear to me in the abstract/introduction, and perhaps an illustrative example or figure would help clarify this point. Another option would be to clarify >>early on<< that rankings are non-optimal due to e.g., greedy inference procedures which do not search over the whole space of rankings. Similarly, a figure representing the intervention procedure and CLWV metric might be useful.\n - Pitfall II is not especially novel, since it is well-known that most probing methods in NLP are fundamentally correlational in nature. Despite this, findings about the PROBELESS model and interventions are a useful contribution to the literature and fall in line with other work (e.g., ICLR paper from Lovering, et al. 2021) showing that extractability of information correlates with its usefulness in model predictions. \n\nQuestions:\n - Why is consistency across languages a desirable attribute here? (End of Section 2.2)\n - Can you re-explain what I'm supposed to be getting out of Figure 3a? Must be missing something.\n - In 3.2.2 it seems that \"Gaussian is memorizing\" is posed as a negative, but this is not evident to me. Why would memorization in this case (of a small set of function words) be considered a bad thing?\n\nWriting Comments/Nits: \n - In the introduction: \"We see this framework as <exhibiting> Pitfall I\"\n - The following sentences from Section 2.1 are difficult to interpret without reading the referenced paper: \"However, we observed that the original algorithm distributes the neurons equally among labels, meaning that each label would contribute the same number of neurons at each portion of the ranking, regardless of the amount of neurons that are actually important for this label\" \n - \"PROBELESS prefers neurons that separate different labels over neurons that have a similar value across labels\" could be rewritten as something like \"PROBELESS assigns high values to neurons which are most sensitive to a given attribute\"  \n - Appendix A.2: doesn't this assume that all rankings of PROBELESS are equally likely? This might not be true if some neurons vary more than others, for example. [Please correct me if I am wrong.] ", "summary_of_the_paper": "This paper describes two issues with existing probing methods within the NLP interpretability space and proposes an alternative approach which does not exhibit the same flaws.", "summary_of_the_review": "Although the paper is somewhat difficult to read (see suggestions), it provides a useful new approach to single-neuron probing and summarizes some weaknesses in recent existing work.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "iYYFOe6JyLO", "reviewer": "Reviewer_RqKU", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nStrengths:\n- PROBELESS seems to be genuinely an extremely good neuron ranking method and is simple and intuitive.\n-  The new methods for evaluating neuron ranking could be very useful, especially if the experimenter is interested in rankings that specialize in a particular task. \n- I was very interested in the finding that gaussian models memorize the data, likely because they rely on a smaller number of potential features.\n-  The analysis is extremely in depth. It provides both conjectures and well justified explanations for why certain rankings outperform others.\n-  I particularly like the analysis of the translation method of removing neurons, because it uses the saturation point where the lemma starts to disappear, rather than a single arbitrary point or some kind of AUC.\n-  The paper was generally clear to me.\n\nWeaknesses:\n- It’s not clear to me that the linear and gaussian methods specifically aim to find neurons that specialize (instead possibly targeting much less specialized but significant neurons), whereas their probeless method clearly targets neurons specializing in a particular task label. However, their metric is based on the neuron’s significance for a particular label, while controlling for preserving lemma information, which is not clearly the goal of the other methods. The major problem in this paper is that the decisions made in evaluation and ranking are not entirely well justified.  \n-  I’m not convinced that it’s fair to evaluate rankings based on whether the rankings assigned by one probe are useful to a different probe.\n- I would like to see some discussion of why single neuron analysis actually is preferred over analyzing distributed dimensions. In particular, the fact that translation outperforms the ablation method does indicate that the attributes in question could be encoded in a weighted distribution over neurons rather than directly into single neurons.\n- On page five, the authors hypothesize that the linear  probe method is more table because only a few dimensions are informative and gaussian distributed. I would consider this a conjecture, but I would like to see experiments that might support this hypothesis.\n\nMinor:\n- I would have liked to see an explanation of the tasks that were used in section 2.2.\n\nQUESTIONS:\n- How are A, L (the attribute and lemma labeler) trained?\n", "summary_of_the_paper": "This paper responds to several recent works focused on identifying important individual neurons for particular classifying tasks. They consider 2 existing methods which rely on an external probe to rank the neurons in a network. They also introduce a method that does not rely on a probe, instead ranking neurons according to the difference between their values across labels.\n\nThe primary focus in this paper is on two claimed flaws of the existing neuron ranking methods. The first is that the authors feel evaluation of the neuron rankings is unfair because a high quality probe can have higher performance on a worse ranking, simply because it is better able to take advantage of the data that is offered. The second is that not all rankings actually point to neurons that are specialized for a particular task or label, instead just indicating highly informative neurons.\n\nIn analyzing these two flaws, the authors develop evaluation metrics for the rankings of these neurons. They consider a ranking better if the neurons encode information that is specific to the label while maintaining the particular lemma being encoded. They also consider the ranking to be better if removing the features indicated damages the performance of the language model. They test two ways of modifying the network to remove important neurons, first by ablating the neurons and then by learning a geometric translation that moves a word towards a different attribute label.\n\nThey also consider the ranking better if a ranking from top to bottom outperforms a random ranking, which outperforms a reversed ranking. This seems like a fairly weak standard to hold a ranking to, but some rankings fail to adhere to it.", "summary_of_the_review": "This paper is an in-depth analysis with new perspectives on how we should think about neuron ranking. It does not discuss much the difference between their philosophy towards the purpose of neuron ranking, compared to the implicit philosophy in the prior works which they compare. Overall it seems valuable but may not be entirely fair to the existing literature.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "a908a9GCRe", "reviewer": "Reviewer_zE55", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n# Strengths\n\nThis paper identifies issues with existing methods for ranking neurons containing linguistic attributes and presents a detailed experiment showing it empirically. To address these issues, a new ranking method is proposed and it is effective.\n\n# Weaknesses\n\nn/a\nThis paper identifies issues with existing methods for ranking neurons containing linguistic attributes and proposes an effective solution.\n", "summary_of_the_paper": "This paper identifies two pitfalls of existing methods for ranking individual neuron contributions to certain linguistic attributes, and shows empirically that these pitfalls indeed exist. Additionally, it proposes a new ranking method free of these pitfalls, and shows its effectiveness.\n", "summary_of_the_review": "This paper identifies issues with existing methods for ranking neurons containing linguistic attributes and proposes an effective solution.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "O34_sb3W7v", "reviewer": "Reviewer_LRS7", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper begins by first explaining the background of the two methods in the literature, Linear (Dalvi et al 2019) and Gaussian (Torroba Hennigen et al 2020).\nThen, the authors describe their proposed \"Probeless\" method, and then dive deep into the results and how the above two methods suffer in view of the pitfalls.\n\n## The good\n\nOne great thing about this paper is how methodical the authors are in their experimental setup. I liked the idea of disentangling the rankings from the classifiers and using them together to see whether an expected ranking (top-to-bottom) is actually better than a worse ranking (random or bottom-to-top). The choice of the metrics also make sense to me. The intervention experiments (Pitfall II) are really interesting, as I think the proposed method could be useful for other related works too. It is indeed very crucial to explore whether the \"top-ranked\" neurons are actually contributing to the language model output. The results are very interesting - it seems ablation is not a good intervention approach, whereas Translation is a much better strategy. Similar methods have been used in representational probing literature (Amnesic Probing, https://arxiv.org/abs/2006.00995), it would be great to add a note on how they compare.\n\n## The bad\n\nThe paper can perhaps present the results/discussion in a better way, as several aspects are not clear to me.\n\n### Section 2.1\n\n- The authors mention they do not exactly use the method shown by Dalvi et al 2019 for Linear, and propose an alternate way to use Linear by computing the mean absolute value of the weights associated with it. Does this enhancement mean it is different from the original Linear model? It would be useful to extrapolate this difference in the paper.\n- In the definition of \"Probeless\", it is not clear what is \\hat{z}. Perhaps the authors indicate it is the mean of the words that does not possess the attribute/label information? Maybe I'm missing something obvious.\n\n### Section 2.3\n\n- \"We expect to see overlap in the selected neurons\" - it is not clear why we should expect to see an overlap among multiple languages. Any explanation/citation to back this claim up?\n\n### Section 3.2\n\n- The results section is a bit hard to parse given the introduction of two new concepts, G>L and L>G. Perhaps the authors should clearly state what these are earlier (it becomes slightly clear after reading Section 3.2.2 that G>L corresponds to Gaussian being better than Linear when it is paired with worse rankings than optimal, and vice-versa))\n- Possibly the ordering of the results is incorrect - the t-SNE plot explanation could come after explaining 3.2.1 and 3.2.2\n\n### Section 3.2.2\n\n- Gaussian memorizing in parts-of-speech attribute is expected, as many POS tags can be simply predicted by learning the type of words (POS tagging can be solved by non-neural CRF taggers, and neural probes always gets very high scores in this task anyway). I don't know how this makes Gaussian a \"bad\" probe. In fact, the inability of learning the basic POS tag associations by Linear probe suggests its weakness, not strength. This suggests a capacity issue.\n- Thus, G>L and L>G scenarios are highly task dependent. This calls for their study relative to the task and not in aggregate.\n\n### Section 3.2.3\n\n- The authors say \"Probeless provides decent performance (and is inherently consistent)\" - however I could not find any results of Probeless probe in Figure 3.\n\n### Section 4.4.3\n\n- I am skeptical of the claims in this section as it seems the numbers are cherry-picked. From Table 1, the Linear probe seems to be better than Probless in English tense. In Appendix Table 3, however, authors show more scenarios where Probeless tops other methods. This table should be moved to the main page.\n- However, the claims do not hold well in terms of XLM-R model, as Linear is comparable to the Probless method (gets 3 tasks vs 4 in the latter).\n\n### General comments\n\n- The paper lacks a thorough comparison with recent work. For example, I totally agree in the conclusion that high probing accuracy does not necessarily entail the information is actually useful for the model. Similar investigations have been made in the representational probing literature (Pareto Probing: Trading-Off Accuracy and Complexity, EMNLP 2020)\n\n### Experiment Suggestion\n\n- The authors investigated the results primarily with M-BERT, and they also investigate with XLM-R but the results are not so clear in the latter model. The authors can perhaps also investigate one more similar model, say M2M, to be a tie-breaker for a better conclusion.\n", "summary_of_the_paper": "The paper revisits two methods, Linear and Gaussian, as described in the literature, to probe language models based on individual neurons. The paper suggests that these methods contain two limitations - 1. by ranking the neurons on a linguistic task, the methods conflate between ranking quality and the probe's classification quality; 2. the probing methods do not take into account whether the individual neurons are at all used by the model in the downstream linguistic tasks. Finally, the paper presents a new probing method, which does not rely on training a classifier, and shows that this simple method is able to discern among the neurons better than the preceding two methods in the literature.\n", "summary_of_the_review": "Overall I found the paper to be quite illuminating on highlighting the different aspects of probing with individual neurons. However, the paper requires more polish in presenting the results and discussions of their work. I believe the finding is important for the community, especially the intervention analysis. However, I'm a bit skeptical of the claims (as highlighted in my review). Improving the writing of the paper and performing an additional model experiment could be useful to make the paper strong for acceptance.\n\nEdit: Following the rebuttal, I'm increasing my original score of 5 to 8\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper analyses interpretation methods that use probes to evaluate the information in individual neurons of a deep network and shows that it confounds probe quality and ranking quality, and encoded information and used information. The paper proposes a new method which does not suffer from the same drawbacks. The reviewers were positive about this paper, and the discussion between the reviewers and authors resulted in the authors adding multiple clarifications. I ask the authors to try to optimize the paper for clarity further. I recommend acceptance.
Decision: Accept (Poster)