Year: 2019

Abstract:
Neural machine translation (NMT) models learn representations containing substantial linguistic information. However, it is not clear if such information is fully distributed or if some of it can be attributed to individual neurons. We develop unsupervised methods for discovering important neurons in NMT models. Our methods rely on the intuition that different models learn similar properties, and do not require any costly external supervision. We show experimentally that translation quality depends on the discovered neurons, and find that many of them capture common linguistic phenomena. Finally, we show how to control NMT translations in predictable ways, by modifying activations of individual neurons.

heading: 1 INTRODUCTION
text: Neural machine translation (NMT) systems achieve state-of-the-art results by learning from large amounts of example translations, typically without additional linguistic information. Recent studies have shown that representations learned by NMT models contain a non-trivial amount of linguistic information on multiple levels: morphological (Belinkov et al., 2017a; Dalvi et al., 2017), syntactic (Shi et al., 2016b), and semantic (Hill et al., 2017). These studies use trained NMT models to generate feature representations for words, and use these representations to predict certain linguistic properties. This approach has two main limitations. First, it targets the whole vector representation and fails to analyze individual dimensions in the vector space. In contrast, previous work found meaningful individual neurons in computer vision (Zeiler & Fergus, 2014; Zhou et al., 2016; Bau et al., 2017, among others) and in a few NLP tasks (Karpathy et al., 2015; Radford et al., 2017; Qian et al., 2016a). Second, these methods require external supervision in the form of linguistic annotations. They are therefore limited by available annotated data and tools.
In this work, we make initial progress towards addressing these limitations by developing unsupervised methods for analyzing the contribution of individual neurons to NMT models. We aim to answer the following questions:
• How important are individual neurons for obtaining high-quality translations? • Do individual neurons in NMT models contain interpretable linguistic information? • Can we control MT output by intervening in the representation at the individual neuron level?
To answer these questions, we develop several unsupervised methods for ranking neurons according to their importance to an NMT model. Inspired by work in machine vision (Li et al., 2016b), we hypothesize that different NMT models learn similar properties, and therefore similar important neurons should emerge in different models. To test this hypothesis, we map neurons between pairs of trained NMT models using several methods: correlation analysis, regression analysis, and SVCCA, a recent method combining singular vectors and canonical correlation analysis (Raghu et al., 2017). Our mappings yield lists of candidate neurons containing shared information across models. We
∗Equal contribution
then evaluate whether these neurons carry important information to the NMT model by masking their activations during testing. We find that highly-shared neurons impact translation quality much more than unshared neurons, affirming our hypothesis that shared information matters.
Given the list of important neurons, we then investigate what linguistic properties they capture, both qualitatively by visualizing neuron activations and quantitatively by performing supervised classification experiments. We were able to identify neurons corresponding to several linguistic phenomena, including morphological and syntactic properties.
Finally, we test whether intervening in the representation at the individual neuron level can help control the translation. We demonstrate the ability to control NMT translations on three linguistic properties—tense, number, and gender—to varying degrees of success. This sets the ground for controlling NMT in desirable ways, potentially reducing system bias to properties like gender.
Our work indicates that not all information is distributed in NMT models, and that many humaninterpretable grammatical and structural properties are captured by individual neurons. Moreover, modifying the activations of individual neurons allows controlling the translation output according to specified linguistic properties. The methods we develop here are task-independent and can be used for analyzing neural networks in other tasks. More broadly, our work contributes to the localist/distributed debate in artificial intelligence and cognitive science (Gayler & Levy, 2011) by investigating the important case of neural machine translation.

heading: 2 RELATED WORK
text: Much recent work has been concerned with analyzing neural representations of linguistic units, such as word embeddings (Köhn, 2015; Qian et al., 2016b), sentence embeddings (Adi et al., 2016; Ganesh et al., 2017; Brunner et al., 2018), and NMT representations at different linguistic levels: morphological (Belinkov et al., 2017a), syntactic (Shi et al., 2016b), and semantic (Hill et al., 2017; Belinkov et al., 2017b). These studies follow a common methodology of evaluating learned representations on external supervision by training classifiers or measuring other kinds of correlations (Belinkov & Glass, 2019). Thus they are limited to the available supervised annotation. In addition, these studies do not typically consider individual dimensions. In contrast, we propose intrinsic unsupervised methods for detecting important neurons based on correlations between independently trained models. A similar approach was used to analyze vision models (Li et al., 2016b), but to the best of our knowledge these ideas were not applied to NMT or other NLP models before.
In computer vision, individual neurons were shown to capture meaningful information (Zeiler & Fergus, 2014; Zhou et al., 2016; Bau et al., 2017). Even though some doubts were cast on the importance of individual units (Morcos et al., 2018), recent work stressed their contribution to predicting specific object classes via masking experiments similar to ours (Zhou et al., 2018). A few studies analyzed individual neurons in NLP. For instance, neural language models learn specific neurons that activate on brackets (Karpathy et al., 2015), sentiment (Radford et al., 2017), and length (Qian et al., 2016a). Length-specific neurons were also found in NMT (Shi et al., 2016a), but generally not much work has been devoted to analyzing individual neurons in NMT. We aim to address this gap.

heading: 3 METHODOLOGY
text: Much recent work on analyzing NMT relies on supervised learning, where NMT representations are used as features for predicting linguistic annotations (see Section 2). However, such annotations may not be available, or may constrain the analysis to a particular scheme.
Instead, we propose to use different kinds of correlations between neurons from different models as a measure of their importance. Suppose we have M such models and let hmt [i] denote the activation of the i-th neuron in the encoder of the m-th model for the t-th word.1 These may be models from different training epochs, trained with different random initializations or datasets, or even different architectures—all realistic scenarios that researchers often experiment with. Let xmi denote a random variable corresponding to the i-th neuron in the m-th model. xmi maps words to their neuron activations: xmi : t 7→ hmt [i]. Similarly, let xm denote a random vector corresponding to the activations of all neurons in the m-th model: xm : t 7→ hmt .
1 We only consider neurons from the top layer, although the approach can also be applied to other layers.
We consider four methods for ranking neurons, based on correlations between pairs of models. Our hypothesis is that different NMT models learn similar properties, and therefore similar important neurons emerge in different models, akin to neural vision models (Li et al., 2016b). Our methods capture different levels of localization/distributivity, as described next. See Figure 1 for illustration.

heading: 3.1 UNSUPERVISED CORRELATION METHODS
text: Maximum correlation The maximum correlation (MaxCorr) of neuron xmi looks for the highest correlation with any neuron in all other models:
MaxCorr(xmi ) = max j,m′ 6=m
|ρ(xmi , xm ′ j )| (1)
where ρ(x, y) is the Pearson correlation coefficient between x and y. We then rank the neurons in model m according to their MaxCorr score. We repeat this procedure for every model m. This score looks for neurons that capture properties that emerge strongly in two separate models.
Minimum correlation The minimum correlation (MinCorr) of neuron xmi looks for the neurons most correlated withXmi in each of the other models, but selects the one with the lowest correlation:
MinCorr(xmi ) = min m′ 6=m max j |ρ(xmi , xm
′
j )| (2)
Neurons in model m are ranked according to their MinCorr score. This tries to find neurons that are well correlated with many other models, even if they are not the overall most correlated ones.
Regression ranking We perform linear regression (LinReg) from the full representation of another model xm
′ to the neuron xmi . Then we rank neurons by the regression mean squared error.
This attempts to find neurons whose information might be distributed in other models.
SVCCA Singular vector canonical correlation analysis (SVCCA) is a recent method for analyzing neural networks (Raghu et al., 2017). In our implementation, we perform PCA on each model’s representations xm and take enough dimensions to account for 99% of the variance. For each pair of models, we obtain the canonically correlated basis, and rank the basis directions by their CCA coefficients. This attempts to capture information that may be distributed in less dimensions than the whole representation. In this case we get a ranking of directions, rather than individual neurons.

heading: 3.2 VERIFYING DETECTED NEURONS
text: We want to verify that neurons ranked highly by the unsupervised methods are indeed important for the NMT models. We consider quantitative and qualitative techniques for verifying their importance.
Erasing Neurons We test importance of neurons by erasing some of them during translation. Erasure is a useful technique for analyzing neural networks (Li et al., 2016a). Given a ranked list of neurons π, where π(i) is the rank of neuron xi, we zero-out increasingly more neurons according to the ranking π, starting from either the top or the bottom of the list. Our hypothesis is that erasing neurons from the top would hurt translation performance more than erasing from the bottom.
Concretely, we first run the entire encoder as usual, then zero out specific neurons from all source hidden states {h1, . . . ,hn} before running the decoder. For MaxCorr, MinCorr, and LinReg,
we zero out individual neurons. To erase k directions found by SVCCA, we instead project the embedding E (corresponding to all activations of a given model over a dataset) onto the space spanned by the non-erased directions: E′ = E(C(CTC)−1CT ), where C is the CCA projection matrix with the first or last k columns removed. This corresponds to erasing from the top or bottom.
Supervised Verification While our focus is on unsupervised methods for finding important neurons, we also utilize supervision to verify our results. Importantly, these experiments are done post-hoc, after having a candidate neuron to examine. Since training a supervised classifier on every neuron is costly, we instead report simple metrics that can be easily computed. Specifically, we sometimes report the expected conditional variance of neuron activations conditioned on some property. In other cases we found it useful to estimate a Gaussian mixture model (GMM) for predicting a label and measure its prediction quality. The number of mixtures in the GMM is set according to the number of classes in the predicted property (e.g. 2 mixtures when predicting tokens inside or outside of parentheses), and its parameters are estimated using the mean and variance of the neuron activation conditioned on each class. We obtain linguistic annotations with Spacy: spacy.io.
Visualization Interpretability of machine learning models remains elusive (Lipton, 2016), but visualizing can be an instructive technique. Similar to previous work analyzing neural networks in NLP (Elman, 1991; Karpathy et al., 2015; Kádár et al., 2016), we visualize activations of neurons and observe interpretable behavior. We will illustrate this with example heatmaps below.

heading: 4 EXPERIMENTAL SETUP
text: Data We use the United Nations (UN) parallel corpus (Ziemski et al., 2016) for all experiments. We train models from English to 5 languages: Arabic, Chinese, French, Russian, and Spanish, as well as an English-English auto-encoder. For each target language, we train 3 models on different parts of the training set, each with 500K sentences. In total, we have 18 models. This setting allows us to compare models trained on the same language pairs but different training data, as well as models trained on different language pairs. We evaluate on the official test set.2
MT training We train 500 dimensional 2-layer LSTM encoder-decoder models with attention (Bahdanau et al., 2014). In order to study both word and sub-word properties, we use a word representation based on a character convolutional neural network (charCNN) as input to both encoder and decoder, which was shown to learn morphology in language modeling and NMT (Kim et al., 2015; Belinkov et al., 2017a).3 While we focus here on recurrent NMT, our approach can be applied to other models like the Transformer (Vaswani et al., 2017), which we leave for future work.

heading: 5 RESULTS

heading: 5.1 ERASURE EXPERIMENTS
text: Figure 2 shows erasure results using the methods from Section 3.1, on an English-Spanish model. For all four methods, erasing from the top hurts performance much more than erasing from the
2Our experimental evaluation is focused on models trained on different parts of the training data, but we provide a short discussion of results on models from different epochs of a given training run in Appendix A.4.
3We used this representation rather than BPE sub-word units (Sennrich et al., 2016) to facilitate interpretability with respect to specific words. In the experiments, we report word-based results unless noted otherwise.
bottom. This confirms our hypothesis that neurons ranked higher by our methods have a larger impact on translation quality. Comparing erasure with different rankings, we find similar patterns with MaxCorr, MinCorr, and LinReg: erasing the top ranked 10% (50 neurons) degrades BLEU by 15-20 points, while erasing the bottom 10% neurons only hurts by 2-3 points. In contrast, erasing SVCCA directions results in rapid degradation: 15 BLEU point drop when erasing 1% (5) of the top directions, and poor performance when erasing 10% (50). This indicates that top SVCCA directions capture very important information in the model. We analyze these top neurons and directions in the next section, finding that top SVCCA directions focus mostly on identifying specific words.
Figure 3 shows the results of MaxCorr when erasing neurons from top and bottom, using models trained on three language pairs. In all cases, erasing from the top hurts performance more than erasing from the bottom. We found similar trends with other language pairs and ranking methods.

heading: 5.2 EVALUATING TOP NEURONS
text: What kind of information is captured by the neurons ranked highly by each of our ranking methods? Previous work found specific neurons in NMT that capture position of words in the sentence (Shi et al., 2016a). Do our methods capture similar properties? Indeed, we found that many top neurons capture position. For instance, Table 1 shows the top 10 ranked neurons from an English-Spanish model according to each of the methods. The table shows the percent of variance in neuron activation that is eliminated by conditioning on position in the sentence, calculated over the test set. Similarly, it shows the percent of explained variance by conditioning on the current token identity.
We observe an interesting difference between the ranking methods. LinReg and especially SVCCA, which are both computed by using multiple neurons, tend to find information determined by the identity of the current token. MinCorr and to a lesser extent MaxCorr tend to find position information. This suggests that information about the current token is often distributed in multiple neurons, which can be explained by the fact that tokens carry multiple kinds of linguistic information. In contrast, position is a fairly simple property that the NMT encoder can represent in a small number of neurons. That fact that many top MinCorr neurons capture position suggests that this kind of information is captured in multiple models in a similar way.

heading: 5.3 LINGUISTICALLY INTERPRETABLE NEURONS
text: Neurons that activate on specific tokens or capture position in the sentence are important in some of the methods, as shown in the previous section. But they are not highly ranked in all methods and are also less interesting from the perspective of capturing language information. In this section, we investigate several linguistic properties by measuring predictive capacity and visualizing neuron activations. The supplementary material discusses more properties. Further analysis of linguistically interpretable neurons is available in Dalvi et al. (2019a).
Parentheses Table 2 shows top neurons from each model for predicting that tokens are inside/outside of parentheses, quotes, or brackets, estimated by a GMM model. Often, the parentheses neuron is unique (low scores for the 2nd best neuron), suggesting that this property tends to be relatively localized. Generally, neurons that detect parentheses were ranked highly in most models by the MaxCorr method, indicating that they capture important patterns in multiple networks.
Figure 4 visualizes the most predictive neuron in an English-Spanish model. It activates positively (red) inside parentheses and negatively (blue) outside. Similar neurons were found in RNN language models (Karpathy et al., 2015). Next we consider more complicated linguistic properties.
Figure 4: Visualization of a parentheses neuron from an English-Spanish model.
Tense We annotated the test data for verb tense (with Spacy) and trained a GMM model to predict tense from neuron activations. Figure 5 shows activations of a top-scoring neuron (0.66 F1) from the English-Arabic model on the first 5 test sentences. It tends to activate positively (red color) on present tense (“recognizes”, “recalls”, “commemorate”) and negatively (blue color) on past tense (“published”, “disbursed”, “held”). These results are obtained with a charCNN representation, which is sensitive to common suffixes like “-ed”, “-es”. However, this neuron also detects irregular past tense verbs like “held”, suggesting that it captures context in addition to sub-word information. The neuron also makes some mistakes by activating weakly positively on nouns ending with “s” (“videos”, “punishments”), presumably because it gets confused with the 3rd person present tense. Similarly, it activates positively on “Spreads”, even though it functions as a noun in this context.
negative/positive for past/present. Examples shown are the first 5 sentences in the test set.
Table 3 shows correlations of neurons most correlated with this tense neuron, according to MaxCorr. All these neurons are highly predictive of tense: all but 3 are in the top 10 and 8 out of 15 (non-auto-encoder) neurons have the highest F1 score for predicting tense. The auto-encoder English models are an exception, exhibiting much lower correlations with the English-Arabic tense neuron. This suggests that tense emerges in a “real” NMT model, but not in an auto-encoder that only learns to copy. Interestingly, English-Chinese models have somewhat lower correlated neurons with the tense neuron, possibly due to the lack of explicit tense marking in Chinese. The encoder does not need to pay as much attention to tense when generating representations for the decoder.
Other Properties We found many more linguistic properties by visualizing top neurons ranked by our methods, especially with MaxCorr. We briefly mention some of these here and provide more details and quantitative results in the appendix. We found neurons that activate on numbers, dates, adjectives, plural nouns, auxiliary verbs, and more. We also investigated noun phrase segmentation, a compositional property above the word level, and found high-scoring neurons (60-80% accuracy) in every network. Many of these neurons were ranked highly by the MaxCorr method. In contrast, other methods did not rank such neurons very highly. See Table 5 in the appendix for the full results.
Some neurons have quite complicated behavior. For example, when visualizing neurons highly ranked by MaxCorr we found a neuron that activates on numbers in the beginning of a sentence, but not elsewhere (see Figure 9 in the appendix). It would be difficult to conceive of a supervised prediction task which would capture this behavior a-priori, without knowing what to look for. Our unsupervised methods are flexible enough to find any neurons deemed important by the NMT model, without constraining the analysis to properties for which we have supervised annotations.

heading: 6 CONTROLLING TRANSLATIONS
text: In this section, we explore a potential benefit of finding important neurons with linguistically meaningful properties: controlling the translation output. This may be important for mitigating biases in neural networks. For instance, gender stereotypes are often reflected in automatic translations, as the following motivating examples from Google Translate demonstrate.4
(1) a. o bir doctor b. he is a doctor
(2) a. o bir hemşire b. she is a nurse
The Turkish sentences (1a, 2a) have no gender information—they can refer to either male or female. But the MT system is biased to think that doctors are usually men and nurses are usually women, so its generated translations (1b, 2b) represent these biases. If we know the correct gender from another source such as metadata, we may want to encourage the system to output a translation with the correct gender. We make here a modest step towards this goal by intervening in neuron activations to induce a desired translation.5
4Retrieved on 9/16/2018. This particular example has recently been addressed in Google Translate by providing alternative gendered translation. For more biased examples, see mashable.com/2017/11/30/ google-translate-sexism.
5In concurrent work, Giulianelli et al. (2018) intervened in activations of a language model to improve its ability to capture verb number.
We conjecture that if a given neuron matters to the model, then we can control the translation in predictable ways by modifying its activations. To do this, we first encode the source sentence as usual. Before decoding, we set the activation of a particular neuron in the encoder state to a value α, which is a function of the mean activations over a particular property (defined below). To evaluate our ability to control the translation, we design the following protocol:
1. Tag the source and target sentences in the development set with a desired property, such as gender (masculine/feminine). We use Spacy for these tags.
2. Obtain word alignments for the development set using an alignment model trained on 2 million sentences of the UN data. We use fast align (Dyer et al., 2013) with default settings.
3. For every neuron in the encoder, predict the target property on the word aligned to its source word activations using a supervised GMM model.6
4. For every word having a desired property, modify the source activations of the top k neurons found in step 3 and generate a modified translation. The modification value is defined as α = µ1 + β(µ1 − µ2), where µ1 and µ2 are mean activations of the property we modify from and to, respectively (e.g. modifying gender from masculine to feminine), and β is a hyperparameter.
5. Tag the output translation and word-align it to the source. Declare success if the source word was aligned to a target word with the desired property value (e.g. feminine).

heading: 6.1 RESULTS
text: Figure 6 shows translation control results in an English-Spanish model. We report success rate—the percentage of cases where the word was aligned to a target word with the desired property—and the effect on BLEU scores, when varying α. Our tense control results are the most successful, with up to 67% success rate for changing past-to-present. Modifications generally degrade BLEU, but the loss at the best success rate is not large (2 BLEU points). Appendix A.2 provides more tense results.
Controlling other properties seems more difficult, with the best success rate for controlling number at 37%, using the top 5 number neurons. Gender is the most difficult to control, with a 21% success rate using the top 5 neurons. Modifying even more neurons did not help. We conjecture that these properties are more distributed than tense, which makes controlling them more difficult. Future work can explore more sophisticated methods for controlling multiple neurons simultaneously.

heading: 6.2 EXAMPLE TRANSLATIONS
text: We provide examples of controlling translation of number, gender, and tense. While these are cherrypicked examples, they illustrate that the controlling procedure can work in multiple properties and languages. We discuss language-specific patterns below.
6This is different from our results in the previous section, where we predicted a source-side property, because here we seek neurons that are predictive of target-side properties to facilitate controlling the translation.
Number Table 4a shows translation control results for a number neuron from an English-Spanish model, which activates negatively/positively on plural/singular nouns. The translation changes from plural to singular as we increase the modification α. We notice that using too high α values yields nonsense translations, but with correct number: transitioning from the plural adjective particulares (“particular”) to the singular adjective útil (“useful”). In between, we see a nice transition between plural and singular translations. Interestingly, the translations exhibit correct agreement between the modified noun and its adjectives and determines, e.g., Las partes interesadas vs. La parte interesada. This is probably due to the strong language model in the decoder.
Gender Table 4b shows examples of controlling gender translation for a gender neuron from the same model, which activates negatively/positively on masculine/feminine nouns. The translations of “parties” and “questions” change from masculine to feminine synonyms as we increase the modification α. Generally, we found it difficult to control gender, as also suggested by the relatively low success rate.
Tense Table 4c shows examples of controlling tense when translating from English to five target languages. In all language pairs, we are able to change the translation from past to present by modifying the activation of the tense neurons from the previous section (Table 3). Occasionally, modifying the activation on a single word leads to a change in phrasing; in Arabic the translation changes to “the efforts that the authorities invest”. In Spanish, we find a transition from past (apoyó) to imperfect (apoyaba) to present (apoya). Interestingly, in Chinese, we had to use a fairly large α value (in absolute terms), consistent with the fact that tense is not usually marked in Chinese. In fact, our modification generates a Chinese expression (正在) that is used to express an action in progress, similar to English “-ing”, resulting in the meaning “is supporting”.

heading: 7 CONCLUSION
text: Neural machine translation models learn vector representations that contain linguistic information while being trained solely on example translations. In this work, we developed unsupervised methods for finding important neurons in NMT, and evaluated how these neurons impact translation
quality. We analyzed several linguistic properties that are captured by individual neurons using quantitative prediction tasks and qualitative visualizations. We also designed a protocol for controlling translations by modifying neurons that capture desired properties.
Our analysis can be extended to other NMT components (e.g. the decoder) and architectures (Gehring et al., 2017; Vaswani et al., 2017), as well as other tasks. We believe that more work should be done to analyze the spectrum of localized vs. distributed information in neural language representations. We would also like to expand the translation control experiments to other architectures and components (e.g. the decoder), and to develop more sophisticated ways to control translation output, for example by modifying representations in variational NMT architectures (Zhang et al., 2016; Su et al., 2018). Our code is publicly available as part of the NeuroX toolkit (Dalvi et al., 2019b).7

heading: ACKNOWLEDGEMENTS
text: This research was carried out in collaboration between the HBKU Qatar Computing Research Institute (QCRI) and the MIT Computer Science and Artificial Intelligence Laboratory (CSAIL). Y.B. is also supported by the Harvard Mind, Brain, and Behavior Initiative (MBB).

heading: A ADDITIONAL RESULTS AND VISUALIZATIONS
text: A.1 NOUN PHRASE SEGMENTATION
Table 5 shows the top neurons from each network by accuracy when classifying interior, exterior, or beginning of a noun phrase. The annotations were obtained with Spacy. We found high-scoring neurons (60-80% accuracy) in every network. Many of these neurons were ranked highly by the MaxCorr ranking methods. In contrast, other correlation methods did not rank such neurons very highly. Thus there is correspondence between a high rank by our intrinsic unsupervised measure MaxCorr and the neuron’s capacity to predict external annotation.
A.2 CONTROLLING TRANSLATIONS
We provide additional translation control results. Table 6 shows the tense results using the best modification value from Figure 6a. We report the number of times the source word was aligned to a target word which is past or present, or to multiple words that include both or neither of these tenses. The success rate is the percentage of cases where the word was aligned to a target word with the desired tense. By modifying the activation of only one neuron (the most predictive one), we were able to change the translation from past to present in 67% of the times and vice-versa in 49% of the times. In many other cases, the tense was erased, that is, the modified source word was not aligned to any tensed word, which is a partial success.
A.3 VISUALIZATIONS
Here we provide additional visualizations of neurons capturing different linguistic properties.
Noun phrases We visualize the top scoring neuron (79%) from an English-Spanish model in Figure 7. Notice how the neuron activates positively (red color) on the first word in the noun phrases, but negatively (blue color) on the rest of the noun phrase (e.g. “Regional” in “Regional Service Centre”).
Dates and Numbers Figure 8 shows activations of neurons capturing dates and numbers. These neurons were ranked highly (top 30) by MaxCorr when ranking an English-Arabic model trained with charCNN representations. We note that access to character information leads to many neurons capturing sub-word information such as years (4-digit numbers). The first neuron is especially sensitive to month names (“May”, “April”). The second neuron is an approximate year-detector: it is sensitive to years (“2015”) as well as other tokens with four digits (“7439th”, “10.15”).
List items Figure 9 shows an interesting case of a neuron that is sensitive to the appearance of two properties simultaneously: position in the beginning of the sentence and number format. Notice that it activates strongly (negatively) on numbers when they open a sentence but not in the middle of the sentence. Conversely, it does not activate strongly on non-number words that open a sentence. This neuron aims to capture patterns of opening list items.
A.4 RESULTS ON MODELS FROM DIFFERENT EPOCHS
In the main experiments in the paper, we have used models trained on different parts of the training data, as well as on different language pairs. However, our methodology can be applied to any collection of models that we think should exhibit correlations in their neurons. We have verified that this approach works with model checkpoints from different epochs of the same training run. Concretely, we computed MaxCorr scores for the last checkpoint in two models—English-Spanish and English-Arabic—when comparing to other checkpoints. In both cases, we found highly correlated neurons across checkpoints, especially in the last few checkpoints. We also observed that erasing the top neurons hurt translation performance more than erasing the bottom neurons, similar to the findings in Section 5.1. Moreover, we noticed a significant overlap between the top ranked neurons in this case and the ones found by correlating with other models, as in the rest of the paper. In particular, for the English-Spanish model, we found that 8 out of 10 and 34 out of 50 top ranked neurons are the same in these two rankings. For the English-Arabic model, we found a similar behavior (7 out of 10 and 33 out of 50 top ranked neurons are the same). This indicates that our method may be applied to different checkpoints as well.

heading: B A CATALOG OF TOP RANKED NEURONS
text: In order to illustrate the range of linguistic phenomena captures by individual neurons, we provide here a list of the top 20 neurons (or projected directions, in the case of SVCCA) found by each of our methods, for an example English-Spanish model. For each neuron, we give the percentage of variance that is eliminated by conditioning on position in the sentence or identity of the current token. We also comment on what properties each neuron appears to capture, based on visualizations of neuron activation. Where possible, we give F1 scores of a GMM model predicting certain properties such as detecting noun phrase segmentation, parenthetical phrases, adjectives, and plural nouns. Annotations are obtained with Spacy: https://spacy.io.

Review summaries:
The authors propose a number of methods to identify individual important neurons in a machine translation system. The crucial assumption, drawn from the computer vision literature, is that important neurons are going to be correlated across related models (e.g. models that are trained on different subsets of the data). This hypothesis is validated to some extent: erasing the neurons that scored highly on these measures reduced BLEU score substantially. However, it turns out that most of the activation of the important neurons can be explained using sentence position. Supervised classification experiments on the important neurons revealed neurons that tracked properties such as the span of parentheses or word classes (e.g., auxiliary verbs, plural nouns, etc).