Identifying and Controlling Important Neurons in Neural Machine Translation

1 INTRODUCTION . Neural machine translation ( NMT ) systems achieve state-of-the-art results by learning from large amounts of example translations , typically without additional linguistic information . Recent studies have shown that representations learned by NMT models contain a non-trivial amount of linguistic information on multiple levels : morphological ( Belinkov et al. , 2017a ; Dalvi et al. , 2017 ) , syntactic ( Shi et al. , 2016b ) , and semantic ( Hill et al. , 2017 ) . These studies use trained NMT models to generate feature representations for words , and use these representations to predict certain linguistic properties . This approach has two main limitations . First , it targets the whole vector representation and fails to analyze individual dimensions in the vector space . In contrast , previous work found meaningful individual neurons in computer vision ( Zeiler & Fergus , 2014 ; Zhou et al. , 2016 ; Bau et al. , 2017 , among others ) and in a few NLP tasks ( Karpathy et al. , 2015 ; Radford et al. , 2017 ; Qian et al. , 2016a ) . Second , these methods require external supervision in the form of linguistic annotations . They are therefore limited by available annotated data and tools . In this work , we make initial progress towards addressing these limitations by developing unsupervised methods for analyzing the contribution of individual neurons to NMT models . We aim to answer the following questions : • How important are individual neurons for obtaining high-quality translations ? • Do individual neurons in NMT models contain interpretable linguistic information ? • Can we control MT output by intervening in the representation at the individual neuron level ? To answer these questions , we develop several unsupervised methods for ranking neurons according to their importance to an NMT model . Inspired by work in machine vision ( Li et al. , 2016b ) , we hypothesize that different NMT models learn similar properties , and therefore similar important neurons should emerge in different models . To test this hypothesis , we map neurons between pairs of trained NMT models using several methods : correlation analysis , regression analysis , and SVCCA , a recent method combining singular vectors and canonical correlation analysis ( Raghu et al. , 2017 ) . Our mappings yield lists of candidate neurons containing shared information across models . We ∗Equal contribution then evaluate whether these neurons carry important information to the NMT model by masking their activations during testing . We find that highly-shared neurons impact translation quality much more than unshared neurons , affirming our hypothesis that shared information matters . Given the list of important neurons , we then investigate what linguistic properties they capture , both qualitatively by visualizing neuron activations and quantitatively by performing supervised classification experiments . We were able to identify neurons corresponding to several linguistic phenomena , including morphological and syntactic properties . Finally , we test whether intervening in the representation at the individual neuron level can help control the translation . We demonstrate the ability to control NMT translations on three linguistic properties—tense , number , and gender—to varying degrees of success . This sets the ground for controlling NMT in desirable ways , potentially reducing system bias to properties like gender . Our work indicates that not all information is distributed in NMT models , and that many humaninterpretable grammatical and structural properties are captured by individual neurons . Moreover , modifying the activations of individual neurons allows controlling the translation output according to specified linguistic properties . The methods we develop here are task-independent and can be used for analyzing neural networks in other tasks . More broadly , our work contributes to the localist/distributed debate in artificial intelligence and cognitive science ( Gayler & Levy , 2011 ) by investigating the important case of neural machine translation . 2 RELATED WORK . Much recent work has been concerned with analyzing neural representations of linguistic units , such as word embeddings ( Köhn , 2015 ; Qian et al. , 2016b ) , sentence embeddings ( Adi et al. , 2016 ; Ganesh et al. , 2017 ; Brunner et al. , 2018 ) , and NMT representations at different linguistic levels : morphological ( Belinkov et al. , 2017a ) , syntactic ( Shi et al. , 2016b ) , and semantic ( Hill et al. , 2017 ; Belinkov et al. , 2017b ) . These studies follow a common methodology of evaluating learned representations on external supervision by training classifiers or measuring other kinds of correlations ( Belinkov & Glass , 2019 ) . Thus they are limited to the available supervised annotation . In addition , these studies do not typically consider individual dimensions . In contrast , we propose intrinsic unsupervised methods for detecting important neurons based on correlations between independently trained models . A similar approach was used to analyze vision models ( Li et al. , 2016b ) , but to the best of our knowledge these ideas were not applied to NMT or other NLP models before . In computer vision , individual neurons were shown to capture meaningful information ( Zeiler & Fergus , 2014 ; Zhou et al. , 2016 ; Bau et al. , 2017 ) . Even though some doubts were cast on the importance of individual units ( Morcos et al. , 2018 ) , recent work stressed their contribution to predicting specific object classes via masking experiments similar to ours ( Zhou et al. , 2018 ) . A few studies analyzed individual neurons in NLP . For instance , neural language models learn specific neurons that activate on brackets ( Karpathy et al. , 2015 ) , sentiment ( Radford et al. , 2017 ) , and length ( Qian et al. , 2016a ) . Length-specific neurons were also found in NMT ( Shi et al. , 2016a ) , but generally not much work has been devoted to analyzing individual neurons in NMT . We aim to address this gap . 3 METHODOLOGY . Much recent work on analyzing NMT relies on supervised learning , where NMT representations are used as features for predicting linguistic annotations ( see Section 2 ) . However , such annotations may not be available , or may constrain the analysis to a particular scheme . Instead , we propose to use different kinds of correlations between neurons from different models as a measure of their importance . Suppose we have M such models and let hmt [ i ] denote the activation of the i-th neuron in the encoder of the m-th model for the t-th word.1 These may be models from different training epochs , trained with different random initializations or datasets , or even different architectures—all realistic scenarios that researchers often experiment with . Let xmi denote a random variable corresponding to the i-th neuron in the m-th model . xmi maps words to their neuron activations : xmi : t 7→ hmt [ i ] . Similarly , let xm denote a random vector corresponding to the activations of all neurons in the m-th model : xm : t 7→ hmt . 1 We only consider neurons from the top layer , although the approach can also be applied to other layers . We consider four methods for ranking neurons , based on correlations between pairs of models . Our hypothesis is that different NMT models learn similar properties , and therefore similar important neurons emerge in different models , akin to neural vision models ( Li et al. , 2016b ) . Our methods capture different levels of localization/distributivity , as described next . See Figure 1 for illustration . 3.1 UNSUPERVISED CORRELATION METHODS . Maximum correlation The maximum correlation ( MaxCorr ) of neuron xmi looks for the highest correlation with any neuron in all other models : MaxCorr ( xmi ) = max j , m′ 6=m |ρ ( xmi , xm ′ j ) | ( 1 ) where ρ ( x , y ) is the Pearson correlation coefficient between x and y . We then rank the neurons in model m according to their MaxCorr score . We repeat this procedure for every model m. This score looks for neurons that capture properties that emerge strongly in two separate models . Minimum correlation The minimum correlation ( MinCorr ) of neuron xmi looks for the neurons most correlated withXmi in each of the other models , but selects the one with the lowest correlation : MinCorr ( xmi ) = min m′ 6=m max j |ρ ( xmi , xm ′ j ) | ( 2 ) Neurons in model m are ranked according to their MinCorr score . This tries to find neurons that are well correlated with many other models , even if they are not the overall most correlated ones . Regression ranking We perform linear regression ( LinReg ) from the full representation of another model xm ′ to the neuron xmi . Then we rank neurons by the regression mean squared error . This attempts to find neurons whose information might be distributed in other models . SVCCA Singular vector canonical correlation analysis ( SVCCA ) is a recent method for analyzing neural networks ( Raghu et al. , 2017 ) . In our implementation , we perform PCA on each model ’ s representations xm and take enough dimensions to account for 99 % of the variance . For each pair of models , we obtain the canonically correlated basis , and rank the basis directions by their CCA coefficients . This attempts to capture information that may be distributed in less dimensions than the whole representation . In this case we get a ranking of directions , rather than individual neurons . 3.2 VERIFYING DETECTED NEURONS . We want to verify that neurons ranked highly by the unsupervised methods are indeed important for the NMT models . We consider quantitative and qualitative techniques for verifying their importance . Erasing Neurons We test importance of neurons by erasing some of them during translation . Erasure is a useful technique for analyzing neural networks ( Li et al. , 2016a ) . Given a ranked list of neurons π , where π ( i ) is the rank of neuron xi , we zero-out increasingly more neurons according to the ranking π , starting from either the top or the bottom of the list . Our hypothesis is that erasing neurons from the top would hurt translation performance more than erasing from the bottom . Concretely , we first run the entire encoder as usual , then zero out specific neurons from all source hidden states { h1 , . . . , hn } before running the decoder . For MaxCorr , MinCorr , and LinReg , we zero out individual neurons . To erase k directions found by SVCCA , we instead project the embedding E ( corresponding to all activations of a given model over a dataset ) onto the space spanned by the non-erased directions : E′ = E ( C ( CTC ) −1CT ) , where C is the CCA projection matrix with the first or last k columns removed . This corresponds to erasing from the top or bottom . Supervised Verification While our focus is on unsupervised methods for finding important neurons , we also utilize supervision to verify our results . Importantly , these experiments are done post-hoc , after having a candidate neuron to examine . Since training a supervised classifier on every neuron is costly , we instead report simple metrics that can be easily computed . Specifically , we sometimes report the expected conditional variance of neuron activations conditioned on some property . In other cases we found it useful to estimate a Gaussian mixture model ( GMM ) for predicting a label and measure its prediction quality . The number of mixtures in the GMM is set according to the number of classes in the predicted property ( e.g . 2 mixtures when predicting tokens inside or outside of parentheses ) , and its parameters are estimated using the mean and variance of the neuron activation conditioned on each class . We obtain linguistic annotations with Spacy : spacy.io . Visualization Interpretability of machine learning models remains elusive ( Lipton , 2016 ) , but visualizing can be an instructive technique . Similar to previous work analyzing neural networks in NLP ( Elman , 1991 ; Karpathy et al. , 2015 ; Kádár et al. , 2016 ) , we visualize activations of neurons and observe interpretable behavior . We will illustrate this with example heatmaps below .

Summary: The authors propose a number of methods to identify individual important neurons in a machine translation system. The crucial assumption, drawn from the computer vision literature, is that important neurons are going to be correlated across related models (e.g. models that are trained on different subsets of the data). This hypothesis is validated to some extent: erasing the neurons that scored highly on these measures reduced BLEU score substantially. However, it turns out that most of the activation of the important neurons can be explained using sentence position. Supervised classification experiments on the important neurons revealed neurons that tracked properties such as the span of parentheses or word classes (e.g., auxiliary verbs, plural nouns, etc).