Venue: ACL 2017

# Visualizing and Understanding Neural Machine Translation

Yanzhuo Ding\({}^{\dagger}\) Yang Liu\({}^{\dagger\ddagger}\) Huanbo Luan\({}^{\dagger}\) Maosong Sun\({}^{\dagger\ddagger}\)

\({}^{\dagger}\)State Key Laboratory of Intelligent Technology and Systems

Tsinghua National Laboratory for Information Science and Technology

Department of Computer Science and Technology, Tsinghua University, Beijing, China

\({}^{\ddagger}\)Jiangsu Collaborative Innovation Center for Language Competence, Jiangsu, China

djx133@yeah.net, liuyang2011@tsinghua.edu.cn

luanhuanbo@gmail.com, sms@tsinghua.edu.cn

Corresponding author.

###### Abstract

While neural machine translation (NMT) has made remarkable progress in recent years, it is hard to interpret its internal workings due to the continuous representations and non-linearity of neural networks. In this work, we propose to use layer-wise relevance propagation (LRP) to compute the contribution of each contextual word to arbitrary hidden states in the attention-based encoder-decoder framework. We show that visualization with LRP helps to interpret the internal workings of NMT and analyze translation errors.

## 1 Introduction

End-to-end neural machine translation (NMT), which leverages neural networks to directly map between natural languages, has gained increasing popularity recently Sutskever et al. (2014); Bahdanau et al. (2015). NMT proves to outperform conventional statistical machine translation (SMT) significantly across a variety of language pairs Junczys-Dowmunt et al. (2016) and becomes the new _de facto_ method in practical MT systems Wu et al. (2016).

However, there still remains a severe challenge: it is hard to interpret the internal workings of NMT. In SMT Koehn et al. (2003); Chiang (2005), the translation process can be denoted as a derivation that comprises a sequence of translation rules (e.g., phrase pairs and synchronous CFG rules). Defined on language structures with varying granularities, these translation rules are interpretable from a linguistic perspective. In contrast, NMT takes an end-to-end approach: all internal information is represented as real-valued vectors or matrices. It is challenging to associate hidden states in neural networks with interpretable language structures. As a result, the lack of interpretability makes it very difficult to understand translation process and debug NMT systems.

Therefore, it is important to develop new methods for visualizing and understanding NMT. Existing work on visualizing and interpreting neural models has been extensively investigated in computer vision Krizhevsky et al. (2012); Mahendran and Vedaldi (2015); Szegedy et al. (2014); Simonyan et al. (2014); Nguyen et al. (2015); Girshick et al. (2014); Bach et al. (2015). Although visualizing and interpreting neural models for natural language processing has started to attract attention recently Karpathy et al. (2016); Li et al. (2016), to the best of our knowledge, there is no existing work on visualizing NMT models. Note that the attention mechanism Bahdanau et al. (2015) is restricted to demonstrate the connection between words in source and target languages and unable to offer more insights in interpreting how target words are generated (see Section 4.5).

In this work, we propose to use layer-wise relevance propagation (LRP) Bach et al. (2015) to visualize and interpret neural machine translation. Originally designed to compute the contributions of single pixels to predictions for image classifiers, LRP back-propagates relevance recursively from the output layer to the input layer. In contrast to visualization methods relying on derivatives, a major advantage of LRP is that it does not require neural activations to be differentiable or smooth Bach et al. (2015). We adapt LRP to the attention-based encoder-decoder framework Bahdanau et al. (2015) to calculate relevance that measures the association degree between two arbitrary neurons in neural networks. Case studies on Chinese-English translation show that visualization helps to interpret the internal workings of

NMT and analyze translation errors.

## 2 Background

Given a source sentence \({\bf x}=x_{1},\ldots,x_{i},\ldots,x_{I}\) with \(I\) source words and a target sentence \({\bf y}=y_{1},\ldots,y_{j},\ldots,y_{J}\) with \(J\) target words, neural machine translation (NMT) decomposes the sentence-level translation probability as a product of word-level translation probabilities:

\[P({\bf y}|{\bf x};{\boldsymbol{\theta}})=\prod_{j=1}^{J}P(y_{j}|{\bf x},{\bf y }_{<j};{\boldsymbol{\theta}}), \tag{1}\]

where \({\bf y}_{<j}=y_{1},\ldots,y_{j-1}\) is a partial translation.

In this work, we focus on the attention-based encoder-decoder framework Bahdanau et al. (2015). As shown in Figure 1, given a source sentence \({\bf x}\), the encoder first uses source word embeddings to map each source word \(x_{i}\) to a real-valued vector \(\mathbf{x}_{i}\).1

Footnote 1: Note that we use \({\bf x}\) to denote a source sentence and \(\mathbf{x}\) to denote the vector representation of a single source word.

Then, a forward recurrent neural network (RNN) with GRU units Cho et al. (2014) runs to calculate source forward hidden states:

\[\overrightarrow{{\bf h}}_{i}=f(\overrightarrow{{\bf h}}_{i-1},\mathbf{x}_{i}), \tag{2}\]

where \(f(\cdot)\) is a non-linear function.

Similarly, the source backward hidden states can be obtained using a backward RNN:

\[\overleftarrow{{\bf h}}_{i}=f(\overleftarrow{{\bf h}}_{i+1},\mathbf{x}_{i}). \tag{3}\]

To capture global contexts, the forward and backward hidden states are concatenated as the hidden state for each source word:

\[{\bf h}_{i}=[\overrightarrow{{\bf h}}_{i};\overleftarrow{{\bf h}}_{i}]. \tag{4}\]

Bahdanau et al. Bahdanau et al. (2015) propose an attention mechanism to dynamically determine the relevant source context \({\bf c}_{j}\) for each target word:

\[{\bf c}_{j}=\sum_{i=1}^{I+1}\alpha_{j,i}{\bf h}_{i}, \tag{5}\]

where \(\alpha_{j,i}\) is an attention weight that indicates how well the source word \(x_{i}\) and the target word \(y_{j}\) match. Note that an end-of-sentence token is appended to the source sentence.

In the decoder, a target hidden state for the \(j\)-th target word is calculated as

\[{\bf s}_{j}=g({\bf s}_{j-1},\mathbf{y}_{j},{\bf c}_{j}), \tag{6}\]

where \(g(\cdot)\) is a non-linear function, \(\mathbf{y}_{j-1}\) denotes the vector representation of the \((j-1)\)-th target word.

Finally, the word-level translation probability is given by

\[P(y_{j}|{\bf x},{\bf y}_{<j};{\boldsymbol{\theta}})=\rho(\mathbf{y}_{j-1},{\bf s} _{j},{\bf c}_{j}), \tag{7}\]

where \(\rho(\cdot)\) is a non-linear function.

Although NMT proves to deliver state-of-the-art translation performance with the capability to handle long-distance dependencies due to GRU and attention, it is hard to interpret the internal information such as \(\overrightarrow{{\bf h}}_{i}\), \(\overleftarrow{{\bf h}}_{i}\), \({\bf h}_{i}\), \({\bf c}_{j}\), and \({\bf s}_{j}\) in the encoder-decoder framework. Though projecting word embedding space into two dimensions Faruqui and Dyer (2014) and the attention matrix Bahdanau et al. (2015) shed partial light on how NMT works, how to interpret the entire network still remains a challenge.

Therefore, it is important to develop new methods for understanding the translation process and analyzing translation errors for NMT.

## 3 Approach

### Problem Statement

Recent efforts on interpreting and visualizing neural models has focused on calculating the contribution of a unit at the input layer to the final decision at the output layer Simonyan et al. (2014); Mahendran and Vedaldi (2015); Nguyen et al. (2015);

Figure 1: The attention-based encoder-decoder architecture for neural machine translation Bahdanau et al. (2015).

 

Girshick et al., 2014; Bach et al., 2015; Li et al., 2016). For example, in image classification, it is important to understand the contribution of a single pixel to the prediction of classifier (Bach et al., 2015).

In this work, we are interested in calculating the contribution of source and target words to the following internal information in the attention-based encoder-decoder framework:

1. \(\overrightarrow{\mathbf{h}}_{i}\): the \(i\)-th source forward hidden state,
2. \(\overleftarrow{\mathbf{h}}_{i}\): the \(i\)-th source backward hidden state,
3. \(\mathbf{h}_{i}\): the \(i\)-th source hidden state,
4. \(\mathbf{c}_{j}\): the \(j\)-th source context vector,
5. \(\mathbf{s}_{j}\): the \(j\)-th target hidden state,
6. \(\mathbf{y}_{j}\): the \(j\)-th target word embedding.

For example, as shown in Figure 2, the generation of the third target word "York" depends on both the source context (i.e., the source sentence "zai niuyue \(<\)/\(s>\)") and the target context (i.e., the partial translation "in New"). Intuitively, the source word "niuyue" and the target word "New" are more relevant to "York" and should receive higher relevance than other words. The problem is how to quantify and visualize the relevance between hidden states and contextual word vectors.

More formally, we introduce a number of definitions to facilitate the presentation.

**Definition 1**: _The_ **contextual word set** _of a hidden state \(\mathbf{v}\in\mathbb{R}^{M\times 1}\) is denoted as \(\mathcal{C}(\mathbf{v})\), which is a set of source and target contextual word vectors \(\mathbf{u}\in\mathbb{R}^{N\times 1}\) that influences the generation of \(\mathbf{v}\)._

For example, the context word set for \(\overrightarrow{\mathbf{h}}_{i}\) is \(\{\mathbf{x}_{1},\dots,\mathbf{x}_{i}\}\), for \(\overleftarrow{\mathbf{h}}_{i}\) is \(\{\mathbf{x}_{i},\dots,\mathbf{x}_{I+1}\}\), and for \(\mathbf{h}_{i}\) is \(\{\mathbf{x}_{1},\dots,\mathbf{x}_{I+1}\}\). The contextual word set for \(\mathbf{c}_{j}\) is \(\{\mathbf{x}_{1},\dots,\mathbf{x}_{I+1}\}\), for \(\mathbf{s}_{j}\) and \(\mathbf{y}_{j}\) is \(\{\mathbf{x}_{1},\dots,\mathbf{x}_{I+1},\mathbf{y}_{1},\dots,\mathbf{y}_{J-1}\}\).

As both hidden states and contextual words are represented as real-valued vectors, we need to factorize vector-level relevance at the neuron level.

**Definition 2**: _The_ **neuron-level relevance** _between the \(m\)-th neuron in a hidden state \(v_{m}\in\mathbb{R}\) and the \(n\)-th neuron in a contextual word vector \(u_{n}\in\mathbb{R}\) is denoted as \(r_{u_{n}\gets v_{m}}\in\mathbb{R}\), which satisfies the following constraint:_

\[v_{m}=\sum_{\mathbf{u}\in\mathcal{C}(\mathbf{v})}\sum_{n=1}^{N} r_{u_{n}\gets v_{m}} \tag{8}\]

**Definition 3**: _The_ **vector-level relevance** _between a hidden state \(\mathbf{v}\) and one contextual word vector \(\mathbf{u}\in\mathcal{C}(\mathbf{v})\) is denoted as \(R_{\mathbf{u}\leftarrow\mathbf{v}}\in\mathbb{R}\), which quantifies the contribution of \(\mathbf{u}\) to the generation of \(\mathbf{v}\). It is calculated as_

\[R_{\mathbf{u}\leftarrow\mathbf{v}}=\sum_{m=1}^{M}\sum_{n=1}^{ N}r_{u_{n}\gets v_{m}} \tag{9}\]

**Definition 4**: _The_ **relevance vector** _of a hidden state \(\mathbf{v}\) is a sequence of vector-level relevance of its contextual words:_

\[R_{\mathbf{v}}=\{R_{\mathbf{u}_{1}\leftarrow\mathbf{v}},\dots, R_{\mathbf{u}_{|\mathcal{C}(\mathbf{v})|}\leftarrow\mathbf{v}}\} \tag{10}\]

Therefore, our goal is to compute relevance vectors for hidden states in a neural network, as shown in Figure 2. The key problem is how to compute neuron-level relevance.

### Layer-wise Relevance Propagation

We follow (Bach et al., 2015) to use layer-wise relevance propagation (LRP) to compute neuron-level relevance. We use a simple feed-forward network shown in Figure 3 to illustrate the central idea of LRP.

Figure 3: A simple feed-forward network for illustrating layer-wise relevance propagation (Bach et al., 2015).

Figure 2: Visualizing the relevance between the vector representation of a target word “New York” and those of all source words and preceding target words.

 

LRP first propagates the relevance from the output layer to the intermediate layer:

\[r_{z_{1}\gets v_{1}} =\frac{\mathbf{W}^{(2)}_{1,1}z_{1}}{\mathbf{W}^{(2)}_{1,1}z_{1}+ \mathbf{W}^{(2)}_{2,1}z_{2}}v_{1} \tag{11}\] \[r_{z_{2}\gets v_{1}} =\frac{\mathbf{W}^{(2)}_{2,1}z_{2}}{\mathbf{W}^{(2)}_{1,1}z_{1}+ \mathbf{W}^{(2)}_{2,1}z_{2}}v_{1} \tag{12}\]

Note that we ignore the non-linear activation function because Bach et al. (2015) indicate that LRP is _invariant_ against the choice of non-linear function.

Then, the relevance is further propagated to the input layer:

\[r_{u_{1}\gets v_{1}} = \frac{\mathbf{W}^{(1)}_{1,1}u_{1}}{\mathbf{W}^{(1)}_{1,1}u_{1}+ \mathbf{W}^{(1)}_{2,1}u_{2}}r_{z_{1}\leftarrow v_{1}}+ \tag{13}\] \[\frac{\mathbf{W}^{(1)}_{1,2}u_{1}}{\mathbf{W}^{(1)}_{1,2}u_{1}+ \mathbf{W}^{(1)}_{2,2}u_{2}}r_{z_{2}\leftarrow v_{1}}\] \[r_{u_{2}\gets v_{1}} = \frac{\mathbf{W}^{(1)}_{2,1}u_{2}}{\mathbf{W}^{(1)}_{1,1}u_{1}+ \mathbf{W}^{(1)}_{2,1}u_{2}}r_{z_{1}\leftarrow v_{1}}+\] (14) \[\frac{\mathbf{W}^{(1)}_{2,2}u_{2}}{\mathbf{W}^{(1)}_{1,2}u_{1}+ \mathbf{W}^{(1)}_{2,2}u_{2}}r_{z_{2}\leftarrow v_{1}}\]

Note that \(r_{u_{1}\gets v_{1}}+r_{u_{2}\gets v_{1}}=v_{1}\).

More formally, we introduce the following definitions to ease exposition.

**Definition 5**: _Given a neuron \(u\), its_ **incoming neuron set**__\(\mathrm{IN}(u)\) comprises all its direct connected preceding neurons in the network._

For example, in Figure 3, the incoming neuron set of \(z_{1}\) is \(\mathrm{IN}(z_{1})=\{u_{1},u_{2}\}\).

**Definition 6**: _Given a neuron \(u\), its_ **outcoming neuron set**__\(\mathrm{OUT}(u)\) comprises all its direct connected descendant neurons in the network._

For example, in Figure 3, the incoming neuron set of \(z_{1}\) is \(\mathrm{OUT}(z_{1})=\{v_{1},v_{2}\}\).

**Definition 7**: _Given a neuron \(v\) and its incoming neurons \(u\in\mathrm{IN}(v)\), the_ **weight ratio** _that measures the contribution of \(u\) to \(v\) is calculated as_

\[w_{u\to v}=\frac{\mathbf{W}_{u,v}u}{\sum_{u^{\prime}\in\mathrm{IN}(v)} \overline{\mathbf{W}}_{u^{\prime},v}u^{\prime}} \tag{15}\]

Although the NMT model usually involves multiple operators such as matrix multiplication, element-wise multiplication, and maximization, they only influence the way to calculate weight ratios in Eq. (15).

For matrix multiplication such as \(\mathbf{v}=\mathbf{Wu}\), its basic form that is calculated at the neuron level is given by \(v=\sum_{u\in\mathrm{IN}(v)}\mathbf{W}_{u,v}u\) . We follow Bach et al. (2015) to calculate the weight ratio using Eq. (15) 

For element-wise multiplication such as \({\bf v}={\bf u_{1}}\circ{\bf u_{2}}\), its basic form is given by \(v=\prod_{u\in\mathrm{IN}(v)}u\). We use the following method to calculate its weight ratio:

\[w_{u\to v}=\frac{u}{\sum_{u^{\prime}\in\mathrm{IN}(v)}u^{\prime}} \tag{16}\]

For maximization such as \(v=\max\{u_{1},u_{2}\}\), we calculate its weight ratio as follows:

\[w_{u\to v}=\left\{\begin{array}{ll}1&\text{if $u=\max_{u^{\prime}\in \mathrm{IN}(v)}\{u^{\prime}\}$}\\ 0&\text{otherwise}\end{array}\right. \tag{17}\]

Therefore, the general local redistribution rule for LRP is given by

\[r_{u\gets v}=\sum_{z\in\mathrm{OUT}(u)}w_{u\to z}r_{z\leftarrow v} \tag{18}\]

Algorithm 1 gives the layer-wise relevance propagation algorithm for neural machine translation. The input is an attention-based encoder-decoder neural network for a sentence pair after decoding \(G\) and a set of hidden states to be visualized \(\mathcal{V}\). The output is a set of vector-level relevance between intended hidden states and their contextual words \(\mathcal{R}\). The algorithm first computes weight ratios for each neuron in a forward pass (lines 1-4). Then, for each hidden state to be visualized (line 6), the algorithm initializes the neuron-level relevance for itself (lines 7-9). After initialization, the neuron-level relevance is backpropagated through the network (lines 10-12). Finally, vector-level relevance is calculated based on neuron-level relevance (lines 13-16). The time complexity of Algorithm 1 is \(\mathcal{O}(|G|\times|\mathcal{V}|\times O_{\max})\), where \(|G|\) is the number of neuron units in the neural network \(G\), \(|\mathcal{V}|\) is the number of hidden states to be visualized and \(O_{\max}\) is the maximum of outdegree for neurons in the network. Calculating relevance is more computationally expensive than computing attention as it involves all neurons in the network. Fortunately, it is possible to take advantage of parallel architectures of GPUs and relevance caching for speed-up.

## 4 Analysis

### Data Preparation

We evaluate our approach on Chinese-English translation. The training set consists of 1.25M pairs of sentences with 27.93M Chinese words and 34.51M English words. We use the NIST 2003 dataset as the development set for model selection and the NIST 2004 dataset as test set. The BLEU score on NIST 2003 is 32.73.

We use the open-source toolkit GroundHog Bahdanau et al. (2015), which implements the attention-based encoder-decoder framework. After model training and selection on the training and development sets, we use the resulting NMT model to translate the test set. Therefore, the visualization examples in the following subsections are taken from the test set.

### Visualization of Hidden States

#### 4.2.1 Source Side

Figure 4 visualizes the source hidden states for a source content word "nian" (_years_). For each word in the source string "jin liang nian lai , meiguo" (_in recent two years, USA_), we attach a number

Figure 4: Visualizing source hidden states for a source content word “nian” (_years_).

Figure 5: Visualizing target hidden states for a target content word “visit”.

 

to denote the position of the word in the sentence. For example, "nian" (_years_) is the third word.

We are interested in visualizing the relevance between the third source forward hidden state and all its contextual words "jin" (_recent_) and "liang" (_two_). We observe that the direct preceding word "liang" (_two_) contributes more to forming the forward hidden state of "nian" (_years_). For the third source backward hidden state , the relevance of contextual words generally decreases with the increase of the distance to "nian" (_years_). Clearly, the concatenation of forward and backward hidden states capture contexts in both directions.

The situations for function words and punctuation marks are similar but the relevance is usually more concentrated on the word itself. We omit the visualization due to space limit.

#### 4.2.2 Target Side

Figure 5 visualizes the target-side hidden states for the second target word "visit". For comparison, we also give the attention weights , which correctly identifies the second source word "canbai" ("visit") is most relevant to "visit".

The relevance vector of the source context is generally consistent with the attention but reveals that the third word "shi" (_is_) also contributes to the generation of "visit".

For the target hidden state , the contextual word set includes the first target word "my". We find that most contextual words receive high values of relevance. This phenomenon has been frequently observed for most target words in other sentences. Note that relevance vector is not normalized. This is an essential difference between attention and relevance. While attention is defined to be normalized, the only constraint on relevance is that the sum of relevance of contextual words is identical to the value of intended hidden state neuron.

For the target word embedding , the relevance is generally consistent with the attention by identifying that the second source word contributes more to the generation of "visit". But further indicates that the target word "my" is also very important for generating "visit".

Figure 6 shows the hidden states of a target UNK word, which is very common to see in NMT because of limited vocabulary. It is interesting to investigate whether the attention mechanism could put a UNK in the right place in the translation. In this example, the 6-th source word "haiwiguo" is a UNK. We find that the model successfully predicts the correct position of UNK by exploiting surrounding source and target contexts. But the ordering of UNK usually becomes worse if multiple UNK words exist on the source side.

### Translation Error Analysis

Given the visualization of hidden states, it is possible to offer useful information for analyzing translation errors commonly observed in NMT such as word omission, word repetition, unrelated words and negation reversion.

#### 4.3.1 Word Omission

Given a source sentence "bajisitan zongtong muxialafu yingde can zhong liang yuan xinren topiao" (_pakistani president unsharraf wins votes of confidence in senate and house_), the NMT model pro

Figure 7: Analyzing translation error: word omission. The 6-th source word “zhong” is untranslated incorrectly.

 

duces a wrong translation "pakistani president win over democratic vote of confidence in the senate". One translation error is that the 6-th source word "zhong" (_house_) is incorrectly omitted for translation.

As the end-of-sentence token "\(<\)/s\(>\)" occurs early than expected, we choose to visualize its corresponding target hidden states. Although the attention correctly identifies the 6-th source word "zhong" (_house_) to be important for generating the next target word, the relevance of source context \(R_{\mathbf{e}12}\) attaches more importance to the end-of-sentence token.

Finally, the relevance of target word \(R_{\mathbf{y}_{12}}\) reveals that the end-of-sentence token and the 11-th target word "senate" become dominant in the softmax layer for generating the target word.

This example demonstrates that only using attention matrices does not suffice to analyze the internal workings of NMT. The values of relevance of contextual words might vary significantly across different layers.

#### 4.3.2 Word Repetition

Given a source sentence "meiguoren lishi hang you jiang chengxi de chuantong , you fancuo recluco de chuantong" (_in history_ , _the people of america have the tradition of honesty and would not hesitate to admit their mistakes_), the NMT model produces a wrong translation "in the history of the history of the history of the history of the americans , there is a tradition of faith in the history of mistakes". The translation error is that "history" repeats four times in the translation.

Figure 8 visualizes the target hidden states of the 6-th target word "history". According to the relevance of the target word embedding \(R_{\mathbf{y}_{6}}\), the first source word "meiguoren" (_american_), the second source word "lishi" (_history_) and the 5-th target word "the" are most relevant to the generation of "history". Therefore, word repetition not only results from wrong attention but also is significantly influenced by target side context. This finding confirms the importance of controlling source and target contexts to improve fluency and adequacy Tu et al. (2017).

#### 4.3.3 Unrelated Words

Given a source sentence "ci ci huiyi de yi ge zhongya yiti shi kuadaxiyang guaarsi" (_one the top agendas of the meeting is to discuss the cross-atlantic relations_), the model prediction is "a key topic of the meeting is to forge ahead". One translation error is that the 9-th English word "forge" is totally unrelated to the source sentence.

Figure 9 visualizes the hidden states of the 9-th target word "forge". We find that while the attention identifies the 10-th source word "kuadaxiyang" (_cross-atlantic_) to be most relevant, the relevance vector of the target word \(R_{\mathbf{y}_{9}}\) finds that multiple source and target words should contribute to the generation of the next target word.

We observe that unrelated words are more likely to occur if multiple contextual words have high

Figure 8: Analyzing translation error: word repetition. The target word “history” occurs twice in the translation incorrectly.

Figure 9: Analyzing translation error: unrelated words. The 9-th target word “forge” is totally unrelated to the source sentence.

 

values in the relevance vector of the target word being generated.

#### 4.3.4 Negation Reversion

Given a source sentence "bu jiejue shengcun wenti , juu tan bu Shang fa zhan , geng tan bu Shang ke chixu fazhan" (_without solution to the issue of subsistence , there will be no development to speak of , let alone sustainable development_), the model prediction is "if we do not solve the problem of living , we will talk about development and still less can we talk about sustainable development". The translation error is that the 8-th negation source word "bu" (_not_) is untranslated. The omission of negation is a severe translation error it reverses the meaning of the source sentence.

As shown in Figure 10, while both attention and relevance correctly identify the 8-th negation word "bu" (_not_) to be most relevant, the model still generates "about" instead of a negation target word. One possible reason is that target context words "will talk" take the lead in determining the next target word.

### Extra Words

Given a source sentence "bajisitan zongtong muxialafu yingde can zhong liang yuan xinren toujao"(_pakistani president musharraf wins votes of confidence in senate and house_), the model prediction is "pakistani president win over democratic vote of confidence in the senate" The translation error is that the 5-th target word "democratic" is extra generated.

Figure 11 visualizes the hidden states of the 9-th target word "forge". We find that while the attention identifies the 9-th source word "xinen"(_confidence_) to be most relevant, the relevance vector of the target word \(R_{y\!o}\) indicates that the end-of-sentence token and target words contribute more to the generation of "democratic".

### Summary of Findings

We summarize the findings of visualizing and analyzing the decoding process of NMT as follows:

1. Although attention is very useful for understanding the connection between source and target words, only using attention is not sufficient for deep interpretation of target word generation (Figure 9);
2. The relevance of contextual words might vary significantly across different layers of hidden states (Figure 9);
3. Target-side context also plays a critical role in determining the next target word being generated. It is important to control both source and target contexts to produce correct translations (Figure 10);
4. Generating the end-of-sentence token too early might lead to many problems such as word omission, unrelated word generation, and truncated translation (Figures 7 and 9).

Figure 11: Analyzing translation error: extra word. The 5-th target word “democratic” is an extra word.

Figure 10: Analyzing translation error: negation. The 8-th negation source word “bu” (_not_) is not translated.

 



## 5 Related Work

Our work is closely related to previous visualization approaches that compute the contribution of a unit at the input layer to the final decision at the output layer Simonyan et al. (2014); Mahendran and Vedaldi (2015); Nguyen et al. (2015); Girshick et al. (2014); Bach et al. (2015); Li et al. (2016). Among them, our approach bears most resemblance to Bach et al. (2015) since we adapt layer-wise relevance propagation to neural machine translation. The major difference is that word vectors rather than single pixels are the basic units in NMT. Therefore, we propose vector-level relevance based on neuron-level relevance for NMT. Calculating weight ratios has also been carefully designed for the operators in NMT.

The proposed approach also differs from Li et al. (2016) in that we use relevance rather than partial derivative to quantify the contributions of contextual words. A major advantage of using relevance is that it does not require neural activations to be differentiable or smooth Bach et al. (2015).

The relevance vector we used is significantly different from the attention matrix Bahdanau et al. (2015). While attention only demonstrates the association degree between source and target words, relevance can be used to calculate the association degree between two arbitrary neurons in neural networks. In addition, relevance is effective in analyzing the effect of source and target contexts on generating target words.

## 6 Conclusion

In this work, we propose to use layer-wise relevance propagation to visualize and interpret neural machine translation. Our approach is capable of calculating the relevance between arbitrary hidden states and contextual words by back-propagating relevance along the network recursively. Analyses of the state-of-art attention-based encoder-decoder framework on Chinese-English translation show that our approach is able to offer more insights than the attention mechanism for interpreting neutral machine translation.

In the future, we plan to apply our approach to more NMT approaches Sutskever et al. (2014); Shen et al. (2016); Tu et al. (2016); Wu et al. (2016) on more language pairs to further verify its effectiveness. It is also interesting to develop relevance-based neural translation models to explicitly control relevance to produce better translations.

## Acknowledgements

This work is supported by the National Natural Science Foundation of China (No.61522204), the 863 Program (2015AA015407), and the National Natural Science Foundation of China (No.61432013). This research is also supported by the Singapore National Research Foundation under its International Research Centre@Singapore Funding Initiative and administered by the IDM Programme.

## References

* Bach et al. (2015) Sebastian Bach, Alexander Binder, Gregoire Montavon, Frederick Klauschen, Klaus-Robert Muller, and Wojciech Samek. 2015. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. _PLoS ONE_ .
* Bahdanau et al. (2015) Dzmitry Bahdanau, KyungHyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In _Proceedings of ICLR_.
* Chiang (2005) Davie Chiang. 2005. A hierarchical phrase-based model for statistical machine translation. In _Proceedings of ACL_.
* Cho et al. (2014) Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using rnn encoder-decoder for statistical machine translation. In _Proceedings of EMNLP_.
* Faruqui and Dyer (2014) Mannal Faruqui and Chris Dyer. 2014. Improving vector space word representations using multilingual correlation. In _Proceedings of EACL_.
* Girshick et al. (2014) Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. 2014. Rich feature hierarchies for accurate object detection and semantic segmentation. In _Proceedings of CVPR_.
* Junczys-Dowmunt et al. (2016) Marcin Junczys-Dowmunt, Tomasz Drojak, and Hieu Hoang. 2016. Is neural machine translation ready for deployment? a case study on 30 translation directions. arXiv:1610.01108v2.
* Karpathy et al. (2016) Andrej Karpathy, Justin Johnson, and Fei-Fei Li. 2016. Visualizing and understanding recurrent networks. In _Proceedings of ICLR Workshop_.
* Koehn et al. (2003) Philipp Koehn, Franz J. Och, and Daniel Marcu. 2003. Statistical phrase-based translation. In _Proceedings of NAACL_.
* Krizhevsky et al. (2012) Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. 2012. Imagenet classification with deep convolutional nerual networks. In _Proceedings of NIPS_

Jiwei Li, Xinlei Chen, Eduard Hovy, and Dan Jurafsky. 2016. Visualizing and understanding neural models in nlp. In _Proceedings of NAACL_.
* Mahendran and Vedaldi (2015) Aravindh Mahendran and Andrea Vedaldi. 2015. Understanding deep image representations by inverting them. In _Proceedings of CVPR_.
* Nguyen et al. (2015) Anh Nguyen, Jason Yosinski, and Jeff Clune. 2015. Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. In _Proceedings of CVPR_.
* Shen et al. (2016) Shiqi Shen, Yong Cheng, Zhongjun He, Wei He, Hua Wu, Maosong Sun, and Yang Liu. 2016. Minimum risk training for neural machine translation. In _Proceedings of ACL_.
* Simonyan et al. (2014) Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. 2014. Deep inside convolutional networks: Visualizing image classification models and saliency maps. In _Proceedings of ICLR Workshop_.
* Sutskever et al. (2014) Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In _Proceedings of NIPS_.
* Szegedy et al. (2014) Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. 2014. Intriguing properties of neural networks. In _Proceedings of ICLR_.
* Tu et al. (2017)Zhaopeng Tu, Yang Liu, Lifeng Shang, Xiaohua Liu, and Hang Li. 2017. Context gates for neural machine translation. _Transactions of the ACL_ .
* Tu et al. (2016) Zhaopeng Tu, Zhengdong Lu, Yang Liu, Xiaohua Liu, and Hang Li. 2016. Modeling coverage for neural machine translation. In _Proceedings of ACL_.
* Wu et al. (2016) Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2016. Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv:1609.08144v2.