Venue: NAACL 2018

# Self-Attentive Residual Decoder for Neural Machine Translation

Lesly Miculicich Werlen\({}^{*,\dagger}\), Nikolaos Pappas\({}^{*}\), Dhananjay Ram\({}^{*,\dagger}\), Andrei Popescu-Belis\({}^{\ddagger}\)

\({}^{*}\)Idiap Research Institute, Switzerland, Ecole polytechnique federale de Lausanne (EPFL), Switzerland, HEIG-VD / HES-SO, Switzerland

{lmiculicich, npappas, dram}@idiap.ch

andrei.popescu-belis@heig-vd.ch

###### Abstract

Neural sequence-to-sequence networks with attention have achieved remarkable performance for machine translation. One of the reasons for their effectiveness is their ability to capture relevant source-side contextual information at each time-step prediction through an attention mechanism. However, the target-side context is solely based on the sequence model which, in practice, is prone to a recency bias and lacks the ability to capture effectively nonsequential dependencies among words. To address this limitation, we propose a target-side-attentive residual recurrent network for decoding, where attention over previous words contributes directly to the prediction of the next word. The residual learning facilitates the flow of information from the distant past and is able to emphasize any of the previously translated words, hence it gains access to a wider context. The proposed model outperforms a neural MT baseline as well as a memory and self-attention network on three language pairs. The analysis of the attention learned by the decoder confirms that it emphasizes a wider context, and that it captures syntactic-like structures.

## 1 Introduction

Neural machine translation (NMT) has recently become the state-of-the-art approach to machine translation Bojar et al. (2016). Several architectures have been proposed for this task Kalchbrenner and Blunsom (2013); Sutskever et al. (2014); Cho et al. (2014); Gehring et al. (2017); Vaswani et al. (2017), but the attention-based NMT model designed by Bahdanau et al. (2015) is still considered the de-facto baseline. This architecture is composed of two recurrent neural networks (RNNs), an encoder and a decoder, and an attention mechanism between them for modeling a soft word-alignment. First, the model encodes the complete source sentence, and then decodes one word at a time. The decoder has access to all the context on the source side through the attention mechanism. However, on the target side, the contextual information is represented only through a fixed-length vector, namely the hidden state of the decoder. As observed by Bahdanau et al. (2015), this creates a bottleneck which hinders the ability of the sequential model to learn longer-term information effectively.

As pointed out by Cheng et al. (2016), sequential models present two main problems for natural language processing. First, the memory of the encoder is shared across multiple words and is prone to bias towards the recent past. Second, such models do not fully capture the structural composition of language. To address these limitations, several recent models have been proposed, namely memory networks Cheng et al. (2016); Tran et al. (2016); Wang et al. (2016) and self-attention networks Daniluk et al. (2016); Liu and Lapata (2018). We experimented with these methods, applying them to NMT: _memory RNN_Cheng et al. (2016) and _self-attentive RNN_Daniluk et al. (2016). How

Figure 1: Comparison between the decoder of the baseline NMT and the proposed decoder with self-attentive residual connections.



ever, we observed no significant gains in performance over the baseline architecture.

In this paper, we propose a self-attentive residual recurrent decoder, presented in Figure 1b, which, if unfolded over time, represents a densely-connected residual network. The self-attentive residual connections focus selectively on previously translated words and propagate useful information to the output of the decoder, within an attention-based NMT architecture. The attention paid to the previously predicted words is analogous to a read-only memory operation, and enables the learning of syntactic-like structures which are useful for the translation task.

Our evaluation on three language pairs shows that the proposed model improves over several baselines, with only a small increase in computational overhead. In contrast, other similar approaches have lower scores but a higher computational overhead. The contributions of this paper can be summarized as follows:

* We propose and compare several options for using self-attentive residual learning within a standard decoder, which facilitates the flow of contextual information on the target side.
* We demonstrate consistent improvements over a standard baseline, and two advanced variants, which make use of memory and self-attention on three language pairs (English-to-Chinese, Spanish-to-English, and English-to-German).
* We perform an ablation study and analyze the learned attention function, providing additional insights on its actual contributions.

## 2 Related Work

Several studies have been proposed to enhance sequential models by capturing longer contexts. Long short-term memory (LSTM) Hochreiter and Schmidhuber (1997) is the most commonly used recurrent neural network (RNN), because its internal memory allows to retain information from a more distant past than a vanilla RNN. Several studies attempt to increase the memory capacity of LSTMs by using memory networks Weston et al. (2015); Sukhbaatar et al. (2015). For instance, Cheng et al. (2016) incorporate different memory cells for each previous output representation, which are later accessed by an attention mechanism Tran et al. (2016) include a memory block to access recent input words in a selective manner. Both methods show improvements on language modeling. For NMT, Wang et al. (2016) presented a decoder enhanced with an external shared memory. Memory networks extend the capacity of the network and have the potential to read, write, and forget information. Our method, which attends over previously predicted words, can be seen as a read-only memory, which is simpler but computationally more efficient because it does not require additional memory space.

Other studies aim to improve the modeling of source-side contextual information, for example through a context-aware encoder using self-attention Zhang et al. (2017), or a recurrent attention NMT Yang et al. (2017) that is aware of previously attended words on the source-side in order to better predict which words will be attended in future. Additionally, variational NMT Zhang et al. (2016) introduces a latent variable to model the underlying semantics of source sentences. In contrast to these studies, we focus instead on the contextual information _on the target side_.

The application of self-attention mechanisms to RNNs have been previously studied, and in general, they seem to capture syntactic dependencies among distant words Liu and Lapata (2018); Soltani and Jiang (2016); Lee et al. (2017); Lin et al. (2017). Daniluk et al. (2016) explore different approaches to self-attention for language modeling, leading to improvements over a baseline LSTM and over memory-augmented methods. However, the methods do not fully utilize a longer context. The main difference with our approach is that we apply attention on the output embeddings rather than the hidden states. Thus, the connections are independent of the recurrent layer representations, which is beneficial to NMT, as we show below.

Our model relies on residual connections, which have been shown to improve the learning process of deep neural networks by addressing the vanishing gradient problem He et al. (2016). These connections create a direct path from previous layers, helping the transmission of information. Recently, several architectures using residual connections with LSTMs have been proposed for sequence prediction Zhang et al. (2016); Kim et al. (2017); Zilly et al. (2017); Wang and Tian (2016). To our knowledge, our study is the first one to use self-attentive residual connections within residual RNNs for NMT. In parallel to our study, a similar method was recently proposed for sentiment analysis Wang (2017) 

Background: Neural Machine Translation

Neural machine translation aims to compute the conditional distribution of emitting a sentence in a target language given a sentence in a source language, denoted by \(p_{\Theta}(y|x)\), where \(\Theta\) is the set of parameters of the neural model, and \(y=\{y_{1},...,y_{n}\}\) and \(x=\{x_{1},...,x_{m}\}\) are respectively the representations of source and target sentences as sequences of words. The parameters \(\Theta\) are learned by training a sequence-to-sequence neural model on a corpus of parallel sentences. In particular, the learning objective is to maximize the following conditional log-likelihood:

\[\max_{\Theta}\frac{1}{N}\sum_{n=1}^{N}\log(p_{\Theta}(y|x)) \tag{1}\]

The models typically use gated recurrent units (GRUs) (Cho et al., 2014) or LSTMs (Hochreiter and Schmidhuber, 1997). Their architecture has three main components: an encoder, a decoder, and an attention mechanism.

The goal of the encoder is to build meaningful representations of the source sentences. It consists of a bidirectional RNN which includes contextual information from past and future words into the vector representation \(h_{i}\) of a particular word vector \(x_{i}\), formally defined as follows:

\[h_{i}=[\overrightarrow{h_{i}},\overleftarrow{h_{i}}] \tag{2}\]

Here, \(\overrightarrow{h_{i}}=f(x_{i},h_{i-1})\) and \(\overleftarrow{h_{i}}=f(x_{i},h_{i+1})\) are the hidden states of the forward and backward passes of the bidirectional RNN respectively, and \(f\) is a non-linear function.

The decoder (see Figure 0(a)) is in essence a recurrent language model. At each time step, it predicts a target word \(y_{t}\) conditioned over the previous words and the information from the encoder using the following posterior probability:

\[p(y_{t}|y_{1},...,y_{t-1},c_{t})\approx g(s_{t},y_{t-1},c_{t}) \tag{3}\]

where \(g\) is a non-linear multilayer function. The hidden state of the decoder \(s_{t}\) is defined as:

\[s_{t}=f(s_{t-1},y_{t-1},c_{t}) \tag{4}\]

and depends on a _context vector_\(c_{t}\) that is computed by the attention mechanism.

The attention mechanism allows the decoder to select which parts of the source sentence are more useful to predict the next output word. This goal is achieved by considering a weighted sum over all hidden states of the encoder as follows:

\[c_{t}=\sum_{i=1}^{m}\alpha_{i}^{t}h_{i} \tag{5}\]

where \(\alpha_{i}^{t}\) is a weight calculated using a normalized exponential function \(a\), also known as _alignment function_, which computes how good is the match between the input at position \(i\in\{1,...,n\}\) and the output at position \(t\):

\[\alpha_{i}^{t}=softmax(e_{i}^{t}) \tag{6}\] \[e_{i}^{t}=a(s_{t-1},h_{i}) \tag{7}\]

Different types of alignment functions have been used for NMT, as investigated by Luong et al. (2015). Here, we use the one originally defined by Bahdanau et al. (2015).

## 4 Self-Attentive Residual Decoder

The decoder of the attention-based NMT model uses a skip connection from the previously predicted word to the output classifier in order to enhance the performance of translation. As we can see in Eq. (3), the probability of a particular word is calculated by a function \(g\) which takes as input the hidden state of the recurrent layer \(s_{t}\), the representation of the previously predicted word \(y_{t-1}\), and the context vector \(c_{t}\). Within \(g\), these quantities are typically summed up after going through simple linear transformations, hence the addition of \(y_{t-1}\) is indeed a skip connection as in residual networks (He et al., 2016). In theory, \(s_{t}\) should be sufficient for predicting the next word given that it is dependent on the other two local-context components according to Eq. (4). However, the \(y_{t-1}\) quantity makes the model emphasize the last predicted word for generating the next word. How can we make the model consider a broader context?

To answer this question, we propose to include into the decoder's formula skip connections not only from the previous time step \(y_{t-1}\), but from all previous time steps from \(y_{0}\) to \(y_{t-1}\). This defines a residual recurrent network which, unfolded over time, can be seen as a densely connected residual network. These connections are applied to all previously predicted words, and reinforce the memory of the recurrent layer towards what has been translated so far. At each time step, the model 

decides which of the previously predicted words should be emphasized to predict the next one. In order to deal with the dynamic length of this new input, we use a target-side summary vector \(d_{t}\) that can be interpreted as the representation of the decoded sentence until the time \(t\) in the word embedding space. We therefore modify Eq. (3) replacing \(y_{t-1}\) with \(d_{t}\):

\[p(y_{t}|y_{1},...,y_{t-1},c_{t})\approx g(s_{t},d_{t},c_{t}) \tag{8}\]

The replacement of \(y_{t-1}\) with \(d_{t}\) means that the number of parameters added to the model is dependent only on the calculation of \(d_{t}\). Figure 0(b) illustrates the change made to the decoder. We define two methods for summarizing the context into \(d_{t}\), which are described in the following sections.

### Mean Residual Connections

One simple way to aggregate information from multiple word embeddings is by averaging them. This average can be seen as the sentence representation until time \(t\). We hypothesize that this representation is more informative than using only the embedding of the previous word. Formally:

\[d^{avg}_{t}=\frac{1}{t-1}\sum_{i=1}^{t-1}y_{i} \tag{9}\]

### Self-Attentive Residual Connections

Averaging is a simple and cheap way to aggregate information from multiple words, but may not be sufficient for all kinds of dependencies. Instead, we propose a dynamic way to aggregate information in each sentence, such that different words have different importance according to their relation with the prediction of the next word. We propose to use a shared self-attention mechanism to obtain a summary representation of the translation, i.e. a _weighted average representation_ of the words translated from \(y_{0}\) to \(y_{t-1}\). This mechanism aims to model, in part, important non-sequential dependencies among words, and serves as a complementary memory to the recurrent layer.

\[d^{cavg}_{t}=\sum_{i=1}^{t-1}\alpha^{t}_{i}y_{i} \tag{10}\] \[\alpha^{t}_{i}=softmax(e^{t}_{i}) \tag{11}\]

The weights of the attention model are computed by a scoring function \(e^{t}_{i}\) that predicts how important each previous word (\(y_{0},....\), or \(y_{t-1}\)) is for the current prediction \(y_{t}\).

We experiment with two different scoring functions, as follows:

\[e^{t}_{i}=v^{\intercal}tanh(W_{y}y_{i}+W_{s}s_{t})\text{~{} (content+scope)} \tag{12}\]

\[\mathrm{or}\;\;e^{t}_{i}=v^{\intercal}tanh(W_{y}y_{i})\text{ ~{} (content)} \tag{13}\]

where \(v\in\mathbb{R}^{e}\), \(W_{y}\in\mathbb{R}^{e\times e}\), and \(W_{s}\in\mathbb{R}^{e\times d}\) are weight matrices, \(e\) and \(d\) are the dimensions of the embeddings and hidden states respectively. Firstly, we study the scoring function noted _content+scope_, as proposed by Bahdanau et al. (2015) for NMT. Secondly, we explore a scoring function noted as _content_, which is calculated based only on the previous hidden states of the decoder, as proposed by Pappas and Popescu-Belis (2017). In contrast to the first attention function, which makes use of the hidden vector \(s_{t}\), the second one is based only on the previous word representations, therefore, it is independent of the current prediction representation. However, the normalization of this function still depends on \(t\).

## 5 Other Self-Attentive Networks

To compare our approach with similar studies, we adapted two representative self-attentive networks for application to NMT.

### Memory RNN

The _Memory RNN_ decoder is based on the proposal by Cheng et al. (2016) to modify an LSTM layer to include a memory with different cells for each previous output representation. Thus at each time step, the hidden layer can select past information dynamically from the memory. To adapt it to our framework, we modify Eq. (4) as:

\[s_{t}=f(\tilde{s}_{t},y_{t-1},c_{t}) \tag{14}\]

\[\text{where}\qquad\tilde{s}_{t}=\sum_{i=1}^{t-1}\alpha^{t}_{i}s_{i} \tag{15}\]

\[\alpha^{t}_{i}=softmax(e^{t}_{i}) \tag{16}\]

\[e^{t}_{i}=a(h_{i},y_{t-1},\tilde{s}_{t-1}) \tag{17}\]

### Self-Attentive RNN

The _Self-Attentive RNN_ is the simplest one proposed by Daniluk et al. (2016), and incorporates a summary vector from past predictions calculated with an attention mechanism. Here, the attention is applied over previous hidden states. This decoder is formulated as follows:

\[p(y_{t}|y_{1},...,y_{t-1},c_{t})\approx g(s_{t},y_{t-1},c_{t},\tilde{s}_{t})\] (18) 

where \[\tilde{s}_{t}=\sum_{i=1}^{t-1}\alpha_{i}^{t}s_{i}\] (19) \[\alpha_{i}^{t}=softmax(e_{i}^{t})\] (20) \[e_{i}^{t}=a(s_{i},s_{t})\] (21)

Additional details of the formulations in Sections 3, 4, and 5 are described in the Appendix A.

## 6 Experimental Settings

### Datasets

To evaluate the proposed MT models in different conditions, we select three language pairs with increasing amounts of training data: English-Chinese (0.5M sentence pairs), Spanish-English (2.1M), and English-German (4.5M).

For English-to-Chinese, we use a subset of the UN parallel corpus Rafalovitch and Dale (2009)1, with 0.5M sentence pairs for training, 2K for development, and 2K for testing. For training Spanish-to-English MT, we use a subset of WMT 2013 Bojar et al. (2013), corresponding to Europarl v7 and News Commentary v11 with ca. 2.1M sentence pairs. Newstest2012 and Newstest2013 were used for development and testing respectively. Finally, we use the complete English-to-German set from WMT 2016 Bojar et al. (2016) with a total of ca. 4.5M sentence pairs. The development set is Newstest2013, and the testing set is Newstest2014. Additionally, we include as testing sets Newstest2015 and Newstst2016, for comparison with the state of the art. We report translation quality using (a) BLEU over _tokenized_ and _truecased_ texts, and (b) NIST BLEU over _detokenized_ and _detruecased_ texts2.

Footnote 1: [http://www.uncorpora.org/](http://www.uncorpora.org/)

Footnote 2: Scrips from Moses toolkit Koehn et al. (2007): BLEU multi-bleu, NIST BLEU uneval-v13apl, tokenizerpert, ruecase.pet.

### Model Configuration

We use the implementation of the attention-based NMT baseline provided in d14mt-tutorial3 developed in Python using Theano Theano Development Team (2016). The system implements an attention-based NMT model, described above, using one layer of GRUs Cho et al. (2014). The vocabulary size is 25K for English-to-Chinese NMT, and 50K for Spanish-to-English and English-German. We use the byte pair encoding (BPE) strategy for out-of-vocabulary words Sennrich et al. (2016). For all cases, the maximum sentence length of the training samples is 50, the dimension of the word embeddings is 500, and the dimension of the hidden layers is 1,024. We use dropout with a probability of 0.5 after each layer. The parameters of the models are initialized randomly from a standard normal distribution scaled to a factor of 0.01. The loss function is optimized using Adadelta Zeiler (2012) with \(\epsilon=10^{-6}\) and \(\rho=0.95\) as in the original paper. The systems were trained in 7-12 days for each model on a Tesla K40 GPU at the speed of about 1,000 words/sec.

Footnote 3: [https://github.com/nyu-dl/d14mt-tutorial](https://github.com/nyu-dl/d14mt-tutorial)

## 7 Analysis of the Results

Table 1 shows the BLEU scores and the number of parameters used by the different NMT models. Along with the NMT baseline, we included a statistical machine translation (SMT) model based on Moses Koehn et al. (2007) with the same training/tuning/test data as the NMT. The performance of _memory RNN_ is similar to the baseline and, as confirmed later, its focus of attention is mainly on the prediction at \(t-1\). The _self-attentive RNN_ method is inferior to the baseline, which can be attributed to the overhead on the hidden vectors that have to learn the recurrent representations and the attention simultaneously. The proposed models outperform the baseline, and the best scores are obtained by the NMT model with _self-attentive residual connections_. Despite their simplicity, the _mean residual connections_ already improve the translation, without increasing the number of parameters.

Tables 2 and 3 show further experiments with the proposed methods on various English-German test sets, compared to several previous systems.

\begin{table}
\begin{tabular}{p{85.4pt} r r r}  & \multicolumn{2}{c}{**|\(|\mathbf{\Theta}|\)**} & \multicolumn{2}{c}{**BLEU**} \\ \hline
**Models** & \multicolumn{3}{c}{**En-Zh Es–En**} \\ \hline SMT baseline & – & 21.6 & 25.2 \\ NMT baseline & 108.7M & 22.6 & 25.4 \\ + Memory RNN & 109.7M & 22.5 & 25.5 \\ + Self-attentive RNN & 110.2M & 22.0 & 25.1 \\ + Mean residual connections & 108.7M & 23.6 & 25.7 \\ + Self-attentive residual connections & 108.9M & **24.0** & **26.3** \\ \hline \end{tabular}
\end{table}
Table 1: BLEU score (multi-bleu) on _tokenized_ text. The highest score per dataset is marked in bold. The self-attentive residual connections make use of the _content_ attention function. \(|\Theta|\) indicates the number of parameters per model.

 

Table 2 shows BLEU values calculated by _multi-bleu_, and includes the NMT system proposed by Luong et al. (2015) which replaces unknown predicted words with the most strongly aligned word on the source sentence. Also, the table includes other systems described in Section 2. Additionally, Table 3 shows values calculated by the NIST BLEU scorer, as well as results reported by the "Winning WMT" systems for each test set respectively: UEDIN-SYNTAX Williams et al. (2014), UEDIN-SYNTAX Williams et al. (2015), and UEDIN-NMT Sennrich et al. (2016a). Also, we include the results reported by Sennrich et al. (2016b) for a baseline encoder-decoder NMT with BPE for unknown words similar to our configuration, and finally the system proposed by Nadejde et al. (2017), an explicit syntax-aware NMT that introduces combinatory categorial grammar (CCG) supertags on the target side by predicting words and tags alternately. The comparison with this work is relevant for the analysis described later in Section 8.2. The results confirm that the _self-attentive residual connections_ improve significantly the translations. To evaluate the significance of the improvements against the NMT baseline, we performed a one-tailed paired \(t\)-test.

### Impact of the Attention Function

We now examine the two scoring functions that can be used for the _self-attentive residual connections_ model presented in Eq. (12), considering English-to-Chinese and Spanish-to-English. The BLEU scores are presented in Table 4: the best option is the _content_ matching function, which depends only on the word embeddings. The _content+scope_ function, which depends additionally on the hidden representation of the current prediction is better than the baseline but scores lower than _content_. The idea that the importance of the context depends on the current prediction is appealing, because it can be interpreted as learning internal dependencies among words. However, the experimental results show that it does not necessarily lead to the best translation. On the contrary, the _content_ attention function may be extracting representations of the whole sentence which are easier to learn and generalize.

### Performance According to Human Evaluation

Manual evaluation on samples of 50 sentences for each language pair helped to corroborate the conclusions obtained from the BLEU scores, and to provide a qualitative understanding of the improvements brought by our model. For each language, we employed one evaluator who was a native speaker of the target language and had good knowledge of the source language. The evaluators ranked three translations of the same source sentence - one from each of our models: _baseline_, _mean residual connections_, and _self-attentive residual connections_ - according to their translation quality. The three translations were presented in a random order, so that the system that had generated them could not be identified. To integrate

\begin{table}
\begin{tabular}{l c c}  & **BLEU** \\ \hline
**Attention function** & **En-Zh** & **Es-En** \\ \hline _Content+Scope_ & 23.1 & 25.6 \\ _Content_ & **24.0** & **26.3** \\ \end{tabular}
\end{table}
Table 4: BLEU scores for two scoring variants of the attention function of the proposed decoder.

\begin{table}
\begin{tabular}{l r r}  & \multicolumn{2}{c}{**BLEU**} \\ \hline
**Models** & **NT14** & **NT15** \\ \hline NMT (unk. word repl.) Luong et al. (2015) & 20.9 – \\ Context-aware NMT Zhang et al. (2017) & 22.57 – \\ Recurrent attention NMT Yang et al. (2017) & 22.1 & 25.0 \\ Variational NMT Zhang et al. (2016a) & – & 25.49 \\ NMT baseline & 22.3 – & 24.8 \\ + Memory RNN & 22.6 & 24.9 \\ + Self-attentive RNN & 22.0 & 24.3 \\ + Mean residual connections & 22.9 & 24.9 \\ + Self-attentive residual connections & **23.2** & **25.5** \\ \end{tabular}
\end{table}
Table 2: BLEU score (multi-bleu) on _tokenized_ text for English-to-German on _Newstest (NT) 2014, and 2015_. The highest score per dataset is marked in bold. The self-attentive residual connections makes use of the _content_ attention function.

\begin{table}
\begin{tabular}{l l l l}  & \multicolumn{2}{c}{**BLEU** (**NIST**)} \\ \hline
* & **NT14*
* & **NT15*
* & **NT16*
* \\ \hline Winning WMT & 20.1 & 24.4 & **34.2*
* \\ NMT (BPE) Sennrich et al. (2016b) & – & 22.8 & – \\ Syntax NMT Nadejde et al. (2017) & – & – & 29.0 \\ \hline NMT Baseline & 21.0 & 24.4 & 28.8 \\ + Mean Residual connections
* & 21.4 & 24.7 & 29.6 \\ + Self-attentive residual connections*
* & **21.7*
* & **25.0*
* & 29.7 \\ \hline \end{tabular}
\end{table}
Table 3: NIST BLEU scores on _detokenized_ and _detracessed_ text for English-to-German on _Newstest (NT) 2014, 2015, 2016_. Significance test:
* \(p<0.05\), *
* \(p<0.01\). The Winning WMT systems are listed in the text below.
 

the judgments, we proceed in pairs, and count the number of times each system was ranked higher, equal to, or lower than another competing system. The results shown in Table 5 indicate that the _self-attentive residual connections_ model outperforms the one with _mean residual connections_, and both outperform the baseline, for all three language pairs. The rankings are thus identical to those obtained using BLEU in Tables 1 and 3.

### Performance on Language Modeling

To examine whether language modeling (LM) can benefit from the proposed method, we incorporate the residual connections into a neural LM. We use the same setting as Daniluk et al. (2016) for a corpus of Wikipedia articles (22.5M words), and we compare with two methods proposed in the same paper, namely attention LSTM and 4-gram LSTM. As shown in Table 6, the proposed models outperform the LSTM baseline as well as the self-attention model, but not the 4-gram LSTM. Experiments using 4-gram LSTM for NMT showed poor performance (13.9 BLEU points for EnglishChinese) which can be attributed to the difference between the LM and NMT tasks. Both tasks predict one word at a time conditioned over previous words, however, in NMT the previous targetword-inputs are not given, they have to be generated by the decoder. Thus, the output could be conditioned over previous erroneous predictions affecting in higher proportion the 4-gram LSTM model. This shows that even if a model improves language modeling, it does not necessarily improve machine translation.

## 8 Qualitative Analysis

### Distribution of Attention

Figure 2 shows a comparison of the distribution of attention of the different self-attentive models described in this paper, on Spanish-to-English NMT (the other two language pairs exhibit similar distributions). The values correspond to the number of words which received maximal attention for each relative position (\(x\)-axis). We selected, at each prediction, the preceding word with maximal weight, and counted its relative position. We normalized the count by the number of previous words at the time of each prediction.

We observe that the _memory RNN_ almost always selects the immediately previous word (\(t-1\)) and ignores the rest of the context. On the contrary, the other two models distribute attention more evenly among all previous words. In particular, the _self-atenttive RNN_ uses a longer context than the _self-attentive residual connections_ but, as the performance on BLEU score shows, this fact does not necessarily mean better translation.

Figure 3 shows the attention to previous words generated by each model for one sentence translated from Spanish to English. The matrices present the target-side attention weights, with the vertical axis indicating the previous words, and the color shades at each position (cell) representing the attention weights. The weights of the _memory RNN_ are concentrated on the diagonal, indicating that the att

\begin{table}
\begin{tabular}{l c r}
**Systems** & \multicolumn{1}{c}{\(d\)} & **Perplexity** \\ \hline LSTM (Daniluk et al., 2016) & 300 & 85.2 \\ LSTM + Attention (Daniluk et al., 2016) & 296 & 82.0 \\ LSTM + 4-gram (Daniluk et al., 2016) & 968 & 75.9 \\ LSTM + Mean residual connections & 296 & 80.2 \\ LSTM + Self-attentive residual connections & 296 & 80.4 \\ \end{tabular}
\end{table}
Table 6: Evaluation of the proposed methods on language modeling. The number of parameter for all models is 47M.

\begin{table}
\begin{tabular}{l l l l|l l l l} \hline  & \multicolumn{6}{c}{**Ranking (\%)**} \\ \hline
**System** & \multicolumn{3}{c}{**En–Zh**} & \multicolumn{3}{c}{**Es-En**} & \multicolumn{3}{c}{**En-De**} \\ \cline{3-8}  & \(>=<\) & \(>=~{}<\) & \(>=~{}<\) & \(>=~{}<\) \\ \hline Mean vs. Baseline & 26 & 56 & 18 & 20 & 64 & 16 & 28 & 58 & 24 \\ Self-attentive vs. Baseline & 28 & 60 & 12 & 28 & 56 & 16 & 32 & 54 & 14 \\ Self-attentive vs. Mean & 24 & 62 & 14 & 28 & 58 & 14 & 32 & 56 & 12 \\ \hline \end{tabular}
\end{table}
Table 5: Human evaluation of sentence-level translation quality on three language pairs. We compare the models in pairs, indicating the percentages of sentences that were ranked higher (\(>\)), equal to (\(=\)), or lower (\(<\)) for the first system with respect to the second one. The values correspond to percentages (%).

Figure 2: Percentage of words that received maximum attention at a given relative position, ranging from \(-1\) to \(-50\) (maximum length).

 

the previous word, which makes the model almost equivalent to the baseline. The weights of the _self-attentive RNN_ show that attention is more distributed towards the distant past, and they vary for each word because the attention function depends on the current prediction. This model tries to find dependencies among words, although complex relations seem difficult to learn. On the contrary, the proposed _self-attentive residual connections_ model strongly focuses on particular words, and we present a wider analysis of it in the following section.

### Structures Learned by the Model

When visualizing the matrix of attention weights generated by our model (Figure 2(c)), we observed the formation of sub-phrases which are grouped depending on their attention to previous words. To build the sub-phrases in a deterministic fashion, we implemented Algorithm 1, which iteratively splits the sentence into two sub-phrases every time the focus of attention changes to a new word, from left-to-right. The results are binary tree structures containing the sub-phrases, exemplified in Figure 4.

We formally evaluate the syntactic properties of the binary tree structures by comparing them with the results of an automatic constituent parser Manning et al. (2014), using the ParsEval approach Black et al. (1991), i.e. by counting the precision and recall of constituents, excluding single words. Our models reaches a precision of \(0.56\), which is better than the precision of \(0.45\) obtained by a trivial right-branched tree model4. Note that these structures were neither optimized for parsing nor learned using part-of-speech tagging as most parsers do. Our interpretation of the results is that they are "syntactic-like" structures. However, given the simplicity of the model, they could

Figure 3: Matrix of distribution of the attention weights to previous words. The vertical axis represents the previous words. A darker shade indicates a higher attention weight.

 

also be viewed as more limited structures, similar to sentence chunks.

### Translation Examples

Table 7 shows examples of translations produced with the baseline and the _self-attentive residual connections_ model. The first part shows examples for which the proposed model reached a higher BLEU score than the baseline. Here, the structure of the sentences, or at least the word order, are improved. The second part contains examples where the baseline achieved better BLEU score than our model. In the first example, the structure of the sentence is different but the content and quality are similar, while in the second one lexical choices differ from the reference.

## 9 Conclusion

We presented a novel decoder which uses self-attentive residual connections to previously translated words in order to enrich the target-side contextual information in NMT. To cope with the variable lengths of previous predictions, we proposed two methods for context summarization: _mean residual connections_ and _self-attentive residual connections_. Additionally, we showed how similar previous proposals, designed for language modeling, can be adapted to NMT. We evaluated the methods over three language pairs: Chinese-to-English, Spanish-to-English, and English-to-German. In each case, we improved the BLEU score compared to the NMT baseline and two variants with memory-augmented decoders. A manual evaluation over a small set of sentences for each language pair confirmed the improvement. Finally, a qualitative analysis showed that the proposed model distributes weights throughout an entire sentence, and learns structures resembling syntactic ones.

As future work, we plan to enrich the present attention mechanism with the _key-value-prediction_ technique Daniluk et al. (2016); Miller et al. (2016) which was shown to be useful for language modeling. Moreover, we will incorporate relative positional information to the attention function. To encourage further research in _self-attentive residual connections_ for NMT an other similar tasks, our code is made publicly available5.

Footnote 5: This work is part of the project _Towards Document-Level Neural Machine Translation_Miculich Werlen (2017). The code is available at [https://github.com/idiap/Attentive_Residual_Connections_NMT](https://github.com/idiap/Attentive_Residual_Connections_NMT)

#### Acknowledgments

We are grateful for support to the European Union under the Horizon 2020 SUMMA project (grant n. 688139, see www.summa-project.eu). We would also like to thank James Henderson for his valuable feedback and suggestions.

## References

* Bahdanau et al. (2015) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In _Proceedings of the International Conference on Learning Representations_. San Diego, USA.
* Black et al. (1991) Ezra W. Black, Steven Abney, Daniel P. Flickenger, Claudia Gdaniec, Ralph Grishman, Philip Harrison, Donald Hindle, Robert J. P. Ingria, Frederick Jelinek, Judith L. Klavans, Mark Y. Liberman, Mitchell P. Marcus, Salim Roukos, Beatrice Santorini, and Tomek Strzalkowski. 1991. A procedure for quantitatively comparing the syntactic coverage of English grammars. In _Speech and Natural Language: Proceedings of a Workshop Held at Pacific Grove_. California, USA. [http://www.aclweb.org/anthology/H91-1060](http://www.aclweb.org/anthology/H91-1060).
* Bensouss et al. (2014)

\begin{table}
\begin{tabular}{|p{227.6pt}|} \hline  & **Better than baseline** \\ \hline S: Estudiantes y professores se están tomando a la ligera la fécha. \\ R: Students and teachers are taking the date lightly. \\ B: Students and teachers are _being taken lightly to the date_. \\ O: Students and teachers are **taking the date lightly** \\ S: No porporate comparitera su ideologia, sino porque para el los Derechos Humanos son indivisibles. \\ R: Not because he shared their world view, but because for him, human rights are indivisible. \\ B: Not because \(I\) share his ideology, but because _he is indivisible by human rights_. \\ O: Not because **be** shared his ideology, but because **for him human rights are indivisible**. \\ \hline  & **Worse than baseline** \\ S: El gobierno intenta que no se construyan tantas casas pequenás. \\ R: The Government is trying not to build so many small houses. \\ B: The government is trying _not to build so many small houses_. \\ O: The government is trying **to ensure that so many smallhouses are not built**. \\ S: Otras persona’s pueden tener niñós. \\ R: Other people can have children. \\ B: _Other people can have children_. \\ O: Others may have children. \\ \hline \end{tabular}
\end{table}
Table 7: Examples from Spanish to English.

 

* Bojar et al. (2013) Ondrej Bojar, Christian Buck, Chris Callison-Burch, Christian Federmann, Barry Haddow, Philipp Koehn, Christof Monz, Matt Post, Radu Soricut, and Lucia Specia. 2013. Findings of the 2013 Workshop on Statistical Machine Translation. In _Proceedings of the Eighth Workshop on Statistical Machine Translation_. Association for Computational Linguistics, Sofia, Bulgaria, pages 1-44. [http://www.aclweb.org/anthology/Wl3-2201](http://www.aclweb.org/anthology/Wl3-2201).
* Bojar et al. (2016) Ondrej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christiot Monz, Matteo Negri, Aurelie Neveol, Mariana Neves, Martin Popel, Matt Post, Raphael Rubino, Carolina Scarton, Lucia Specia, Marco Turchi, Karin Verspoor, and Marcos Zampieri. 2016. Findings of the 2016 Conference on Machine Translation. In _Proceedings of the First Conference on Machine Translation_. Association for Computational Linguistics, Berlin, Germany, pages 131-198. [http://www.aclweb.org/anthology/Wl6-2301](http://www.aclweb.org/anthology/Wl6-2301).
* Cheng et al. (2016) Jianpeng Cheng, Li Dong, and Mirella Lapata. 2016. Long short-term memory-networks for machine reading. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, Austin, Texas, pages 551-561. [https://aclweb.org/anthology/D16-1053](https://aclweb.org/anthology/D16-1053).
* Cho et al. (2014) Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, Doha, Qatar, pages 1724-1734. [http://www.aclweb.org/anthology/D14-1179](http://www.aclweb.org/anthology/D14-1179).
* Daniluk et al. (2016) Michal Daniluk, Tim Rocktaschel, Johannes Welbl, and Sebastian Riedel. 2016. Frustratingly short attention spans in neural language modeling. In _Proceedings of the International Conference on Learning Representations_. San Juan, Puerto Rico.
* Gehring et al. (2017) Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N. Dauphin. 2017. Convolutional sequence to sequence learning. In Doina Precup and Yee Whye Teh, editors, _Proceedings of the 34th International Conference on Machine Learning_. PMLR, International Convention Centre, Sydney, Australia, volume 70 of _Proceedings of Machine Learning Research_, pages 1243-1252. [http://proceedings.mlr.press/v70/gehring17a.html](http://proceedings.mlr.press/v70/gehring17a.html).
* He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_. volume 00, pages 770-778. [https://doi.org/10.1109/CVPR.2016.90](https://doi.org/10.1109/CVPR.2016.90).
* Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jurgen Schmidhuber. 1997. Long short-term memory. _Neural Computation_ 9(8):1735-1780.
* Kalchbrenner and Blunsom (2013) Nal Kalchbrenner and Phil Blunsom. 2013. Recurrent continuous translation models. In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, Seattle, Washington, USA, pages 1700-1709. [http://www.aclweb.org/anthology/D13-1176](http://www.aclweb.org/anthology/D13-1176).
* Kim et al. (2017) Jaeyoung Kim, Mostafa El-Khamy, and Jungwon Lee. 2017. Residual LSTM: design of a deep recurrent architecture for distant speech recognition. _CoRR_ abs/1701.03360. [http://arxiv.org/abs/1701.03360](http://arxiv.org/abs/1701.03360).
* Koehn et al. (2007) Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine translation. In _Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics Companion Volume Proceedings of the Demo and Poster Sessions_. Association for Computational Linguistics, Prague, Czech Republic, pages 177-180. [http://www.aclweb.org/anthology/P07-2045](http://www.aclweb.org/anthology/P07-2045).
* Lee et al. (2017) Kenton Lee, Omer Levy, and Luke Zettlemoyer. 2017. Recurrent additive networks. _CoRR_ abs/1705.07393. [http://arxiv.org/abs/1705.07393](http://arxiv.org/abs/1705.07393).
* Lin et al. (2017) Zhouhan Lin, Minwei Feng, Cicero Nogueira dos Santos, Mo Yu, Bing Xiang, Bowen Zhou, and Yoshua Bengio. 2017. A structured self-attentive sentence embedding. In _Proceedings of the International Conference on Learning Representations_. Toulon, France.
* Liu and Lapata (2018) Yang Liu and Mirella Lapata. 2018. Learning structured text representations. _Transactions of the Association for Computational Linguistics_ 6:63-75. [https://transacl.org/ojs/index.php/tacl/article/view/1185/280](https://transacl.org/ojs/index.php/tacl/article/view/1185/280).
* Luong et al. (2015) Thang Luong, Hieu Pham, and D. Christopher Manning. 2015. Effective approaches to attention-based neural machine translation. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, pages 1412-1421. [https://doi.org/10.18653/v1/D15-1166](https://doi.org/10.18653/v1/D15-1166).
* Manning et al. (2014) Christopher Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven Bethard, and David McClosky. 2014. The Stanford CoreNLP natural language processing toolkit. In _Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations_. Association for Computational Linguistics, Baltimore, Maryland, pages 55-60. [http://www.aclweb.org/anthology/P14-5010](http://www.aclweb.org/anthology/P14-5010) 

* Werlen (2017) Lesly Miculicich Werlen. 2017. Towards document-level neural machine translation. Idiap-RR Idiap-RR-25-2017, Idiap.
* Miller et al. (2016) Alexander Miller, Adam Fisch, Jesse Dodge, Amir-Hossein Karimi, Antoine Bordes, and Jason Weston. 2016. Key-value memory networks for directly reading documents. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, Austin, Texas, pages 1400-1409. [https://aclweb.org/anthology/D16-1147](https://aclweb.org/anthology/D16-1147).
* Nadejde et al. (2017) Maria Nadejde, Siva Reddy, Rico Sennrich, Tomasz Duvojak, Marcin Junczys-Downunt, Philipp Koehn, and Alexandra Birch. 2017. Predicting target language CCG supertags improves neural machine translation. In _Proceedings of the Second Conference on Machine Translation_. Association for Computational Linguistics, Copenhagen, Denmark, pages 68-79. [http://www.aclweb.org/anthology/Wl7-4707](http://www.aclweb.org/anthology/Wl7-4707).
* Pappas and Popescu-Belis (2017) Nikolaos Pappas and Andrei Popescu-Belis. 2017. Explicit document modeling through weighted multiple-instance learning. _Journal of Artificial Intelligence Research_ 58:591-626.
* Rafalovitch and Dale (2009) Alexandre Rafalovitch and Robert Dale. 2009. United Nations General Assembly resolutions: A six-language parallel corpus. In _Proceedings of the MT Summit_. volume 12, pages 292-299.
* Sennrich et al. (2016a) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016a. Edinburgh Neural Machine Translation Systems for WMT 16. In _Proceedings of the First Conference on Machine Translation_. Association for Computational Linguistics, Berlin, Germany, pages 371-376. [http://www.aclweb.org/anthology/W/Wl6/Wl6-2323](http://www.aclweb.org/anthology/W/Wl6/Wl6-2323).
* Sennrich et al. (2016b) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016b. Neural machine translation of rare words with subword units. In _Proceedings of the 54th Annual Meeting of the ACL (Vol. 1: Long Papers)_. Association for Computational Linguistics, Berlin, Germany, pages 1715-1725. [http://www.aclweb.org/anthology/P16-1162](http://www.aclweb.org/anthology/P16-1162).
* Soltani and Jiang (2016) Rohollah Soltani and Hui Jiang. 2016. Higher order recurrent neural networks. _CoRR_ abs/1605.00064. [http://arxiv.org/abs/1605.00064](http://arxiv.org/abs/1605.00064).
* Sukhbaatar et al. (2015) Sanibayar Sukhbaatar, arthur szlam, Jason Weston, and Rob Fergus. 2015. End-to-end memory networks. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, _Advances in Neural Information Processing Systems 28_, Curran Associates, Inc., pages 2440-2448. [http://papers.nips.cc/paper/5846-end-to-end-memory-networks.pdf](http://papers.nips.cc/paper/5846-end-to-end-memory-networks.pdf).
* Sutskever et al. (2014) Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger, editors, _Advances in Neural Information Processing Systems 27_, Curran Associates, Inc., pages 3104-3112.
* Team (2016) Theano Development Team. 2016. Theano: A Python framework for fast computation of mathematical expressions. _arXiv e-prints_ abs/1605.02688. [http://arxiv.org/abs/1605.02688](http://arxiv.org/abs/1605.02688).
* Tran et al. (2016) Ke Tran, Arianna Bisazza, and Christof Monz. 2016. Recurrent memory networks for language modeling. In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_. Association for Computational Linguistics, San Diego, California, pages 321-331. [http://www.aclweb.org/anthology/N16-1036](http://www.aclweb.org/anthology/N16-1036).
* Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, _Advances in Neural Information Processing Systems 30_, Curran Associates, Inc., pages 5998-6008. [http://papers.nips.cc/paper/7181-attention-is-all-you-need.pdf](http://papers.nips.cc/paper/7181-attention-is-all-you-need.pdf).
* Wang (2017) Cheng Wang. 2017. RPA: recurrent residual attention for sequence learning. _CoRR_ abs/1709.03714. [http://arxiv.org/abs/1709.03714](http://arxiv.org/abs/1709.03714).
* Wang et al. (2016) Mingxuan Wang, Zhengdong Lu, Hang Li, and Qun Liu. 2016. Memory-enhanced decoder for neural machine translation. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, Austin, Texas, pages 278-286. [https://aclweb.org/anthology/D16-1027](https://aclweb.org/anthology/D16-1027).
* Wang and Tian (2016) Yiren Wang and Fei Tian. 2016. Recurrent residual learning for sequence classification. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, Austin, Texas, pages 938-943. [https://aclweb.org/anthology/D16-1093](https://aclweb.org/anthology/D16-1093).
* Weston et al. (2015) Jason Weston, Sumit Chopra, and Antoine Bordes. 2015. Memory networks. In _Proceedings of the International Conference on Learning Representations_. San Diego, USA.
* Williams et al. (2014) Philip Williams, Rico Sennrich, Maria Nadejde, Matthias Huck, Eva Hasler, and Philipp Koehn. 2014. Edinburgh's Syntax-Based Systems at WMT 2014. In _Proceedings of the Ninth Workshop on Statistical Machine Translation_. Association for Computational Linguistics, Baltimore, Maryland, USA 

pages 207-214. [http://www.aclweb.org/anthology/W/W14/W14-3324](http://www.aclweb.org/anthology/W/W14/W14-3324).
* Williams et al. (2015) Philip Williams, Rico Sennrich, Maria Nadejde, Matthias Huck, and Philipp Koehn. 2015. Edinburgh's Syntax-Based Systems at WMT 2015. In _Proceedings of the Tenth Workshop on Statistical Machine Translation_. Association for Computational Linguistics, Lisbon, Portugal, pages 199-209. [http://aclweb.org/anthology/W15-3024.pdf](http://aclweb.org/anthology/W15-3024.pdf).
* Yang et al. (2017) Zichao Yang, Zhiting Hu, Yuntian Deng, Chris Dyer, and Alex Smola. 2017. Neural machine translation with recurrent attention modeling. In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers_. Association for Computational Linguistics, Valencia, Spain, pages 383-387. [http://www.aclweb.org/anthology/E17-2061](http://www.aclweb.org/anthology/E17-2061).
* Zeiler (2012) Matthew D. Zeiler. 2012. ADADELTA: an adaptive learning rate method. _CoRR_ abs/1212.5701. [http://arxiv.org/abs/1212.5701](http://arxiv.org/abs/1212.5701).
* Zhang et al. (2017) Biao Zhang, Deyi Xiong, Jinsong Su, and Hong Duan. 2017. A context-aware recurrent encoder for neural machine translation. _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ 25(12):2424-2432.
* Zhang et al. (2016a) Biao Zhang, Deyi Xiong, jinsong su, Hong Duan, and Min Zhang. 2016a. Variational neural machine translation. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, Austin, Texas, pages 521-530. [https://aclweb.org/anthology/D16-1050](https://aclweb.org/anthology/D16-1050).
* Zhang et al. (2016b) Yu Zhang, Guoguo Chen, Dong Yu, Kaisheng Yaco, Sanjeev Khudanpur, and James Glass. 2016b. Highway long short-term memory RNNs for distant speech recognition. In _2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. pages 5755-5759. [https://doi.org/10.1109/ICASSP.2016.7472780](https://doi.org/10.1109/ICASSP.2016.7472780).
* Zilly et al. (2017) Julian Georg Zilly, Rupesh Kumar Srivastava, Jan Koutnik, and Jurgen Schmidhuber. 2017. Recurrent highway networks. In Doina Precup and Yee Whye Teh, editors, _Proceedings of the 34th International Conference on Machine Learning_. PMLR, International Convention Centre, Sydney, Australia, volume 70 of _Proceedings of Machine Learning Research_, pages 4189-4198. [http://proceedings.mlr.press/v70/zilly17a.html](http://proceedings.mlr.press/v70/zilly17a.html).

## Appendix A Detailed Architecture

This appendix describes in detail the implementation of the _self-attentive residual decoder_ for NMT, which builds on the attention-based NMT implementation of dl4mt-tutorial6.
Footnote 6: [https://github.com/nyu-dl/dl4mt-tutorial](https://github.com/nyu-dl/dl4mt-tutorial)

The input of the model is a source sentence denoted as 1-of-k coded vector, where each element of the sequence corresponds to a word:

\[x=(x_{1},x_{2},...,x_{m}),x_{i}\in\mathbb{R}^{V}\]

and the output is a target sentence denoted as well as 1-of-k coded vector:

\[y=(y_{1},y_{2},...,y_{n}),y_{i}\in\mathbb{R}^{V}\]

where \(V\) is the size of the vocabulary of target and source side, \(m\) and \(n\) are the lengths of the source and target sentences respectively. We omit the bias vectors for simplicity.

### Encoder

Each word of the source sentence is embedded in a \(e\)-dimensional vector space using the embedding matrix \(\bar{E}\in\mathbb{R}^{e\times V}\). The hidden states are \(2d\)-dimensional vectors modeled by a bi-directional GRU. The forward states \(\overrightarrow{h}=(\overrightarrow{h}_{1},...,\overrightarrow{h}_{m})\) are computed as:

\[\overrightarrow{h}_{i}=\overrightarrow{g}_{i}\odot\overrightarrow{h}_{i-1}+(1- \overrightarrow{g}_{i})\odot\overrightarrow{h}_{i}^{\prime}\]

where

\[\overrightarrow{h}_{i}^{\prime} =tanh(\overrightarrow{W}\bar{E}x_{i}+\overrightarrow{U}[ \overrightarrow{r}_{i}\odot\overrightarrow{h}_{i-1}])\] \[\overrightarrow{z}_{i} =\sigma(\overrightarrow{W}_{z}\bar{E}x_{i}+\overrightarrow{U}_{ z}\overrightarrow{h}_{i-1})\] \[\overrightarrow{r}_{i} =\sigma(\overrightarrow{W}_{r}\bar{E}x_{i}+\overrightarrow{U}_{ r}\overrightarrow{h}_{i-1})\]

Here, \(\overrightarrow{W},\overrightarrow{W}_{z},\overrightarrow{W}_{r}\in\mathbb{ R}^{d\times e}\) and \(\overrightarrow{U},\overrightarrow{U}_{z},\overrightarrow{U}_{r}\in\mathbb{R}^{ d\times d}\) are weight matrices. The backward states \(\overleftarrow{h}=(\overleftarrow{h}_{1},...,\overleftarrow{h}_{m})\) are computed in similar manner. The embedding matrix \(\bar{E}\) is shared for both passes, and the final hidden states are formed by the concatenation of them:

\[h_{i}=\left[\begin{array}{c}\overrightarrow{h}_{i}\\ \overleftarrow{h}_{i}\end{array}\right]\]

### Attention Mechanism

The _context vector_ at time \(t\) is calculated by:

\[c_{t}=\sum_{i=1}^{m}\alpha_{i}^{t}h_{i}\] 

where

\[\alpha_{i}^{t} =\frac{exp(e_{t}^{t})}{\sum_{j}exp(e_{j}^{t})}\] \[e_{t}^{t} =v_{a}^{t}tanh(W_{d}s_{t-1}+W_{e}h_{i})\]

Here, \(v_{a}\in\mathbb{R}^{d}\), \(W_{d}\in\mathbb{R}^{d\times d}\) and \(W_{e}\in\mathbb{R}^{d\times 2d}\) are weight matrices.

### Decoder

The input of the decoder are the previous word \(y_{t-1}\) and the _context vector_\(c_{t}\), the objective is to predict \(y_{t}\). The hidden states of the decoder \(s=(s_{1},...,s_{n})\) are initialized with the mean of the _context vectors_:

\[s_{0}=tanh(W_{init}\frac{1}{m}\sum_{i=1}^{m}c_{i})\]

where \(W_{init}\in\mathbb{R}^{d\times 2d}\) is a weight matrix, \(m\) is the size of the source sentence. The following hidden states are calculated with a GRU conditioned over the _context vector_ at time \(t\) as follows:

\[s_{t}=z_{t}\odot s_{t}^{\prime}+(1-z_{t})\odot s_{t}^{\prime\prime}\]

where

\[s_{t}^{\prime\prime} =tanh(Ey_{t-1}+U[r_{t}\odot s_{t-1}]+Cc_{t})\] \[z_{i} =\sigma(W_{z}Ey_{t-1}+U_{z}s_{t-1}+C_{z}c_{t})\] \[r_{i} =\sigma(W_{r}Ey_{t-1}+U_{r}s_{t-1}+C_{r}c_{t})\]

Here, \(E\in\mathbb{R}^{e\times V}\) is the embedding matrix for the target language. \(W,W_{z},W_{r}\in\mathbb{R}^{d\times e}\), \(U,U_{z},U_{r}\in\mathbb{R}^{d\times d}\), and \(C,C_{z},C_{r}\in\mathbb{R}^{d\times 2d}\) are weight matrices. The intermediate vector \(s_{t}^{\prime}\) is calculated from a simple GRU:

\[s_{t}^{\prime}=GRU(y_{t-1},s_{t-1})\]

In the attention-based NMT model, the probability of a target word \(y_{t}\) is given by:

\[p(y_{t}|s_{t},y_{t-1},c_{t})=softmax(W_{o}tanh(\\ W_{st}s_{t}+W_{yt}y_{t-1}+W_{ct}c_{t}))\]

Here, \(W_{o}\in\mathbb{R}^{V\times e}\), \(W_{st}\in\mathbb{R}^{e\times d}\), \(W_{yt}\in\mathbb{R}^{e\times e}\), \(W_{ct}\in\mathbb{R}^{e\times 2d}\) are weight matrices.

#### a.3.1 Self-Attentive Residual Connections

In our model, the probability of a target word \(y_{t}\) is given by:

\[p(y_{t}|s_{t},d_{t},c_{t})= softmax(W_{o}tanh(\\ W_{st}s_{t}+W_{dt}d_{t}+W_{ct}c_{t}))\]

Here, \(W_{o}\in\mathbb{R}^{V\times e}\), \(W_{st}\in\mathbb{R}^{e\times d}\), \(W_{dt},W_{yt}\in\mathbb{R}^{e\times e}\), \(W_{ct}\in\mathbb{R}^{e\times 2d}\) are weight matrices. The summary vector \(d_{t}\) can be calculated in different manners based on previous words \(y_{1}\) to \(y_{t-1}\). First, a simple average:

\[d_{t}^{avg}=\frac{1}{t-1}\sum_{i=1}^{t-1}y_{i}\]

The second, by using an attention mechanism:

\[d_{t}^{cavg}=\sum_{i=1}^{t-1}\alpha_{i}^{t}y_{i}\] \[\alpha_{i}^{t}=\frac{exp(e_{i}^{t})}{\sum_{j=1}^{t-1}exp(e_{j}^{ t})}\] \[e_{i}^{t}=v^{t}tanh(W_{y}y_{i})\]

where \(v\in\mathbb{R}^{e}\), \(W_{y}\in\mathbb{R}^{e\times e}\) are weight matrices.

#### a.3.2 Memory RNN

This model modifies the recurrent layer of the decoder as follows:

\[s_{t}=z_{t}\odot s_{t}^{\prime}+(1-z_{t})\odot s_{t}^{\prime\prime}\]

where

\[s_{t}^{\prime\prime} =tanh(Ey_{t-1}+U[r_{t}\odot\tilde{s}_{t}]+Ce_{t})\] \[z_{i} =\sigma(W_{z}Ev_{t-1}+U_{z}\tilde{s}_{t}+C_{z}c_{t})\] \[r_{i} =\sigma(W_{r}Ey_{t-1}+U_{r}\tilde{s}_{t}+C_{r}c_{t})\]

Here, \(E\in\mathbb{R}^{e\times V}\) is the embedding matrix for the target language. \(W,W_{z},W_{r}\in\mathbb{R}^{d\times e}\), \(U,U_{z},U_{r}\in\mathbb{R}^{d\times d}\), and \(C,C_{z},C_{r}\in\mathbb{R}^{d\times 2d}\) are weight matrices. The intermediate vector \(s_{t}^{\prime}\) is calculated from a simple GRU:

\[s_{t}^{\prime}=GRU(y_{t-1},\tilde{s}_{t})\]

The recurrent vector \(\tilde{s}_{t}\) is calculated as following:

\[\tilde{s}_{t} =\sum_{i=1}^{t-1}\alpha_{i}^{t}s_{i}\] where \[\alpha_{i}^{t} =\frac{exp(e_{i}^{t})}{\sum_{j=1}^{t-1}exp(e_{j}^{t})}\] \[e_{i}^{t} =v^{t}tanh(W_{m}s_{i}+W_{s}s_{t})\]

where \(v\in\mathbb{R}^{d}\), \(W_{m}\in\mathbb{R}^{d\times d}\), and \(W_{s}\in\mathbb{R}^{d\times d}\) are weight matrices.

 



#### a.3.3 Self-Attentive RNN

The formulation of this decoder is as following:

\[p(y_{t}|y_{1},...,y_{t-1},c_{t})\approx softmax(W_{o}tanh(\\ W_{st}s_{t}+W_{yt}y_{t-1}+W_{ct}c_{t}+W_{mt}\tilde{s}_{t}))\] (A.3.25)

Here, \(W_{o}\in\mathbb{R}^{V\times e}\), \(W_{st}\in\mathbb{R}^{e\times d}\), \(W_{yt}\in\mathbb{R}^{e\times e}\), \(W_{ct}\in\mathbb{R}^{e\times 2d}\), and \(W_{mt}\in\mathbb{R}^{e\times d}\) are weight matrices.

\[\tilde{s}_{t}=\sum_{i=1}^{t-1}\alpha_{i}^{t}s_{i}\] \[\alpha_{i}^{t}=\frac{exp(e_{i}^{t})}{\sum_{j=1}^{t-1}exp(e_{j}^{ t})}\] \[e_{i}^{t}=v^{\intercal}tanh(W_{m}s_{i}+W_{s}s_{t})\]

where \(v\in\mathbb{R}^{d}\), \(W_{m}\in\mathbb{R}^{d\times d}\), and \(W_{s}\in\mathbb{R}^{d\times d}\) are weight matrices.