Year: 2018

Abstract:
Most state-of-the-art neural machine translation systems, despite being different in architectural skeletons (e.g., recurrence, convolutional), share an indispensable feature: the Attention. However, most existing attention methods are token-based and ignore the importance of phrasal alignments, the key ingredient for the success of phrase-based statistical machine translation. In this paper, we propose novel phrase-based attention methods to model n-grams of tokens as attention entities. We incorporate our phrase-based attentions into the recently proposed Transformer network, and demonstrate that our approach yields improvements of up to 1.3 and 0.5 BLEU points in English-to-German and German-to-English translation tasks, and 1.75 and 1.35 BLEU points in English-to-Russian and Russian-toEnglish translation tasks on WMT’14 test set using WMT’16 training data.

text: Most state-of-the-art neural machine translation systems, despite being different in architectural skeletons (e.g., recurrence, convolutional), share an indispensable feature: the Attention. However, most existing attention methods are token-based and ignore the importance of phrasal alignments, the key ingredient for the success of phrase-based statistical machine translation. In this paper, we propose novel phrase-based attention methods to model n-grams of tokens as attention entities. We incorporate our phrase-based attentions into the recently proposed Transformer network, and demonstrate that our approach yields improvements of up to 1.3 and 0.5 BLEU points in English-to-German and German-to-English translation tasks, and 1.75 and 1.35 BLEU points in English-to-Russian and Russian-toEnglish translation tasks on WMT’14 test set using WMT’16 training data.

heading: 1 INTRODUCTION
text: Neural Machine Translation (NMT) has established breakthroughs in many different translation tasks, and has quickly become the standard approach to machine translation. NMT offers a simple encoder-decoder architecture that is trained end-to-end. Most NMT models (except a few like (Kaiser & Bengio, 2016) and (Huang et al., 2018)) possess attention mechanisms to perform alignments of the target tokens to the source tokens. The attention module plays a role analogous to the word alignment model in Statistical Machine Translation or SMT (Koehn, 2010). In fact, the Transformer network introduced recently by Vaswani et al. (2017) achieves state-of-the-art performance in both speed and BLEU scores (Papineni et al., 2002) by using only attention modules.
On the other hand, phrasal interpretation is an important aspect for many language processing tasks, and forms the basis of Phrase-Based Machine Translation (Koehn, 2010). Phrasal alignments (Koehn et al., 2003) can model one-to-one, one-to-many, many-to-one, and many-to-many relations between source and target tokens, and use local context for translation. They are also robust to non-compositional phrases. Despite the advantages, the concept of phrasal attentions has largely been neglected in NMT, as most NMT models generate translations token-by-token autoregressively, and use the token-based attention method which is order invariant. Therefore, the intuition of phrase-based translation is vague in existing NMT systems that solely depend on the underlying neural architectures (recurrent, convolutional, or self-attention) to incorporate contextual information. However, the information aggregation strategies employed by the underlying neural architectures provide context-relevant clues only to represent the current token, and do not explicitly model phrasal alignments. We argue that having an explicit inductive bias for phrases and phrasal alignments is necessary for NMT to exploit the strong correlation between source and target phrases.
In this paper, we propose phrase-based attention methods for phrase-level alignments in NMT. Specifically, we propose two novel phrase-based attentions, namely CONVKV and QUERYK, designed to assign attention scores directly to phrases in the source and compute phrase-level attention vector for the target. We also introduce three new attention structures, which apply these methods to conduct phrasal alignments. Our homogeneous and heterogeneous attention structures perform token-to-token and token-to-phrase mappings, while the interleaved heterogeneous attention structure models all token-to-token, token-to-phrase, phrase-to-token, and phrase-to-phrase alignments.
To show the effectiveness of our approach, we apply our phrase-based attention methods to all multi-head attention layers of the Transformer. Our experiments on WMT’14 translation tasks show improvements of up to 1.3 and 0.5 BLEU points for English-to-German and German-to-English
respectively, and up to 1.75 and 1.35 BLEU points for English-to-Russian and Russian-to-English respectively, compared to the baseline Transformer network trained in identical settings.

heading: 2 BACKGROUND
text: Most NMT models adopt an encoder-decoder framework, where the encoder network first transforms an input sequence of symbols x = (x1, x2, . . . , xn) to a sequence of continuous representations Z = (z1, z2, . . . ,zn), from which the decoder generates a target sequence of symbols y = (y1, y2, . . . , ym) autoregressively, one element at a time. Recurrent seq2seq models with diverse structures and complexity (Sutskever et al., 2014; Bahdanau et al., 2015; Luong et al., 2015; Wu et al., 2016) are the first to yield state-of-the-art results. Convolutional seq2seq models (Kalchbrenner et al., 2016; Gehring et al., 2017; Kaiser et al., 2018) alleviate the drawback of sequential computation of recurrent models and leverage parallel computation to reduce training time.
The recently proposed Transformer network (Vaswani et al., 2017) structures the encoder and the decoder entirely with stacked self-attentions and cross-attentions (only in the decoder). In particular, it uses a multi-headed, scaled multiplicative attention defined as follows:
Attention(Q,K,V ,Wq,Wk,Wv) = S( (QWq)(KWk)
T
√ dk
)(V Wv) (1)
Headi = Attention(Q,K,V ,W iq ,W i k,W i v) for i = 1 . . . h (2)
AttentionOutput(Q,K,V ,W ) = concat(Head1,Head2, . . . ,Headh)W (3)
where S is the softmax function, Q, K, V are the matrices with query, key, and value vectors, respectively, dk is the dimension of the query/key vectors; W iq , W i k, W i v are the head-specific weights for query, key, and value vectors, respectively; and W is the weight matrix that combines the outputs of the heads. The attentions in the encoder and decoder are based on self-attention, where all of Q, K and V come from the output of the previous layer. The decoder also has crossattention, where Q comes from the previous decoder layer, and the K-V pairs come from the encoder. We refer readers to (Vaswani et al., 2017) for further details of the network design.
One crucial issue with the attention mechanisms employed in the Transformer network as well as other NMT architectures (Luong et al., 2015; Gehring et al., 2017) is that they are order invariant locally and globally. That is, changing the order of the vectors in Q, K and V does not change the resulted attention weights and vectors. If this problem is not tackled properly, the model may not learn the sequential characteristics of the data. RNN-based models (Bahdanau et al., 2015; Luong et al., 2015) tackle this issue with a recurrent encoder and decoder, CNN-based models like (Gehring et al., 2017) use position embeddings, while the Transformer uses positional encoding. Another limitation is that these attention methods attend to tokens, and play a role analogous to word alignment models in traditional SMT. It is, however, well admitted in SMT that phrases are better than words as translation units (Koehn, 2010). Without explicit attention to phrases, a particular attention function has to depend entirely on the token-level softmax scores of a phrase for phrasal alignment, which is not robust and reliable, thus making it more difficult for the model to learn the required mappings. For example, the attention heatmaps of the Transformer (Vaswani et al., 2017) show concentration of the scores on individual tokens even if it uses multiple heads concurrently in multiple layers. Our main hypothesis is that in order to exploit the strong correlation between source and target phrases, the NMT models should have explicit inductive biases towards phrases.
There exists some research on phrase-based decoding in NMT framework. For example, Huang et al. (2018) proposed a phrase-based decoding approach based on a soft reordering layer and a Sleep-WAke Network (SWAN), a segmentation-based sequence model proposed by Wang et al. (2017a). Their decoder uses a recurrent architecture without any attention on the source. Tang et al. (2016) and Wang et al. (2017b) used an external phrase memory to decode phrases for a Chinese-toEnglish translation task. In addition, hybrid search and PBMT were introduced to perform phrasal translation in (Dahlmann et al., 2017). Nevertheless, to the best of our knowledge, our work is the first to embed phrases into attention modules, which thus propagate the information throughout the entire end-to-end Transformer network, including the encoder, decoder, and the cross-attention.

heading: 3 MULTI-HEAD PHRASE-BASED ATTENTION
text: In this section, we present our proposed methods to compute attention weights and vectors based on n-grams of queries, keys, and values. We compare and discuss the pros and cons of these methods. For simplicity, we describe them in the context of the Transformer network; however, it is straightforward to apply them to other architectures such as RNN-based or CNN-based seq2seq models.

heading: 3.1 PHRASE-BASED ATTENTION METHODS
text: In this subsection, we present two novel methods to achieve phrasal attention. In Subsection 3.2, we present our methods for combining different types of n-gram attentions. The key element in our methods is a temporal (or one-dimensional) convolutional operation that is applied to a sequence of vectors representing tokens. Formally, we can define the convolutional operator applied to each token xt with corresponding vector representation xt ∈ IRd1 as:
ot = w ⊕nk=0 xt±k (4)
where ⊕ denotes vector concatenation, w ∈ IRn×d1 is the weight vector (a.k.a. kernel), and n is the window size. We repeat this process with d2 different weight vectors to get a d2-dimensional latent representation for each token xt. We will use the notation Convn(X,W ) to denote the convolution operation over an input sequence X with window size n and kernel weights W ∈ IRn×d1×d2 .

heading: 3.1.1 KEY-VALUE CONVOLUTION
text: The intuition behind key-value convolution technique is to use trainable kernel parameters Wk and Wv to compute the latent representation of n-gram sequences using convolution operation over key and value vectors. The attention function with key-value convolution is defined as:
CONVKV(Q,K,V ) = S( (QWq)Convn(K,Wk) T
√ dk
) Convn(V ,Wv) (5)
where S is the softmax function, Wq ∈ IRdq×dk ,Wk ∈ IRn×dk×dk ,Wv ∈ IRn×dv×dv are the respective kernel weights for Q, K and V . Throughout this paper, we will use S to denote the softmax function. Note that in this convolution, the key and value sequences are left zero-padded so that the sequence length is preserved after the convolution (i.e., one latent representation per token).
The CONVKV method can be interpreted as indirect query-key attention, in contrast to the direct query-key approach to be described next. This means that the queries do not interact directly with the keys to learn the attention weights; instead the model relies on the kernel weights (Wk) to learn n-gram patterns.

heading: 3.1.2 QUERY-AS-KERNEL CONVOLUTION
text: In order to allow the queries to directly and dynamically influence the word order of phrasal keys and values, we introduce Query-as-Kernel attention method. In this approach, when computing the attention weights, we use the query as kernel parameters in the convolution applied to the series of keys. The attention output in this approach is given by:
QUERYK(Q,K,V ) = S(Convn(KWk,QWq)√ dk ∗ n ) Convn(V ,Wv) (6)
where Wq ∈ IRn×dq×dk ,Wk ∈ IRdk×dk ,Wv ∈ IRn×dv×dv are trainable weights. Notice that we include the window size n (phrase length) in the scaling factor to counteract the fact that there are n times more multiplicative operations in the convolution than the traditional matrix multiplication.

heading: 3.2 MULTI-HEADED PHRASAL ATTENTION
text: Having presented the two phrase-based attention methods in the previous subsection, we now introduce our extensions to the multi-headed attention framework of the Transformer to enable it to pay attention not only to tokens but also to phrases across many sub-spaces and locations.

heading: 3.2.1 HOMOGENEOUS N-GRAM ATTENTION
text: In homogeneous n-gram attention, we distribute the attention heads to different n-gram types with each head attending to one particular n-gram type (n = 1, 2, . . . , N ). For instance, Figure 1 shows a homogeneous structure, where the first four heads attend to unigrams, and the last four attend to bigrams. A head can apply one of the phrasal attention methods described in Subsection 3.1. The selection of which n-gram to assign to how many heads is considered as hyperparameters to the model. Since all heads must have consistent sequence length, phrasal attention heads in the homogeneous setting require left-padding of keys and values before convolution.
Since each head attends to a subspace resulting from one type of n-gram, homogeneous attention learns the mappings in a distributed way. However, the homogeneity restriction may limit the model to learn interactions between different n-gram types since the gradients for different n-gram types flow in parallel paths with no explicit interactions. Furthermore, the homogeneous heads force the model to assign each query with attentions on all n-gram types (e.g., unigrams and bigrams) even when it does not need to do so, thus possibly inducing more noise into the model.

heading: 3.2.2 HETEROGENEOUS N-GRAM ATTENTION
text: The heterogeneous n-gram attention relaxes the constraint of the homogeneous approach. Instead of limiting each head’s attention to a particular type of n-gram, it allows the query to freely attend to all types of n-grams simultaneously. To achieve this, we first compute the attention logit for each n-gram type separately (i.e., for n = 1, 2, . . . , N ), then we concatenate all the logits before passing them through the softmax layer to compute the attention weights over all n-gram types. Similarly, the value vectors for the n-gram types are concatenated to produce the overall attention output. Figure 2 demonstrates the heterogeneous attention process for unigrams and bigrams.
For CONVKV technique in Equation 5, the attention output is given by:
S( (QWq)[(KWk,1) T ;Conv2(K,Wk,2)T ; ...]√
dk )[(V Wv,1);Conv2(V ,Wv,2); ...] (7)
For QUERYK technique (Equation 6), the attention output is given as follows:
S([ (QWq,1)(KWk,1) T
√ d
; Conv2(KWk,2,QWq,2)√
d ∗ n2 ; ...])[(V Wv,1);Conv2(V ,Wv,2); ...] (8)
Note that in heterogeneous attention, we do not need to pad the input sequences before the convolution operation to ensure identical sequence length. Also, the key/value sequences that are shorter than the window size do not have any valid phrasal component to be attended.

heading: 3.3 INTERLEAVED PHRASES TO PHRASE HETEROGENEOUS ATTENTION
text: All the methods presented above perform attention mappings from token-based queries to phrasebased key-value pairs. In other words, they model token-to-token and token-to-phrase structures. These types of attentions are beneficial when there exists a translation of a phrase in the source language (keys and values) to a single token in the target language (query). However, these methods are not explicitly designed to work in the reverse direction when a phrase or a token in the source language should be translated to a phrase in the target language. In this section, we present a novel approach to heterogeneous phrasal attention that allows phrasal queries to attend to tokens and phrases of keys and values (i.e., phrase-to-token and phrase-to-phrase mappings).
We accomplish this with the QUERYK and CONVKV methods as follows. We first apply convolutions Convn(Q,Wqn) on the query sequence with window size n to obtain the n-gram hidden representations of the query. Consider Figure 3, where we apply convolution on Q for n = 1 and n = 2 to generate the respective unigram and bigram queries.1 These queries are then used to attend over unigram and bigram key-values to generate the heterogeneous attention vectors as follows.
A1,ConvKV = S( (QWq1 )[(KWk,1)
T ;Conv2(K,Wk,2) T ;...]√
dk )[(V Wv,1);Conv2(V ,Wv,2); . . .] (9)
A2,ConvKV = S( Conv2(Q,Wq2 )[(KWk,1) T ;Conv2(K,Wk,2) T ;...]√
dk )[(V Wv,1);Conv2(V ,Wv,2); . . .] (10)
A1,QueryK = S([ (QWq1,1)(KWk,1)
T
√ d
; Conv2(KWk,2,QWq1,2)√
d∗n2 ; . . .])[(V Wv,1);Conv2(V ,Wv,2); . . .] (11)
A2,QueryK =S([ Conv2(Q,Wq2,1)(KWk,1)
T
√ d
; Conv2(KWk,2,Conv2(Q,Wq2,2))√
d∗n2 ; ..])[(V Wv,1);Conv2(V ,Wv,2); ..](12)
The result of these operations (Eq. 9 - 12) is a sequence of unigram and bigram attention vectors A1 = (u1,u2, . . . ,uN ) and A2 = (b1,b2, . . . ,bN−1) respectively, where N is the query length. Note that each ui ∈ A1 represents the attention vector for Qi unigram query, and each bi ∈ A2
1Conv1(Q,Wq1) is equivalent to a feed-forward connection.
represents the attention vector for (Qi-Qi+1) bigram queries, and these vectors are generated by attending to n-grams (n = 1, 2, . . . , N ) of keys and values.
In the next step, the phrase-level attentions in A2 are interleaved with the unigram attentions in A1 to form an interleaved attention sequence I such that the vectors are aligned. For unigram and bigram queries, the interleaved vector sequences at the encoder and decoder are formed as
Ienc = (0,u1,b1,u2,b2,u3, . . . ,bN−1,uN ,0) (13) {Idec, Icross} = (0,u1,b1,u2,b2,u3, . . . ,bN−1,uN ) (14)
where Ienc and Idec denote the interleaved sequence for self-attention at the encoder and decoder respectively, and Icross denotes the interleaved sequence for cross-attention between the encoder and the decoder. Note that although Idec and Icross are computed using the same formula (Eq. 14), they are different entities, operating over different input sequences – the input to Idec comes from the self-attended features in the target side, whereas the input to Icross comes from the cross-attended features from source. Also, to prevent information flow from the future in the decoder, the right connections are masked out in Idec and Icross (similar to the original Transformer). The interleaving operation places the phrase- and token-based representations of a token next to each other. The interleaved vectors are finally passed through a convolution layer (as opposed to a point-wise feedforward layer in the Transformer) to compute the overall representation for each token. By doing so, each query is intertwined with the n-gram representations of the phrases containing itself, which enables the model to learn the query’s correlation with neighboring tokens. For unigram and bigram queries, the encoder uses a convolution layer with a window size of 3 and stride of 2 to allow the token to intertwine with its past and future phrase representations, while the ones in the decoder (self- and cross-attention) use a window size of 2 and stride of 2 to incorporate only the past phrase representations to preserve the autoregressive property. More formally,
Oenc = Convwindow=3,stride=2(Ienc,Wenc) (15) Ocross = Convwindow=2,stride=2(Icross,Wcross) (16) Odec = Convwindow=2,stride=2(Idec,Wdec) (17)

heading: 4 EXPERIMENTS
text: In this section, we present the training settings, experimental results and analysis of our models.

heading: 4.1 TRAINING SETTINGS
text: We preserve most of the training settings from Vaswani et al. (2017) to enable a fair comparison with the original Transformer. Specifically, we use the Adam optimizer (Kingma & Ba, 2014) with β1 = 0.9, β2 = 0.98, and = 10−9. We follow a similar learning rate schedule withwarmup steps of 16000: LearningRate = 2 ∗ d−0.5 ∗ min(step num−0.5, step num ∗ warmup steps−1.5). While Vaswani et al. (2017) trained their base and big models at a massive scale with 8 GPUs, we could train our models only on a single GPU because of limited GPU facilities. We trained our models and the baseline Transformer on the same GPU for 500,000 steps. The batches were formed by sentence pairs containing approximately 4096 source and 4096 target tokens. Similar to Vaswani et al. (2017), we also applied residual dropout with 0.1 probability and label smoothing with ls = 0.1. Our models are implemented in the tensor2tensor2 library (Vaswani et al., 2018), on top of the original Transformer codebase. We conducted all the experiments with our models and the original Transformer in an identical setup for a fair comparison.
We trained our models on the standard WMT’16 English-German (En-De) and English-Russian (En-Ru) datasets constaining about 4.5 and 25 million sentence pairs, respectively. We used WMT newstest2013 as our development sets and newstest2014 as our test sets for all the translation tasks. We used byte-pair encoding (Sennrich et al., 2016) with combined source and target vocabulary of 37,000 sub-words for English-German and 40,000 sub-words for English-Russian. We took the average of the last 5 checkpoints (saved at 10,000-iteration intervals) for evaluation, and used a beam search size of 5 and length penalty of 0.6 (Wu et al., 2016).
2https://github.com/tensorflow/tensor2tensor

heading: 4.2 RESULTS
text: Table 1 compares our model variants with the Transformer base and big models for En-De and EnRu translation tasks (both directions). We notice that almost all of our models achieve higher BLEU scores than the Transformer base, showing the effectiveness of our approach.
On the En→De translation task, our best homogeneous model (QUERYK with 3/2/3 head distribution) achieves a BLEU of 26.86, already outperforming the Transformer base by about 0.8 points. When we compare the results of the homogeneous models with those of the heterogeneous, we notice even higher scores for the heterogeneous models; the best heterogeneous model yields a BLEU of 27.15, which is about 1.1 points higher than the Transformer base. Finally, we notice that our interleaved heterogeneous models surpass all aforementioned scores achieving up to 27.4 BLEU and establishing a 1.33 BLEU improvement over the Transformer base. This demonstrates the existence of phrase-to-token and phrase-to-phrase mappings from target (De) to source (En) language.
Likewise, on De→En, our models achieve improvements compared to the Transformer, but the gain is not as high as in En→De. Specifically, homogeneous and heterogeneous attentions perform comparably, giving up to +0.38 BLEU improvements compared to the Transformer base. Our interleaved models show more improvements (up to 30.3 BLEU), outperforming the Transformer by about 0.5 points. This again demonstrates the importance of phrase-level query representation in the target.
Similarly, on the En→Ru translation task, all of our models surpass the Transformer base. Homogeneous models with CONVKV perform slightly better than their QUERYK counterparts. In the heterogeneous family, all of the model variants experimented with achieved higher score than the homogeneous models. The QUERYK heterogeneous model with 1-2-3-4 N-grams achieves the highest performance with 37.39 BLEU, 1.75 points higher than the Transformer base. Interleaved homogeneous models with 1-2 N-grams also perform well on this task with a BLEU of 37.24, which is more than the heterogeneous models with 1-2-3 N-grams, having similar number of parameters.
On the Ru→En task, homogeneous models perform slightly better than the Transformer base, while heterogeneous models excel in performance. In particular, CONVKV heterogeneous model with 1-2-3-4 N-grams achieves 35.91 BLEU, which outperforms the Transformer base by 1.35 BLEU points. However, interleaved attention models do not provide significant improvements on this task. We suspect this is because the morphological richness of Russian is not fully leveraged in the current interleaved models as they are limited to only 1-2 N-grams. We put this issue in our future work.
Homogeneous vs. Heterogeneous. Heterogeneous models generally perform better than their homogeneous counterparts. This shows the effectiveness of relaxing the ‘same n-gram type’ attention constraint in the heterogeneous head, which allows it to attend to all n-gram types simultaneously, avoiding any forced attention to a particular n-gram type. In fact, our experiments show that the models perform worse than the baseline if we remove token-level attentions.
Effect of Higher-Order N-grams. We now analyze the effect of including higher-order n-grams. For homogeneous models, including 3-gram components does not have any significant effect in performance, while it increases the number of parameters. On the other hand, heterogeneous models benefit significantly from 3-gram and 4-gram components. For En→De, 3-gram and 4-gram components offer 0.1 - 0.2 improvements in BLEU. Meanwhile, for En→ Ru and Ru→ En, they yield up to 0.64 BLEU improvements compared to the model using only uni- and bi-grams. This supports the argument that the usefulness of 3-grams and 4-grams depends on the language pair, and our heterogeneous models are more robust in selecting any type of n-grams that are useful.3
Comparison with State-of-the-art. Table 2 places our BLEU scores with the state-of-the-art results reported on the En→De translation task. Note that Vaswani et al. (2017) and others conducted their experiments at a massive scale with 8 GPUs, which we could not replicate due to limited number of GPUs. Therefore, we conducted all experiments using a single GPU, and make comparisons in this ‘low-resource’ setting. There have been many evidences that practical training of the Transformer networks (theirs and ours) is significantly susceptible to the batch size (which increases with the number of GPUs used), and training on a single GPU with lower batch size for sufficiently long does not produce similar results as in 8 or more GPU settings; see (Popel & Bojar, 2018) for details.4
As can be noticed, in our training setting (1 GPU), the Transformer big performs only slightly better than the Transformer base, giving 26.63 BLEU in contrast to the reported 28.40 BLEU with 8 GPUs. We believe, the difference in performance is due to the batch size, for which we could conduct trials only at 2048 tokens instead of 25000, although we trained the big model 10 times longer.
In order to verify if phrasal attention (which adds an additional convolution layer in parallel to the original Transformer layer), does indeed help more than just placing extra Transformer layers, we conducted additional experiments with the Transformer base with 10 layers, and with our heterogeneous model with 4 layers. We notice that our heterogeneous model with 6 layers outperforms the Transformer base with 10 layers and the Transformer big, and with 4 layers, it performs on par, even though it has much less parameters. These differences are even more significant for En→Ru, where the Transformer big underperforms the Transformer base with a BLEU of 34.64 (Table 1). On the other hand, our heterogeneous model achieves 37.39 BLEU (+ 2.75). Therefore, our phrasal attention can achieve similar or higher results with a wider but shallower network that consumes less parameters compared to deep Transformer base or Transformer big. This increases parallelizability.
3It is nontrivial to include higher-order n-grams (i.e., n>2) in the interleaved model, because one needs to mix the sequences into one such that different n-gram types align properly. We leave it to future work.
4Also see the discussion https://github.com/tensorflow/tensor2tensor/issues/444
Phrasal attentions are also more suitable for ‘low-resource’ (GPU) setup, and for complex language pairs like English-Russian, where our methods show larger improvements.
Model Interpretation. To interpret our phrasal attention models, we now discuss how they learn the alignments. Figure 4 shows attention heatmaps for an En→De sample in newstest2014; figure 4a displays the heatmap in layer 3 (mid layer), while figure 4b shows the one in layer 6 (top layer) within a 6-layer Transformer based on our interleaved attention. Each figure shows 4 quadrants representing token-to-token, token-to-phrase, phrase-to-token, phrase-to-phrase attentions, respectively. We can see in this example that phrasal attentions are activated strongly in the mid-layers; particularly, the phrase-to-phrase attention is the most concentrated one. On the other hand, token-to-token attention is activated the most in the top layer. Although the distribution of attentions can be quite different depending on the model initialization, we observed a large portion of the attentions for phrases, as shown in Tables 3 and 4 in the Appendix for two different random seeds.

heading: 5 CONCLUSIONS
text: We have presented novel approaches to incorporating phrasal alignments into the attention mechanism of state-of-the-art neural machine translation models. Our methods assign attentions to all four possible mapping relations between target and source sequences: token-to-token, token-to-phrase, phrase-to-phrase and phrase-to-phrase. While we have applied our attention mechanisms to the Transformer network, they are generic and can be implemented in other architectures. On WMT’14 English-German translation tasks, all of our methods surpass the Transformer base model. Our model with interleaved heterogeneous attention, which tackles all the possible phrasal mappings in a unified framework, achieves improvements of 1.33 BLEU for English-to-German and 0.48 BLEU for German-to-English translation tasks over the baseline model. Likewise for WMT’14 EnglishRussian translation tasks, our models achieve up to 1.75 BLEU for English-to-Russian and 1.35 BLEU for Russian-to-English translation compared to the baseline. We are planning future extensions of our techniques to other tasks, such as summarization and question answering. We also plan to improve our models with a phrase-based decoding procedure.

heading: 6 APPENDIX

heading: 6.1 ATTENTION HEAT MAP OF QUERYK HETEROGENEOUS MODEL

heading: 6.2 ATTENTION STATISTICS FOR INTERLEAVED HETEROGENEOUS MODEL
text: Tables 3 and 4 show the distributions of phrase- and token-based attentions across different layers for two different random seeds. Depending on the initial states, the allocations can be different, but phrasal attentions always play an important role in learning the mapping from source to target.
6.3 ATTENTION HEAT MAP OF QUERYK INTERLEAVED MODEL

Review summaries:
The authors propose to include phrases (contiguous n-grams of wordpieces) in both the self-attention and encoder-decoder attention modules of the Transformer model (Vaswani et al., 2017). In standard multi-head attention, the logits of the attention distribution of each head is computed as the dot-product between query and key representations, which are position-specific. In the phrase-based attention proposed here, a convolution is first computed over the query, key, value sequences before logits are computed as before (a few variants of this scheme are explored). Results show an improvement of up to 1.3 BLEU points compared to the baseline Transformer model. However, the lack of a controlled experiment sheds substantial doubt on the efficiacy of the model (see below).