Year: 2019

Abstract:
The attention mechanism is an indispensable component of any state-of-the-art neural machine translation system. However, existing attention methods are often token-based and ignore the importance of phrasal alignments, which are the backbone of phrase-based statistical machine translation. We propose a novel phrasebased attention method to model n-grams of tokens as the basic attention entities, and design multi-headed phrasal attentions within the Transformer architecture to perform token-to-token and token-to-phrase mappings. Our approach yields improvements in English-German, English-Russian and English-French translation tasks on the standard WMT’14 test set. Furthermore, our phrasal attention method shows improvements on the one-billion-word language modeling benchmark.

text: The attention mechanism is an indispensable component of any state-of-the-art neural machine translation system. However, existing attention methods are often token-based and ignore the importance of phrasal alignments, which are the backbone of phrase-based statistical machine translation. We propose a novel phrasebased attention method to model n-grams of tokens as the basic attention entities, and design multi-headed phrasal attentions within the Transformer architecture to perform token-to-token and token-to-phrase mappings. Our approach yields improvements in English-German, English-Russian and English-French translation tasks on the standard WMT’14 test set. Furthermore, our phrasal attention method shows improvements on the one-billion-word language modeling benchmark.

heading: 1 INTRODUCTION
text: The encoder-decoder neural architectures have established breakthroughs in many natural language processing (NLP) tasks including machine translation (Luong et al., 2015), summarization (See et al., 2017), and parsing (Vinyals et al., 2015). Particularly for machine translation, most state of the art neural machine translation (NMT) models possess attention mechanisms to perform alignments of the target tokens to the source tokens. The attention module therefore plays a role analogous to the word alignment model in Statistical Machine Translation or SMT (Koehn, 2010). In fact, the Transformer network introduced recently by Vaswani et al. (2017) achieves state-of-the-art performance in both speed and BLEU scores (Papineni et al., 2002) by using only attention modules. Furthermore, its proposed self-attention layer extends the application of attention way beyond the scope of machine translation and achieves tremendous success in other NLP tasks such as contextual representation learning (Devlin et al., 2018) and machine reading comprehension (Yu et al., 2018).
On the other hand, phrasal interpretation is an important aspect for many NLP tasks, and forms the basis of Phrase-Based Machine Translation (Koehn, 2010). Phrasal (n-gram based) alignments (Koehn et al., 2003) can model one-to-one, one-to-many, many-to-one, and many-to-many relations between target and source tokens while exploiting local context. They are also robust to noncompositional phrases. Despite the advantages, the concept of explicit phrasal attentions has largely been neglected in neural NLP. In fact, most language generation models produce sentences tokenby-token autoregressively, and tend to use the token-based attention method which is order invariant.
Therefore, the intuition of phrasal alignments is vague in existing systems that solely depend on the underlying neural architectures (recurrent, convolutional, or self-attention) to incorporate contextual information. However, the information aggregation strategies employed by the underlying neural architectures provide context-relevant clues to represent only the current token, and do not explicitly model phrasal alignments. We argue that having an explicit inductive bias for phrases and phrasal alignments is beneficial for neural sequence transduction models to exploit the strong correlation between source and target language phrases. This would also make self-attention more capable of tackling other sequence modeling tasks such as language modeling.
In this paper, we propose a novel n-gram-level attention technique to leverage phrasal alignments in various NLP tasks. The technique is designed to assign attention scores directly to phrases in the source and compute phrase-level attention vectors for the target token. It is then applied in a new attention structure to conduct token-to-token and token-to-phrase mappings.
To show the effectiveness of our approach, we apply our phrase-based attention method to all multihead attention layers of the Transformer network. Our experiments on WMT’14 translation tasks show improvements of up to 0.69, 1.84 and 1.65 BLEU in English-to-German, English-to-Russian and English-to-French translation tasks respectively, compared to the baseline Transformer trained in identical settings. Furthermore, when evaluated on the language modeling task, our phrasal models outperform the Transformer base model by up to 4.6 points in perplexity on the one-billion-word language modeling task. We make our code available at anonymous for research purposes.

heading: 2 BACKGROUND
text: Most NMT models adopt an encoder-decoder framework, where the encoder network first transforms an input sequence of symbols x = (x1, x2, . . . , xn) to a sequence of continuous representations Z = (z1, z2, . . . ,zn). From this, the decoder generates a target sequence of symbols y = (y1, y2, . . . , ym) autoregressively, one element at a time. Recurrent seq2seq models with diverse structures and complexity (Sutskever et al., 2014; Bahdanau et al., 2014; Luong et al., 2015; Wu et al., 2016) were the first to yield state-of-the-art results. Convolutional seq2seq models (Kalchbrenner et al., 2016; Gehring et al., 2017; Kaiser et al., 2018) alleviate the drawback of sequential computation of recurrent models and leverage parallel computation to reduce training time. Wu et al. (2019) recently proposed a light-weight convolution, which shows very promising results.
The Transformer network (Vaswani et al., 2017) structures the encoder and the decoder entirely with stacked self-attentions and cross-attentions (only in the decoder). In particular, it uses a multiheaded, scaled multiplicative attention defined as follows:
Attention(Q,K,V ,Wq,Wk,Wv) = softmax( (QWq)(KWk)
T
√ dk
)(V Wv) (1)
Headi = Attention(Q,K,V ,W iq ,W i k,W i v) for i = 1 . . . h (2)
AttentionOutput(Q,K,V ,W ) = concat(Head1,Head2, . . . ,Headh)W (3)
where Q ∈ IRlq×d, K ∈ IRlk×d, and V ∈ IRlk×d are the matrices with query, key, and value vectors respectively, with d being the number of dimensions; W iq , W i k, W i v are the head-specific weights for query, key, and value vectors respectively, and W is the weight matrix that combines the outputs of the heads. To encode a source sequence, the encoder applies self-attention, where Q, K and V identically contain the same vectors coming from the output of the previous layer. In the decoder, each decoder layer first applies the masked-self-attention over the outputs from the previous layer. The results are then used as queries to compute cross-attentions (encoder-decoder attentions) over the encoder states. For cross-attention, Q comprises of the decoder self-attention states, while K and V comprise of the encoder states. We refer the reader to (Vaswani et al., 2017) for further details. Since the self-attention based encoder (or decoder) considers the input as a fullyconnected directed graph, it can model long-range dependencies by explicitly attending to all tokens, and the maximum path length that signals need to travel is O(1). The non-sequential computation also makes it highly parallelizable to multiple threads.
However, one crucial issue with the attention mechanisms employed in the Transformer as well as other NMT architectures is that they are order invariant locally and globally. That is, changing the order of the vectors in Q, K and V does not change the resulting attention weights and vectors. If this problem is not tackled properly, the model may not learn the sequential characteristics of the sentences. RNN-based models (Bahdanau et al., 2014; Luong et al., 2015) tackle this issue with a recurrent encoder and decoder, CNN-based models like (Gehring et al., 2017) use position embeddings, while the Transformer uses positional encoding. Shaw et al. (2018) further encode relative positions inside the self-attention layers.
Another limitation is that these attention methods attend to tokens, and play a role analogous to word alignment models in traditional SMT. It is, however, well admitted in SMT that phrases are better as translation units than words (Koehn, 2010). Without explicit attention to phrases, a particular attention function has to depend entirely on the token-level scores of a phrase for phrasal alignment, which is not robust or reliable, thus making it difficult for the model to learn the required mappings. For example, the attention heatmaps of the Transformer (Vaswani et al., 2017) show a concentration
of the scores on individual tokens even if it uses multiple heads concurrently in multiple layers. Our main hypothesis is that in order to exploit the strong correlation between source and target phrases, the NMT models should have explicit inductive biases for phrases.
There exists some research on phrase-based decoding in the NMT framework. For example, Huang et al. (2018) proposed a phrase-based decoding approach based on a soft reordering layer and a Sleep-WAke Network (SWAN), a segmentation-based sequence model proposed by Wang et al. (2017a). Their decoder uses a recurrent architecture without any attention on the source. Tang et al. (2016) and Wang et al. (2017b) used an external phrase memory to decode phrases for Chineseto-English translation. In addition, hybrid beam search with phrase translation features from the statistical phrase table of a phrase-based SMT system was used to perform phrasal translation in (Dahlmann et al., 2017). Nevertheless, to the best of our knowledge, our work is the first to embed phrases into attention modules, which then propagate the information through the entire end-to-end Transformer network, including the encoder, decoder, and the cross-attention.

heading: 3 PHRASAL ATTENTION MODEL
text: In this section, we present our phrase-based attention model. In Subsection 3.1, we first describe our proposed module to compute attention weights and attention vectors based on n-grams of queries, keys, and values. Then, in Subsection 3.2, we describe how we use this module to compute different kinds of phrasal attentions. We describe our methods in the context of the Transformer (Vaswani et al., 2017). However, it is straight-forward to adopt them into other architectures such as the RNNor CNN-based seq2seq models.
Key Operations and Notations. The core element in our method is a temporal (or onedimensional) convolutional operation that is applied to a sequence of vectors representing the tokens in a sequence. Formally, we define the convolutional operator applied to each token xi with the corresponding vector representation xi ∈ IRd as:
ai,j = w T j (⊕n−1k=0xi+k) (4)
where n is the convolution window size, wj ∈ IRnd is the kernel weight taken from the weight matrix W ∈ IRn×d×d, and ⊕ denotes a vector concatenation that produces a vector in IRnd. The convolution over the entire sequence X ∈ IRl×d of length l gives a feature map aj . By repeating this process with d different weight vectors from W , we get a d-dimensional representation for each token xi. In the rest of the paper, we will use the following notations.
• Convn(X,W ) to denote a convolution operation with window size n and kernel weights W ∈ IRn×d×d over an input sequence X ∈ IRl×d, based on the definition in Equation 4. The result of this convolution is a matrix in IR(l−n+1)×d, whose columns represent the feature maps corresponding to the kernels, and rows represent feature representations corresponding to the tokens in the sequence.
• SConvn(X,Y) to denote a serial convolution operation with window size n over an input X ∈ IRl×d for all the rows Wi ∈ IRn×d×1 in the kernel tensor Y = (W1, . . . ,Wt) ∈ IRt×n×d×1. In other words, SConvn(X,Y) = ⊕t−1i=0Convn(X,Wi) that produces a matrix in IR t×(l−n+1).
• ρn(x) to denote a reshape1 operation on the vector x ∈ IRnd to reshape it to a matrix X ∈ IRn×d×1, and ψn(X) to denote a serial reshape operation to reshape the matrix X ∈ IRt×(nd) to a tensor X ∈ IRt×n×d×1. • softmax(A) to denote a softmax operation over each row of the matrix A. Formally, for A′ = softmax(A), each entry A′(i, j) is computed as: A′(i, j) = exp(A(i,j))∑
j exp(A(i,j)) .
1Tensorflow: tf.reshape(); Pytorch: .view()
3.1 n-GRAM-LEVEL ATTENTION
Our n-gram based attention method computes the attention weights and vectors based on n-gram representations of the tokens. In particular, when computing the attention scores ai ∈ IRlk for each query qi in Q = (q1, . . . , qlq ) ∈ IR
lq×d, we use the query vector qi as the kernel in the convolution operation applied to the set of keys K = (k1, . . . ,klk) ∈ IR lk×d. More formally,
ai = Convn(KWk, ρn(qTi Wq))√
d ∗ n (5)
where Wk ∈ IRd×d and Wq ∈ IRd×(nd) are trainable weights (head weights) to linearly transform K and Q respectively, and ρn is the required reshaping operation to make the dimensions compatible. Applying Equation 5 for all the queries in Q gives the score matrix A = (a1, . . . ,alq ) ∈ IRlq×lk , where each row vector ai ∈ A contains the attention scores for the query vector qi ∈ Q. Next, we compute the attention (or context) vectors as follows.
C = softmax(A) Convn(V ,Wv) (6)
where Wv ∈ IRn×d×d are the kernel parameters to achieve the n-gram representations by convolving over the value vectors V = (v1, . . . ,vlk) ∈ IR
lk×d, which are in turn linearly combined using the attention weights computed by the softmax.
By using the queries as the kernel parameters, we allow the queries to dynamically (kernels vary for different queries) and directly interact with the window of key vectors and compute the n-gram based attention scores. An alternative way to achieve phrasal attentions would be to use A = (QWq) Convn(K,Wk)T√
d to compute the attention scores in Equation 5. In contrast to our approach,
this method is static and indirect in the sense that the convolution uses a static kernel and the queries do not interact directly with the keys; instead the model relies on the kernel weights (Wk) to learn n-gram patterns. In our initial experiments, we also found the direct approach to perform better.

heading: 3.2 MULTI-HEADED PHRASAL ATTENTION
text: Having presented our core method to perform attentions based on n-grams, we now introduce our novel extension to the multi-headed attention framework of the Transformer to enable it to pay attention not only to tokens but also to other n-grams across many sub-spaces and locations. In particular, each head uses the token (unigram) representation of the query to attend to all n-gram types (e.g., n = 1, 2, . . . , N ) simultaneously. To achieve this, we first use the query vectors to compute the attention scores for each n-gram type separately by performing convolution over the key vectors with the respective window sizes. All the n-gram scores are then concatenated before passing them through a softmax to compute the attention weights over all n-grams. Similarly, the value vectors for the n-gram types are concatenated to produce the overall attention output. Figure 1 exemplifies the process for attentions over unigrams and bigrams. For self-attention, the query vectors in Q represent the unigrams (‘India’, ‘and’, ‘Japan’) of the input sequence, whereas for cross-attention, they represent the decoder states of the target side tokens. The overall attention output Cn can be formally described by the following sequence of equations:
Q1 = QWq,1; Qn = QWq,n (7)
A1 = Q1(KWk)
T
√ d
(8)
An = SConvn(KWk, ψn(Qn))√
d ∗ n (9)
Sn = softmax([A1;A2; . . . ;An]) (10) Vn = [(V Wv,1);Conv2(V ,Wv,2); . . . ;Convn(V ,Wv,n)] (11) Cn = SnVn (12)
where Wq,n ∈ IRd×(nd) and Wv,n ∈ IRn×d×d are the respective weight matrices for the queries and values, and Wk ∈ IRd×d is a weight matrix for the keys that is shared across all n-gram types. Note that the serial convolution SConvn() and the serial reshape ψn() functions in Equation 9 implies the computation of all attention scores ai ∈ An (i.e., applying Convn and ρn in Equation 5 for each row qi of Q). We can aggregate multiple n-gram types within an attention module (e.g., 1-2-3 grams). In our method, we do not need to pad the input sequences before the convolution operation to ensure identical sequence length. The key/value sequences that are shorter than the window size do not have any valid phrasal component to be attended.

heading: 4 EXPERIMENTS
text: To demonstrate the effectiveness of our phrase-based attention method, we experiment with two different tasks: machine translation (MT) and language modeling (LM). In the following, we present the training settings, experimental results and analysis of our models on these two tasks.

heading: 4.1 TRAINING SETTINGS
text: We replicate most of the training settings from (Vaswani et al., 2017) for our models, to enable a fair comparison with the original Transformer and the Transformer with relative positioning (Shaw et al., 2018). Specifically, we use the Adam optimizer (Kingma & Ba, 2014) with β1 = 0.9, β2 = 0.98, and = 10−9. We follow a similar learning rate schedule with a warmup steps of 16000 updates: LearningRate = 2× d−0.5 ×min(step num−0.5, step num× warmup steps−1.5). Similar to Vaswani et al. (2017), we also applied residual dropout with 0.1 probability and label smoothing with ls = 0.1. Our models are implemented in the tensor2tensor2 library (Vaswani et al., 2018), on top of the original Transformer codebase.
We conducted all the experiments with our models and the original Transformer in an identical setup for a fair comparison. While Vaswani et al. (2017) trained their base and big models at a massive scale with 8 GPUs, we could only train our models and the baselines on a single GPU because of limited GPU facilities. However, we could (virtually) replicate the 8-GPU setup with a single GPU following the gradient aggregation method proposed recently by Ott et al. (2018).
Setup for MT Experiments. To compare our models with state-of-the-art models, we train all the models with the identical 8-GPU (by gradient aggregation) setup on WMT’16 English-German (En-De), WMT’17 English-Russian (En-Ru) and WMT’14 English-French (En-Fr) datasets. The training datasets contain about 4.5, 25, and 35 million sentence pairs for En-De, En-Ru, and En-Fr, respectively. The effective batches were formed by sentence pairs containing approximately 32,768 source and 32,768 target tokens.
All translation tasks are evaluated in case-sensitive tokenized BLEU. For validation (development) purposes, we use newstest2013 for En-De, newstest2016 for En-Ru, and a random split from the
2https://github.com/tensorflow/tensor2tensor
training set for En-Fr. We evaluate all our models on WMT’14 translation tasks (newstest2014 test sets). We use Byte-Pair Encoding or BPE (Sennrich et al., 2016) with a combined (source and target) vocabulary of 37,000 subwords for En-De, 40,000 subwords for En-Ru and En-Fr. We take the average of the last 5 checkpoints (saved at 5000-update intervals) for evaluation, and use a beam search size of 5 and length penalty of 0.6 (Wu et al., 2016).
Setup for LM Experiments. For our LM experiments, we use the One Billion Word Benchmark dataset (Chelba et al., 2013), which contains 768 million words of data compiled from WMT 2011 News Crawl data,3 with a vocabulary of 32,000 words. We use its held-out data as the test set. We train the base-size models (monolingual decoders) on virtually 4 GPUs for 100,000 updates by using the gradient aggregation technique on a single GPU. The effective batch size is 16,384 tokens.

heading: 4.2 MACHINE TRANSLATION RESULTS
text: Comparison with State-of-the-art. In Table 1, we compare our models on the En→De, En→Ru and En→Fr translation tasks with the base and big size Transformer models (Vaswani et al., 2017), and the base size Transformer with relative position model (Shaw et al., 2018). All the models are trained in the identical 8-GPU setup using gradient aggregation for 100K updates for En→De, 120K updates for En→Ru, and 150K updates for En→Fr. Generally, we can see that our models outperform the base models. Some of them also surpass the Transformer big model trained in identical settings, while having only half the number of parameters.
More specifically, on the En→De translation task, our base-size model with 1-2 grams achieves a BLEU of 27.70, exceeding the Transformer base by 0.6 BLEU. Including higher order n-grams contributes further improvements, reaching up to 27.85 BLEU for 1-2-3 grams. It also performs on par with the big Transformer, while requiring less than half of the number of parameters.
Likewise, on the En→Ru translation task, our models boost the performances significantly over the Transformer base. They achieve 36.21 BLEU with 1-2-3 grams, which is 1.84 points higher than that of the Transformer base. They also outperform the Transformer big by 0.67 points.
Similar trends can also be seen in the En→Fr translation task, where our model with 1-2-3 grams scores at 40.86 BLEU, exceeding the Transformer base by 1.65 points. It also outdoes the Transformer big by 0.2, even though it has considerably fewer parameters. These results demonstrate the effectiveness of our approach over the existing methods.
Effect of Higher-Order n-grams and Training Batch Size. To analyze our models further with respect to the effect of higher-order n-grams and the impact of training batch size, we conducted another set of experiments on En-De and En-Ru translation tasks (both directions) in a single-GPU setup, and compared with the Transformer base and big models on the same setup. The effective
3http://www.statmt.org/lm-benchmark/
batch size in this experiment was 4096 tokens. En-De models were trained for 500K updates while En-Ru ones were trained for 900K updates. The results are shown in Table 2.
It is evident that almost all of our models achieve higher BLEU scores than the Transformer base and big models. For the En-De pair, our model with 1-2 grams achieves 27.11 and 30.16 BLEU for En→De and De→En translation tasks, giving improvements of 0.8 and 0.4 points, respectively. Including higher-order n-grams does improve the performance (for 1-2-3 grams), but diminishing patterns are also observed (for 1-2-3-4 grams) in these tasks.
On the other hand, for the En-Ru pair, the models with higher-order n-grams excel in BLEU performances. Specifically, with 1-2-3-4 grams it yields up to 34.90 and 33.70 BLEU for En→Ru and Ru→En, surpassing the baselines by 1.78 and 0.83 points, respectively. If we compare the corresponding translation results for En→De and En→Ru in Table 1 vs. Table 2, we notice a significant drop in BLEU for the 1-GPU setup across all the models, though the 1- GPU models were offset by longer training (more updates). The performance of the Transformer big is outstandingly low in the 1-GPU setup. There has been evidence that practical training of the Transformer (theirs and ours) is significantly susceptible to the batch size (which increases with the number of GPUs used), and training on a single GPU with a lower batch size for sufficiently long does not produce similar results as with 8 or more GPU settings; please see Popel & Bojar (2018); Ott et al. (2018) and the discussions4 for details on this issue.

heading: 4.3 LANGUAGE MODELING RESULTS
text: Table 3 presents the results on the language modeling task, where we compare our model with uni- and bi-gram attentions against the three baseline models in terms of perplexity. Our model outperforms the Transformer base by 4.6 perplexity points. It also surpasses the Transformer with relative positional encoding (Shaw et al., 2018) by a comparable margin. These results demonstrate that our models are superior to the baselines in the language modeling task, as well.

heading: 4.4 MODEL INTERPRETATION
text: To interpret our phrasal attention models, we now discuss how they learn the alignments. Figure 2 and 3 show attention heatmaps for an En→De sample in newstest2014; Figure 2 displays the heatmap in layer 3 (mid layer), while Figure 3 shows the one in layer 6 (top layer) within a 6-layer
4https://github.com/tensorflow/tensor2tensor/issues/444
Transformer with our phrasal attention. Each figure shows two squares representing token-to-token (left square) and token-to-phrase (right square) attentions, respectively. We can see in the two figures that phrasal (token-to-phrase) attentions are activated strongly in the mid-layers. On the other hand, token-token attentions are activated the most in the top layer, whose final representations are used to predict translated tokens. Although the distribution of attentions can vary depending on model initialization, we observed that 50%-60% of the attentions are concentrated on phrases.

heading: 5 CONCLUSION
text: We have presented novel approaches to incorporating phrasal alignments into the attention mechanism of state-of-the-art sequence transduction models. Our methods assign attentions to both tokens and phrases of the source sequences. While we have applied our attention mechanism to the Transformer network, it is generic and can be implemented in other architectures. We have shown the effectiveness of our approach on two NLP tasks. On machine translation, our models show significant gains on WMT’14 English-German, English-Russian, and English-French translation tasks. Our phrasal models also outperform the Transformer in the one-billion-word language modeling task. We are planning future extensions of our techniques to other tasks, such as summarization and question answering. We also plan to improve our models with a phrase-based decoding procedure.

Review summaries:
This paper proposes an extension of the attention module that explicitly incorporates phrase information. Using convolution, attention scores are obtained independently for each n-gram type, and then combined. Transformer models with the proposed phrase attention are evaluated on multiple translation tasks, as well as on language modelling, generally obtaining better results than by simply increasing model size.

This work aims to incorporate phrase representation into attention mechanism. The proposed method is straightforward (which is good): a convolution window with size n is used to calculate representation for an n-gram, which then replaces the token representation in a standard attention model. The paper implements the multihead version of the proposed phrase-based attention, more specifically, in a transformer model. Experiments with machine translation and language modeling show that it outperforms the token attention counterpart.