Enhancing Attention with Explicit Phrasal Alignments

The attention mechanism is an indispensable component of any state-of-the-art neural machine translation system . However , existing attention methods are often token-based and ignore the importance of phrasal alignments , which are the backbone of phrase-based statistical machine translation . We propose a novel phrasebased attention method to model n-grams of tokens as the basic attention entities , and design multi-headed phrasal attentions within the Transformer architecture to perform token-to-token and token-to-phrase mappings . Our approach yields improvements in English-German , English-Russian and English-French translation tasks on the standard WMT ’ 14 test set . Furthermore , our phrasal attention method shows improvements on the one-billion-word language modeling benchmark . 1 INTRODUCTION . The encoder-decoder neural architectures have established breakthroughs in many natural language processing ( NLP ) tasks including machine translation ( Luong et al. , 2015 ) , summarization ( See et al. , 2017 ) , and parsing ( Vinyals et al. , 2015 ) . Particularly for machine translation , most state of the art neural machine translation ( NMT ) models possess attention mechanisms to perform alignments of the target tokens to the source tokens . The attention module therefore plays a role analogous to the word alignment model in Statistical Machine Translation or SMT ( Koehn , 2010 ) . In fact , the Transformer network introduced recently by Vaswani et al . ( 2017 ) achieves state-of-the-art performance in both speed and BLEU scores ( Papineni et al. , 2002 ) by using only attention modules . Furthermore , its proposed self-attention layer extends the application of attention way beyond the scope of machine translation and achieves tremendous success in other NLP tasks such as contextual representation learning ( Devlin et al. , 2018 ) and machine reading comprehension ( Yu et al. , 2018 ) . On the other hand , phrasal interpretation is an important aspect for many NLP tasks , and forms the basis of Phrase-Based Machine Translation ( Koehn , 2010 ) . Phrasal ( n-gram based ) alignments ( Koehn et al. , 2003 ) can model one-to-one , one-to-many , many-to-one , and many-to-many relations between target and source tokens while exploiting local context . They are also robust to noncompositional phrases . Despite the advantages , the concept of explicit phrasal attentions has largely been neglected in neural NLP . In fact , most language generation models produce sentences tokenby-token autoregressively , and tend to use the token-based attention method which is order invariant . Therefore , the intuition of phrasal alignments is vague in existing systems that solely depend on the underlying neural architectures ( recurrent , convolutional , or self-attention ) to incorporate contextual information . However , the information aggregation strategies employed by the underlying neural architectures provide context-relevant clues to represent only the current token , and do not explicitly model phrasal alignments . We argue that having an explicit inductive bias for phrases and phrasal alignments is beneficial for neural sequence transduction models to exploit the strong correlation between source and target language phrases . This would also make self-attention more capable of tackling other sequence modeling tasks such as language modeling . In this paper , we propose a novel n-gram-level attention technique to leverage phrasal alignments in various NLP tasks . The technique is designed to assign attention scores directly to phrases in the source and compute phrase-level attention vectors for the target token . It is then applied in a new attention structure to conduct token-to-token and token-to-phrase mappings . To show the effectiveness of our approach , we apply our phrase-based attention method to all multihead attention layers of the Transformer network . Our experiments on WMT ’ 14 translation tasks show improvements of up to 0.69 , 1.84 and 1.65 BLEU in English-to-German , English-to-Russian and English-to-French translation tasks respectively , compared to the baseline Transformer trained in identical settings . Furthermore , when evaluated on the language modeling task , our phrasal models outperform the Transformer base model by up to 4.6 points in perplexity on the one-billion-word language modeling task . We make our code available at anonymous for research purposes . 2 BACKGROUND . Most NMT models adopt an encoder-decoder framework , where the encoder network first transforms an input sequence of symbols x = ( x1 , x2 , . . . , xn ) to a sequence of continuous representations Z = ( z1 , z2 , . . . , zn ) . From this , the decoder generates a target sequence of symbols y = ( y1 , y2 , . . . , ym ) autoregressively , one element at a time . Recurrent seq2seq models with diverse structures and complexity ( Sutskever et al. , 2014 ; Bahdanau et al. , 2014 ; Luong et al. , 2015 ; Wu et al. , 2016 ) were the first to yield state-of-the-art results . Convolutional seq2seq models ( Kalchbrenner et al. , 2016 ; Gehring et al. , 2017 ; Kaiser et al. , 2018 ) alleviate the drawback of sequential computation of recurrent models and leverage parallel computation to reduce training time . Wu et al . ( 2019 ) recently proposed a light-weight convolution , which shows very promising results . The Transformer network ( Vaswani et al. , 2017 ) structures the encoder and the decoder entirely with stacked self-attentions and cross-attentions ( only in the decoder ) . In particular , it uses a multiheaded , scaled multiplicative attention defined as follows : Attention ( Q , K , V , Wq , Wk , Wv ) = softmax ( ( QWq ) ( KWk ) T √ dk ) ( V Wv ) ( 1 ) Headi = Attention ( Q , K , V , W iq , W i k , W i v ) for i = 1 . . . h ( 2 ) AttentionOutput ( Q , K , V , W ) = concat ( Head1 , Head2 , . . . , Headh ) W ( 3 ) where Q ∈ IRlq×d , K ∈ IRlk×d , and V ∈ IRlk×d are the matrices with query , key , and value vectors respectively , with d being the number of dimensions ; W iq , W i k , W i v are the head-specific weights for query , key , and value vectors respectively , and W is the weight matrix that combines the outputs of the heads . To encode a source sequence , the encoder applies self-attention , where Q , K and V identically contain the same vectors coming from the output of the previous layer . In the decoder , each decoder layer first applies the masked-self-attention over the outputs from the previous layer . The results are then used as queries to compute cross-attentions ( encoder-decoder attentions ) over the encoder states . For cross-attention , Q comprises of the decoder self-attention states , while K and V comprise of the encoder states . We refer the reader to ( Vaswani et al. , 2017 ) for further details . Since the self-attention based encoder ( or decoder ) considers the input as a fullyconnected directed graph , it can model long-range dependencies by explicitly attending to all tokens , and the maximum path length that signals need to travel is O ( 1 ) . The non-sequential computation also makes it highly parallelizable to multiple threads . However , one crucial issue with the attention mechanisms employed in the Transformer as well as other NMT architectures is that they are order invariant locally and globally . That is , changing the order of the vectors in Q , K and V does not change the resulting attention weights and vectors . If this problem is not tackled properly , the model may not learn the sequential characteristics of the sentences . RNN-based models ( Bahdanau et al. , 2014 ; Luong et al. , 2015 ) tackle this issue with a recurrent encoder and decoder , CNN-based models like ( Gehring et al. , 2017 ) use position embeddings , while the Transformer uses positional encoding . Shaw et al . ( 2018 ) further encode relative positions inside the self-attention layers . Another limitation is that these attention methods attend to tokens , and play a role analogous to word alignment models in traditional SMT . It is , however , well admitted in SMT that phrases are better as translation units than words ( Koehn , 2010 ) . Without explicit attention to phrases , a particular attention function has to depend entirely on the token-level scores of a phrase for phrasal alignment , which is not robust or reliable , thus making it difficult for the model to learn the required mappings . For example , the attention heatmaps of the Transformer ( Vaswani et al. , 2017 ) show a concentration of the scores on individual tokens even if it uses multiple heads concurrently in multiple layers . Our main hypothesis is that in order to exploit the strong correlation between source and target phrases , the NMT models should have explicit inductive biases for phrases . There exists some research on phrase-based decoding in the NMT framework . For example , Huang et al . ( 2018 ) proposed a phrase-based decoding approach based on a soft reordering layer and a Sleep-WAke Network ( SWAN ) , a segmentation-based sequence model proposed by Wang et al . ( 2017a ) . Their decoder uses a recurrent architecture without any attention on the source . Tang et al . ( 2016 ) and Wang et al . ( 2017b ) used an external phrase memory to decode phrases for Chineseto-English translation . In addition , hybrid beam search with phrase translation features from the statistical phrase table of a phrase-based SMT system was used to perform phrasal translation in ( Dahlmann et al. , 2017 ) . Nevertheless , to the best of our knowledge , our work is the first to embed phrases into attention modules , which then propagate the information through the entire end-to-end Transformer network , including the encoder , decoder , and the cross-attention . 3 PHRASAL ATTENTION MODEL . In this section , we present our phrase-based attention model . In Subsection 3.1 , we first describe our proposed module to compute attention weights and attention vectors based on n-grams of queries , keys , and values . Then , in Subsection 3.2 , we describe how we use this module to compute different kinds of phrasal attentions . We describe our methods in the context of the Transformer ( Vaswani et al. , 2017 ) . However , it is straight-forward to adopt them into other architectures such as the RNNor CNN-based seq2seq models . Key Operations and Notations . The core element in our method is a temporal ( or onedimensional ) convolutional operation that is applied to a sequence of vectors representing the tokens in a sequence . Formally , we define the convolutional operator applied to each token xi with the corresponding vector representation xi ∈ IRd as : ai , j = w T j ( ⊕n−1k=0xi+k ) ( 4 ) where n is the convolution window size , wj ∈ IRnd is the kernel weight taken from the weight matrix W ∈ IRn×d×d , and ⊕ denotes a vector concatenation that produces a vector in IRnd . The convolution over the entire sequence X ∈ IRl×d of length l gives a feature map aj . By repeating this process with d different weight vectors from W , we get a d-dimensional representation for each token xi . In the rest of the paper , we will use the following notations . • Convn ( X , W ) to denote a convolution operation with window size n and kernel weights W ∈ IRn×d×d over an input sequence X ∈ IRl×d , based on the definition in Equation 4 . The result of this convolution is a matrix in IR ( l−n+1 ) ×d , whose columns represent the feature maps corresponding to the kernels , and rows represent feature representations corresponding to the tokens in the sequence . • SConvn ( X , Y ) to denote a serial convolution operation with window size n over an input X ∈ IRl×d for all the rows Wi ∈ IRn×d×1 in the kernel tensor Y = ( W1 , . . . , Wt ) ∈ IRt×n×d×1 . In other words , SConvn ( X , Y ) = ⊕t−1i=0Convn ( X , Wi ) that produces a matrix in IR t× ( l−n+1 ) . • ρn ( x ) to denote a reshape1 operation on the vector x ∈ IRnd to reshape it to a matrix X ∈ IRn×d×1 , and ψn ( X ) to denote a serial reshape operation to reshape the matrix X ∈ IRt× ( nd ) to a tensor X ∈ IRt×n×d×1 . • softmax ( A ) to denote a softmax operation over each row of the matrix A . Formally , for A′ = softmax ( A ) , each entry A′ ( i , j ) is computed as : A′ ( i , j ) = exp ( A ( i , j ) ) ∑ j exp ( A ( i , j ) ) . 1Tensorflow : tf.reshape ( ) ; Pytorch : .view ( ) 3.1 n-GRAM-LEVEL ATTENTION Our n-gram based attention method computes the attention weights and vectors based on n-gram representations of the tokens . In particular , when computing the attention scores ai ∈ IRlk for each query qi in Q = ( q1 , . . . , qlq ) ∈ IR lq×d , we use the query vector qi as the kernel in the convolution operation applied to the set of keys K = ( k1 , . . . , klk ) ∈ IR lk×d . More formally , ai = Convn ( KWk , ρn ( qTi Wq ) ) √ d ∗ n ( 5 ) where Wk ∈ IRd×d and Wq ∈ IRd× ( nd ) are trainable weights ( head weights ) to linearly transform K and Q respectively , and ρn is the required reshaping operation to make the dimensions compatible . Applying Equation 5 for all the queries in Q gives the score matrix A = ( a1 , . . . , alq ) ∈ IRlq×lk , where each row vector ai ∈ A contains the attention scores for the query vector qi ∈ Q . Next , we compute the attention ( or context ) vectors as follows . C = softmax ( A ) Convn ( V , Wv ) ( 6 ) where Wv ∈ IRn×d×d are the kernel parameters to achieve the n-gram representations by convolving over the value vectors V = ( v1 , . . . , vlk ) ∈ IR lk×d , which are in turn linearly combined using the attention weights computed by the softmax . By using the queries as the kernel parameters , we allow the queries to dynamically ( kernels vary for different queries ) and directly interact with the window of key vectors and compute the n-gram based attention scores . An alternative way to achieve phrasal attentions would be to use A = ( QWq ) Convn ( K , Wk ) T√ d to compute the attention scores in Equation 5 . In contrast to our approach , this method is static and indirect in the sense that the convolution uses a static kernel and the queries do not interact directly with the keys ; instead the model relies on the kernel weights ( Wk ) to learn n-gram patterns . In our initial experiments , we also found the direct approach to perform better .

Summary: This work aims to incorporate phrase representation into attention mechanism. The proposed method is straightforward (which is good): a convolution window with size n is used to calculate representation for an n-gram, which then replaces the token representation in a standard attention model. The paper implements the multihead version of the proposed phrase-based attention, more specifically, in a transformer model. Experiments with machine translation and language modeling show that it outperforms the token attention counterpart.