Enhancing Attention with Explicit Phrasal Alignments

The attention mechanism is an indispensable component of any state-of-the-art neural machine translation system . However , existing attention methods are often token-based and ignore the importance of phrasal alignments , which are the backbone of phrase-based statistical machine translation . We propose a novel phrasebased attention method to model n-grams of tokens as the basic attention entities , and design multi-headed phrasal attentions within the Transformer architecture to perform token-to-token and token-to-phrase mappings . Our approach yields improvements in English-German , English-Russian and English-French translation tasks on the standard WMT ’ 14 test set . Furthermore , our phrasal attention method shows improvements on the one-billion-word language modeling benchmark . 1 INTRODUCTION . The encoder-decoder neural architectures have established breakthroughs in many natural language processing ( NLP ) tasks including machine translation ( Luong et al. , 2015 ) , summarization ( See et al. , 2017 ) , and parsing ( Vinyals et al. , 2015 ) . Particularly for machine translation , most state of the art neural machine translation ( NMT ) models possess attention mechanisms to perform alignments of the target tokens to the source tokens . The attention module therefore plays a role analogous to the word alignment model in Statistical Machine Translation or SMT ( Koehn , 2010 ) . In fact , the Transformer network introduced recently by Vaswani et al . ( 2017 ) achieves state-of-the-art performance in both speed and BLEU scores ( Papineni et al. , 2002 ) by using only attention modules . Furthermore , its proposed self-attention layer extends the application of attention way beyond the scope of machine translation and achieves tremendous success in other NLP tasks such as contextual representation learning ( Devlin et al. , 2018 ) and machine reading comprehension ( Yu et al. , 2018 ) . On the other hand , phrasal interpretation is an important aspect for many NLP tasks , and forms the basis of Phrase-Based Machine Translation ( Koehn , 2010 ) . Phrasal ( n-gram based ) alignments ( Koehn et al. , 2003 ) can model one-to-one , one-to-many , many-to-one , and many-to-many relations between target and source tokens while exploiting local context . They are also robust to noncompositional phrases . Despite the advantages , the concept of explicit phrasal attentions has largely been neglected in neural NLP . In fact , most language generation models produce sentences tokenby-token autoregressively , and tend to use the token-based attention method which is order invariant . Therefore , the intuition of phrasal alignments is vague in existing systems that solely depend on the underlying neural architectures ( recurrent , convolutional , or self-attention ) to incorporate contextual information . However , the information aggregation strategies employed by the underlying neural architectures provide context-relevant clues to represent only the current token , and do not explicitly model phrasal alignments . We argue that having an explicit inductive bias for phrases and phrasal alignments is beneficial for neural sequence transduction models to exploit the strong correlation between source and target language phrases . This would also make self-attention more capable of tackling other sequence modeling tasks such as language modeling . In this paper , we propose a novel n-gram-level attention technique to leverage phrasal alignments in various NLP tasks . The technique is designed to assign attention scores directly to phrases in the source and compute phrase-level attention vectors for the target token . It is then applied in a new attention structure to conduct token-to-token and token-to-phrase mappings . To show the effectiveness of our approach , we apply our phrase-based attention method to all multihead attention layers of the Transformer network . Our experiments on WMT ’ 14 translation tasks show improvements of up to 0.69 , 1.84 and 1.65 BLEU in English-to-German , English-to-Russian and English-to-French translation tasks respectively , compared to the baseline Transformer trained in identical settings . Furthermore , when evaluated on the language modeling task , our phrasal models outperform the Transformer base model by up to 4.6 points in perplexity on the one-billion-word language modeling task . We make our code available at anonymous for research purposes . 2 BACKGROUND . Most NMT models adopt an encoder-decoder framework , where the encoder network first transforms an input sequence of symbols x = ( x1 , x2 , . . . , xn ) to a sequence of continuous representations Z = ( z1 , z2 , . . . , zn ) . From this , the decoder generates a target sequence of symbols y = ( y1 , y2 , . . . , ym ) autoregressively , one element at a time . Recurrent seq2seq models with diverse structures and complexity ( Sutskever et al. , 2014 ; Bahdanau et al. , 2014 ; Luong et al. , 2015 ; Wu et al. , 2016 ) were the first to yield state-of-the-art results . Convolutional seq2seq models ( Kalchbrenner et al. , 2016 ; Gehring et al. , 2017 ; Kaiser et al. , 2018 ) alleviate the drawback of sequential computation of recurrent models and leverage parallel computation to reduce training time . Wu et al . ( 2019 ) recently proposed a light-weight convolution , which shows very promising results . The Transformer network ( Vaswani et al. , 2017 ) structures the encoder and the decoder entirely with stacked self-attentions and cross-attentions ( only in the decoder ) . In particular , it uses a multiheaded , scaled multiplicative attention defined as follows : Attention ( Q , K , V , Wq , Wk , Wv ) = softmax ( ( QWq ) ( KWk ) T √ dk ) ( V Wv ) ( 1 ) Headi = Attention ( Q , K , V , W iq , W i k , W i v ) for i = 1 . . . h ( 2 ) AttentionOutput ( Q , K , V , W ) = concat ( Head1 , Head2 , . . . , Headh ) W ( 3 ) where Q ∈ IRlq×d , K ∈ IRlk×d , and V ∈ IRlk×d are the matrices with query , key , and value vectors respectively , with d being the number of dimensions ; W iq , W i k , W i v are the head-specific weights for query , key , and value vectors respectively , and W is the weight matrix that combines the outputs of the heads . To encode a source sequence , the encoder applies self-attention , where Q , K and V identically contain the same vectors coming from the output of the previous layer . In the decoder , each decoder layer first applies the masked-self-attention over the outputs from the previous layer . The results are then used as queries to compute cross-attentions ( encoder-decoder attentions ) over the encoder states . For cross-attention , Q comprises of the decoder self-attention states , while K and V comprise of the encoder states . We refer the reader to ( Vaswani et al. , 2017 ) for further details . Since the self-attention based encoder ( or decoder ) considers the input as a fullyconnected directed graph , it can model long-range dependencies by explicitly attending to all tokens , and the maximum path length that signals need to travel is O ( 1 ) . The non-sequential computation also makes it highly parallelizable to multiple threads . However , one crucial issue with the attention mechanisms employed in the Transformer as well as other NMT architectures is that they are order invariant locally and globally . That is , changing the order of the vectors in Q , K and V does not change the resulting attention weights and vectors . If this problem is not tackled properly , the model may not learn the sequential characteristics of the sentences . RNN-based models ( Bahdanau et al. , 2014 ; Luong et al. , 2015 ) tackle this issue with a recurrent encoder and decoder , CNN-based models like ( Gehring et al. , 2017 ) use position embeddings , while the Transformer uses positional encoding . Shaw et al . ( 2018 ) further encode relative positions inside the self-attention layers . Another limitation is that these attention methods attend to tokens , and play a role analogous to word alignment models in traditional SMT . It is , however , well admitted in SMT that phrases are better as translation units than words ( Koehn , 2010 ) . Without explicit attention to phrases , a particular attention function has to depend entirely on the token-level scores of a phrase for phrasal alignment , which is not robust or reliable , thus making it difficult for the model to learn the required mappings . For example , the attention heatmaps of the Transformer ( Vaswani et al. , 2017 ) show a concentration of the scores on individual tokens even if it uses multiple heads concurrently in multiple layers . Our main hypothesis is that in order to exploit the strong correlation between source and target phrases , the NMT models should have explicit inductive biases for phrases . There exists some research on phrase-based decoding in the NMT framework . For example , Huang et al . ( 2018 ) proposed a phrase-based decoding approach based on a soft reordering layer and a Sleep-WAke Network ( SWAN ) , a segmentation-based sequence model proposed by Wang et al . ( 2017a ) . Their decoder uses a recurrent architecture without any attention on the source . Tang et al . ( 2016 ) and Wang et al . ( 2017b ) used an external phrase memory to decode phrases for Chineseto-English translation . In addition , hybrid beam search with phrase translation features from the statistical phrase table of a phrase-based SMT system was used to perform phrasal translation in ( Dahlmann et al. , 2017 ) . Nevertheless , to the best of our knowledge , our work is the first to embed phrases into attention modules , which then propagate the information through the entire end-to-end Transformer network , including the encoder , decoder , and the cross-attention . 3 PHRASAL ATTENTION MODEL . In this section , we present our phrase-based attention model . In Subsection 3.1 , we first describe our proposed module to compute attention weights and attention vectors based on n-grams of queries , keys , and values . Then , in Subsection 3.2 , we describe how we use this module to compute different kinds of phrasal attentions . We describe our methods in the context of the Transformer ( Vaswani et al. , 2017 ) . However , it is straight-forward to adopt them into other architectures such as the RNNor CNN-based seq2seq models . Key Operations and Notations . The core element in our method is a temporal ( or onedimensional ) convolutional operation that is applied to a sequence of vectors representing the tokens in a sequence . Formally , we define the convolutional operator applied to each token xi with the corresponding vector representation xi ∈ IRd as : ai , j = w T j ( ⊕n−1k=0xi+k ) ( 4 ) where n is the convolution window size , wj ∈ IRnd is the kernel weight taken from the weight matrix W ∈ IRn×d×d , and ⊕ denotes a vector concatenation that produces a vector in IRnd . The convolution over the entire sequence X ∈ IRl×d of length l gives a feature map aj . By repeating this process with d different weight vectors from W , we get a d-dimensional representation for each token xi . In the rest of the paper , we will use the following notations . • Convn ( X , W ) to denote a convolution operation with window size n and kernel weights W ∈ IRn×d×d over an input sequence X ∈ IRl×d , based on the definition in Equation 4 . The result of this convolution is a matrix in IR ( l−n+1 ) ×d , whose columns represent the feature maps corresponding to the kernels , and rows represent feature representations corresponding to the tokens in the sequence . • SConvn ( X , Y ) to denote a serial convolution operation with window size n over an input X ∈ IRl×d for all the rows Wi ∈ IRn×d×1 in the kernel tensor Y = ( W1 , . . . , Wt ) ∈ IRt×n×d×1 . In other words , SConvn ( X , Y ) = ⊕t−1i=0Convn ( X , Wi ) that produces a matrix in IR t× ( l−n+1 ) . • ρn ( x ) to denote a reshape1 operation on the vector x ∈ IRnd to reshape it to a matrix X ∈ IRn×d×1 , and ψn ( X ) to denote a serial reshape operation to reshape the matrix X ∈ IRt× ( nd ) to a tensor X ∈ IRt×n×d×1 . • softmax ( A ) to denote a softmax operation over each row of the matrix A . Formally , for A′ = softmax ( A ) , each entry A′ ( i , j ) is computed as : A′ ( i , j ) = exp ( A ( i , j ) ) ∑ j exp ( A ( i , j ) ) . 1Tensorflow : tf.reshape ( ) ; Pytorch : .view ( ) 3.1 n-GRAM-LEVEL ATTENTION Our n-gram based attention method computes the attention weights and vectors based on n-gram representations of the tokens . In particular , when computing the attention scores ai ∈ IRlk for each query qi in Q = ( q1 , . . . , qlq ) ∈ IR lq×d , we use the query vector qi as the kernel in the convolution operation applied to the set of keys K = ( k1 , . . . , klk ) ∈ IR lk×d . More formally , ai = Convn ( KWk , ρn ( qTi Wq ) ) √ d ∗ n ( 5 ) where Wk ∈ IRd×d and Wq ∈ IRd× ( nd ) are trainable weights ( head weights ) to linearly transform K and Q respectively , and ρn is the required reshaping operation to make the dimensions compatible . Applying Equation 5 for all the queries in Q gives the score matrix A = ( a1 , . . . , alq ) ∈ IRlq×lk , where each row vector ai ∈ A contains the attention scores for the query vector qi ∈ Q . Next , we compute the attention ( or context ) vectors as follows . C = softmax ( A ) Convn ( V , Wv ) ( 6 ) where Wv ∈ IRn×d×d are the kernel parameters to achieve the n-gram representations by convolving over the value vectors V = ( v1 , . . . , vlk ) ∈ IR lk×d , which are in turn linearly combined using the attention weights computed by the softmax . By using the queries as the kernel parameters , we allow the queries to dynamically ( kernels vary for different queries ) and directly interact with the window of key vectors and compute the n-gram based attention scores . An alternative way to achieve phrasal attentions would be to use A = ( QWq ) Convn ( K , Wk ) T√ d to compute the attention scores in Equation 5 . In contrast to our approach , this method is static and indirect in the sense that the convolution uses a static kernel and the queries do not interact directly with the keys ; instead the model relies on the kernel weights ( Wk ) to learn n-gram patterns . In our initial experiments , we also found the direct approach to perform better .

Summary: This paper proposes an extension of the attention module that explicitly incorporates phrase information. Using convolution, attention scores are obtained independently for each n-gram type, and then combined. Transformer models with the proposed phrase attention are evaluated on multiple translation tasks, as well as on language modelling, generally obtaining better results than by simply increasing model size.