Phrase-Based Attentions

Most state-of-the-art neural machine translation systems , despite being different in architectural skeletons ( e.g. , recurrence , convolutional ) , share an indispensable feature : the Attention . However , most existing attention methods are token-based and ignore the importance of phrasal alignments , the key ingredient for the success of phrase-based statistical machine translation . In this paper , we propose novel phrase-based attention methods to model n-grams of tokens as attention entities . We incorporate our phrase-based attentions into the recently proposed Transformer network , and demonstrate that our approach yields improvements of up to 1.3 and 0.5 BLEU points in English-to-German and German-to-English translation tasks , and 1.75 and 1.35 BLEU points in English-to-Russian and Russian-toEnglish translation tasks on WMT ’ 14 test set using WMT ’ 16 training data . 1 INTRODUCTION . Neural Machine Translation ( NMT ) has established breakthroughs in many different translation tasks , and has quickly become the standard approach to machine translation . NMT offers a simple encoder-decoder architecture that is trained end-to-end . Most NMT models ( except a few like ( Kaiser & Bengio , 2016 ) and ( Huang et al. , 2018 ) ) possess attention mechanisms to perform alignments of the target tokens to the source tokens . The attention module plays a role analogous to the word alignment model in Statistical Machine Translation or SMT ( Koehn , 2010 ) . In fact , the Transformer network introduced recently by Vaswani et al . ( 2017 ) achieves state-of-the-art performance in both speed and BLEU scores ( Papineni et al. , 2002 ) by using only attention modules . On the other hand , phrasal interpretation is an important aspect for many language processing tasks , and forms the basis of Phrase-Based Machine Translation ( Koehn , 2010 ) . Phrasal alignments ( Koehn et al. , 2003 ) can model one-to-one , one-to-many , many-to-one , and many-to-many relations between source and target tokens , and use local context for translation . They are also robust to non-compositional phrases . Despite the advantages , the concept of phrasal attentions has largely been neglected in NMT , as most NMT models generate translations token-by-token autoregressively , and use the token-based attention method which is order invariant . Therefore , the intuition of phrase-based translation is vague in existing NMT systems that solely depend on the underlying neural architectures ( recurrent , convolutional , or self-attention ) to incorporate contextual information . However , the information aggregation strategies employed by the underlying neural architectures provide context-relevant clues only to represent the current token , and do not explicitly model phrasal alignments . We argue that having an explicit inductive bias for phrases and phrasal alignments is necessary for NMT to exploit the strong correlation between source and target phrases . In this paper , we propose phrase-based attention methods for phrase-level alignments in NMT . Specifically , we propose two novel phrase-based attentions , namely CONVKV and QUERYK , designed to assign attention scores directly to phrases in the source and compute phrase-level attention vector for the target . We also introduce three new attention structures , which apply these methods to conduct phrasal alignments . Our homogeneous and heterogeneous attention structures perform token-to-token and token-to-phrase mappings , while the interleaved heterogeneous attention structure models all token-to-token , token-to-phrase , phrase-to-token , and phrase-to-phrase alignments . To show the effectiveness of our approach , we apply our phrase-based attention methods to all multi-head attention layers of the Transformer . Our experiments on WMT ’ 14 translation tasks show improvements of up to 1.3 and 0.5 BLEU points for English-to-German and German-to-English respectively , and up to 1.75 and 1.35 BLEU points for English-to-Russian and Russian-to-English respectively , compared to the baseline Transformer network trained in identical settings . 2 BACKGROUND . Most NMT models adopt an encoder-decoder framework , where the encoder network first transforms an input sequence of symbols x = ( x1 , x2 , . . . , xn ) to a sequence of continuous representations Z = ( z1 , z2 , . . . , zn ) , from which the decoder generates a target sequence of symbols y = ( y1 , y2 , . . . , ym ) autoregressively , one element at a time . Recurrent seq2seq models with diverse structures and complexity ( Sutskever et al. , 2014 ; Bahdanau et al. , 2015 ; Luong et al. , 2015 ; Wu et al. , 2016 ) are the first to yield state-of-the-art results . Convolutional seq2seq models ( Kalchbrenner et al. , 2016 ; Gehring et al. , 2017 ; Kaiser et al. , 2018 ) alleviate the drawback of sequential computation of recurrent models and leverage parallel computation to reduce training time . The recently proposed Transformer network ( Vaswani et al. , 2017 ) structures the encoder and the decoder entirely with stacked self-attentions and cross-attentions ( only in the decoder ) . In particular , it uses a multi-headed , scaled multiplicative attention defined as follows : Attention ( Q , K , V , Wq , Wk , Wv ) = S ( ( QWq ) ( KWk ) T √ dk ) ( V Wv ) ( 1 ) Headi = Attention ( Q , K , V , W iq , W i k , W i v ) for i = 1 . . . h ( 2 ) AttentionOutput ( Q , K , V , W ) = concat ( Head1 , Head2 , . . . , Headh ) W ( 3 ) where S is the softmax function , Q , K , V are the matrices with query , key , and value vectors , respectively , dk is the dimension of the query/key vectors ; W iq , W i k , W i v are the head-specific weights for query , key , and value vectors , respectively ; and W is the weight matrix that combines the outputs of the heads . The attentions in the encoder and decoder are based on self-attention , where all of Q , K and V come from the output of the previous layer . The decoder also has crossattention , where Q comes from the previous decoder layer , and the K-V pairs come from the encoder . We refer readers to ( Vaswani et al. , 2017 ) for further details of the network design . One crucial issue with the attention mechanisms employed in the Transformer network as well as other NMT architectures ( Luong et al. , 2015 ; Gehring et al. , 2017 ) is that they are order invariant locally and globally . That is , changing the order of the vectors in Q , K and V does not change the resulted attention weights and vectors . If this problem is not tackled properly , the model may not learn the sequential characteristics of the data . RNN-based models ( Bahdanau et al. , 2015 ; Luong et al. , 2015 ) tackle this issue with a recurrent encoder and decoder , CNN-based models like ( Gehring et al. , 2017 ) use position embeddings , while the Transformer uses positional encoding . Another limitation is that these attention methods attend to tokens , and play a role analogous to word alignment models in traditional SMT . It is , however , well admitted in SMT that phrases are better than words as translation units ( Koehn , 2010 ) . Without explicit attention to phrases , a particular attention function has to depend entirely on the token-level softmax scores of a phrase for phrasal alignment , which is not robust and reliable , thus making it more difficult for the model to learn the required mappings . For example , the attention heatmaps of the Transformer ( Vaswani et al. , 2017 ) show concentration of the scores on individual tokens even if it uses multiple heads concurrently in multiple layers . Our main hypothesis is that in order to exploit the strong correlation between source and target phrases , the NMT models should have explicit inductive biases towards phrases . There exists some research on phrase-based decoding in NMT framework . For example , Huang et al . ( 2018 ) proposed a phrase-based decoding approach based on a soft reordering layer and a Sleep-WAke Network ( SWAN ) , a segmentation-based sequence model proposed by Wang et al . ( 2017a ) . Their decoder uses a recurrent architecture without any attention on the source . Tang et al . ( 2016 ) and Wang et al . ( 2017b ) used an external phrase memory to decode phrases for a Chinese-toEnglish translation task . In addition , hybrid search and PBMT were introduced to perform phrasal translation in ( Dahlmann et al. , 2017 ) . Nevertheless , to the best of our knowledge , our work is the first to embed phrases into attention modules , which thus propagate the information throughout the entire end-to-end Transformer network , including the encoder , decoder , and the cross-attention . 3 MULTI-HEAD PHRASE-BASED ATTENTION . In this section , we present our proposed methods to compute attention weights and vectors based on n-grams of queries , keys , and values . We compare and discuss the pros and cons of these methods . For simplicity , we describe them in the context of the Transformer network ; however , it is straightforward to apply them to other architectures such as RNN-based or CNN-based seq2seq models . 3.1 PHRASE-BASED ATTENTION METHODS . In this subsection , we present two novel methods to achieve phrasal attention . In Subsection 3.2 , we present our methods for combining different types of n-gram attentions . The key element in our methods is a temporal ( or one-dimensional ) convolutional operation that is applied to a sequence of vectors representing tokens . Formally , we can define the convolutional operator applied to each token xt with corresponding vector representation xt ∈ IRd1 as : ot = w ⊕nk=0 xt±k ( 4 ) where ⊕ denotes vector concatenation , w ∈ IRn×d1 is the weight vector ( a.k.a . kernel ) , and n is the window size . We repeat this process with d2 different weight vectors to get a d2-dimensional latent representation for each token xt . We will use the notation Convn ( X , W ) to denote the convolution operation over an input sequence X with window size n and kernel weights W ∈ IRn×d1×d2 . 3.1.1 KEY-VALUE CONVOLUTION . The intuition behind key-value convolution technique is to use trainable kernel parameters Wk and Wv to compute the latent representation of n-gram sequences using convolution operation over key and value vectors . The attention function with key-value convolution is defined as : CONVKV ( Q , K , V ) = S ( ( QWq ) Convn ( K , Wk ) T √ dk ) Convn ( V , Wv ) ( 5 ) where S is the softmax function , Wq ∈ IRdq×dk , Wk ∈ IRn×dk×dk , Wv ∈ IRn×dv×dv are the respective kernel weights for Q , K and V . Throughout this paper , we will use S to denote the softmax function . Note that in this convolution , the key and value sequences are left zero-padded so that the sequence length is preserved after the convolution ( i.e. , one latent representation per token ) . The CONVKV method can be interpreted as indirect query-key attention , in contrast to the direct query-key approach to be described next . This means that the queries do not interact directly with the keys to learn the attention weights ; instead the model relies on the kernel weights ( Wk ) to learn n-gram patterns .

Summary: The authors propose to include phrases (contiguous n-grams of wordpieces) in both the self-attention and encoder-decoder attention modules of the Transformer model (Vaswani et al., 2017). In standard multi-head attention, the logits of the attention distribution of each head is computed as the dot-product between query and key representations, which are position-specific. In the phrase-based attention proposed here, a convolution is first computed over the query, key, value sequences before logits are computed as before (a few variants of this scheme are explored). Results show an improvement of up to 1.3 BLEU points compared to the baseline Transformer model. However, the lack of a controlled experiment sheds substantial doubt on the efficiacy of the model (see below).