Monotonic Multihead Attention

1 INTRODUCTION . Simultaneous machine translation adds the capability of a live interpreter to machine translation : a simultaneous model starts generating a translation before it has finished reading the entire source sentence . Such models are useful in any situation where translation needs to be done in real time . For example , simultaneous models can translate live video captions or facilitate conversations between people speaking different languages . In a usual translation model , the encoder first reads the entire sentence , then the decoder writes the target sentence . On the other hand , a simultaneous neural machine translation model alternates between reading the input and writing the output using either a fixed or learned policy . Monotonic attention mechanisms fall into the flexible policy category , in which the policies are automatically learned from data . Recent work exploring monotonic attention variants for simultaneous translation include : hard monotonic attention ( Raffel et al. , 2017 ) , monotonic chunkwise attention ( MoChA ) ( Chiu & Raffel , 2018 ) and monotonic infinite lookback attention ( MILk ) ( Arivazhagan et al. , 2019 ) . MILk in particular has shown better quality/latency trade-offs than fixed policy approaches , such as wait-k ( Ma et al. , 2019 ) or wait-if- * ( Cho & Esipova , 2016 ) policies . MILk also outperforms hard monotonic attention and MoChA ; while the other two monotonic attention mechanisms only consider a fixed window , MILk computes a softmax attention over all previous encoder states , which may be the key to its improved latency-quality tradeoffs . These monotonic attention approaches also provide a closed-form expression for the expected alignment between source and target tokens . However , monotonic attention-based models , including the state-of-the-art MILk , were built on top of RNN-based models . RNN-based models have been outperformed by the recent state-of-the-art Transformer model ( Vaswani et al. , 2017 ) , which features multiple encoder-decoder attention layers and multihead attention at each layer . ∗Work conducted during an internship at Facebook 1The code is available at https : //github.com/pytorch/fairseq/tree/master/ examples/simultaneous_translation We thus propose monotonic multihead attention ( MMA ) , which combines the high translation quality from multilayer multihead attention and low latency from monotonic attention . We propose two variants , Hard MMA ( MMA-H ) and Infinite Lookback MMA ( MMA-IL ) . MMA-H is designed with streaming systems in mind where the attention span must be limited . MMA-IL emphasizes the quality of the translation system . We also propose two novel latency regularization methods . The first encourages the model to be faster by directly minimizing the average latency . The second encourages the attention heads to maintain similar positions , preventing the latency from being dominated by a single or a few heads . The main contributions of this paper are : ( 1 ) A novel monotonic attention mechanism , monotonic multihead attention , which enables the Transformer model to perform online decoding . This model leverages the power of the Transformer and the efficiency of monotonic attention . ( 2 ) Better latency/quality tradeoffs compared to the MILk model , the previous state-of-the-art , on two standard translation benchmarks , IWSLT15 English-Vietnamese ( En-Vi ) and WMT15 German-English ( DeEn ) . ( 3 ) Analyses on how our model is able to control the attention span and on the relationship between the speed of a head and the layer it belongs to . We motivate the design of our model with an ablation study on the number of decoder layers and the number of decoder heads . 2 MONOTONIC MULTIHEAD ATTENTION MODEL . In this section , we review the monotonic attention-based approaches in RNN-based encoder-decoder models . We then introduce the two types of Monotonic Multihead Attention ( MMA ) for Transformer models : MMA-H and MMA-IL . Finally , we introduce strategies to control latency and coverage . 2.1 MONOTONIC ATTENTION . The hard monotonic attention mechanism ( Raffel et al. , 2017 ) was first introduced in order to achieve online linear time decoding for RNN-based encoder-decoder models . We denote the input sequence as x = { x1 , ... , xT } , and the corresponding encoder states as m = { m1 , ... , mT } , with T being the length of the source sequence . The model generates a target sequence y = { y1 , ... , yU } with U being the length of the target sequence . At the i-th decoding step , the decoder only attends to one encoder state mti with ti = j . When generating a new target token yi , the decoder chooses whether to move one step forward or to stay at the current position based on a Bernoulli selection probability pi , j , so that ti ≥ ti−1 . Denoting the decoder state at the i-th position , starting from j = ti−1 , ti−1 + 1 , ti−1 + 2 , ... , this process can be calculated as follows : 2 ei , j = MonotonicEnergy ( si−1 , mj ) ( 1 ) pi , j = Sigmoid ( ei , j ) ( 2 ) zi , j ∼ Bernoulli ( pi , j ) ( 3 ) When zi , j = 1 , we set ti = j and start generating a target token yi ; otherwise , we set ti = j+ 1 and repeat the process . During training , an expected alignment α is introduced to replace the softmax attention . It can be calculated in a recurrent manner , shown in Equation 4 : αi , j = pi , j j∑ k=1 ( αi−1 , k j−1∏ l=k ( 1− pi , l ) ) = pi , j ( ( 1− pi , j−1 ) αi , j−1 pi , j−1 + αi−1 , j ) ( 4 ) Raffel et al . ( 2017 ) also introduce a closed-form parallel solution for the recurrence relation in Equation 5 : αi , : = pi , :cumprod ( 1− pi , : ) cumsum ( αi−1 , : cumprod ( 1− pi , : ) ) ( 5 ) where cumprod ( x ) = [ 1 , x1 , x1x2 , ... , ∏|x|−1 i=1 xi ] and cumsum ( x ) = [ x1 , x1 + x2 , ... , ∑|x| i=1 xi ] . In practice , the denominator in Equation 5 is clamped into a range of [ , 1 ] to avoid numerical instabilities introduced by cumprod . Although this monotonic attention mechanism achieves online 2Note that during training , to encourage discreteness , Raffel et al . ( 2017 ) added a zero mean , unit variance pre-sigmoid noise to ei , j . linear time decoding , the decoder can only attend to one encoder state . This limitation can diminish translation quality as there may be insufficient information for reordering . Moreover , the model lacks a mechanism to adjust latency based on different requirements at decoding time . To address these issues , Chiu & Raffel ( 2018 ) introduce Monotonic Chunkwise Attention ( MoChA ) , which allows the decoder to apply softmax attention to a fixed-length subsequence of encoder states . Alternatively , Arivazhagan et al . ( 2019 ) introduce Monotonic Infinite Lookback Attention ( MILk ) which allows the decoder to access encoder states from the beginning of the source sequence . The expected attention for the MILk model is defined in Equation 6. βi , j = |x|∑ k=j ( αi , k exp ( ui , j ) ∑k l=1 exp ( ui , l ) ) ( 6 ) 2.2 MONOTONIC MULTIHEAD ATTENTION . Previous monotonic attention approaches are based on RNN encoder-decoder models with a single attention and haven ’ t explored the power of the Transformer model . 3 The Transformer architecture ( Vaswani et al. , 2017 ) has recently become the state-of-the-art for machine translation ( Barrault et al. , 2019 ) . An important feature of the Transformer is the use of a separate multihead attention module at each layer . Thus , we propose a new approach , Monotonic Multihead Attention ( MMA ) , which combines the expressive power of multihead attention and the low latency of monotonic attention . Multihead attention allows each decoder layer to have multiple heads , where each head can compute a different attention distribution . Given queries Q , keys K and values V , multihead attention MultiHead ( Q , K , V ) is defined in Equation 7 . MultiHead ( Q , K , V ) = Concat ( head1 , ... , headH ) WO where headh = Attention ( QWQh , KW K h , V W V h , ) ( 7 ) The attention function is the scaled dot-product attention , defined in Equation 8 : Attention ( Q , K , V ) = Softmax ( QKT√ dk ) V ( 8 ) There are three applications of multihead attention in the Transformer model : 1 . The Encoder contains self-attention layers where all of the queries , keys and values come from previous layers . 2 . The Decoder contains self-attention layers that allow each position in the decoder to attend to all positions in the decoder up to and including that position . 3 . The Encoder-Decoder attention contains multihead attention layers where queries come from the previous decoder layer and the keys and values come from the output of the encoder . Every decoder layer has a separate encoder-decoder attention . For MMA , we assign each head to operate as a separate monotonic attention in encoder-decoder attention . For a transformer with L decoder layers and H attention heads per layer , we define the selection process of the h-th head encoder-decoder attention in the l-th decoder layer as el , hi , j = ( mjW K l , h ( si−1W Q l , h ) T √ dk ) i , j ( 9 ) pl , hi , j = Sigmoid ( ei , j ) ( 10 ) zl , hi , j ∼ Bernoulli ( pi , j ) ( 11 ) 3MILk was based on a strengthened RNN-based model called RNMT+ . The original RNMT+ model ( Chen et al. , 2018 ) uses multihead attention , computes attention only once , and then concatenates that single attention layer to the output of each decoder layer block . However , the RNMT+ model used for MILk in Arivazhagan et al . ( 2019 ) only uses a single head . where Wl , h is the input projection matrix , dk is the dimension of the attention head . We make the selection process independent for each head in each layer . We then investigate two types of MMA , MMA-H ( ard ) and MMA-IL ( infinite lookback ) . For MMA-H , we use Equation 4 in order to calculate the expected alignment for each layer each head , given pl , hi , j . For MMA-IL , we calculate the softmax energy for each head as follows : ul , hi , j = SoftEnergy = ( mjŴ K l , h ( si−1Ŵ Q l , h ) T √ dk ) i , j ( 12 ) and then use Equation 6 to calculate the expected attention . Each attention head in MMA-H hardattends to one encoder state . On the other hand , each attention head in MMA-IL can attend to all previous encoder states . Thus , MMA-IL allows the model to leverage more information for translation , but MMA-H may be better suited for streaming systems with stricter efficiency requirements . Finally , our models use unidirectional encoders : the encoder self-attention can only attend to previous states , which is also required for simultaneous translation . At inference time , our decoding strategy is shown in Algorithm 1 . For each l , h , at decoding step i , we apply the sampling processes discussed in subsection 2.1 individually and set the encoder step at tl , hi . Then a hard alignment or partial softmax attention from encoder states , shown in Equation 13 , will be retrieved to feed into the decoder to generate the i-th token . The model will write a new target token only after all the attentions have decided to write . In other words , the heads that have decided to write must wait until the others have finished reading . cli = Concat ( c l,1 i , c l,2 i , ... , c l , H i ) where cl , hi = fcontext ( h , t l , h i ) = mtl , hi MMA-H tl , hi∑ j=1 exp ( ul , hi , j ) ∑tl , hi j=1 exp ( ul , hi , j ) mj MMA-IL ( 13 ) Figure 1 illustrates a comparison between our model and the monotonic model with one attention head . Compared with the monotonic model , the MMA model is able to set attention to different positions so that it can still attend to previous states while reading each new token . Each head can adjust its speed on-the-fly . Some heads read new inputs , while the others can stay in the past to retain the source history information . Even with the hard alignment variant ( MMA-H ) , the model is still able to preserve the history information by setting heads to past states . In contrast , the hard monotonic model , which only has one head , loses the previous information at the attention layer .

Summary: The paper proposes an approach for simultaneous neural machine translation. While prior works deal with recurrent models, the authors adopt previous approaches for Transformer. Specifically, for decoder-encoder attention they introduce Monotonic Multihead Attention (MMA) designed to deal with several attention heads. Each attention head operates as a separate monotonic attention head similar to the ones from previous works by Raffel et al. (2017) and Arivazhagan et al. (2019). MMA has two versions: MMA-H (hard), where each head attends only to one token, and MMA-IL (infinite lookback), where each head attends to all tokens up to a selected position.