PNAT: Non-autoregressive Transformer by Position Learning

1 INTRODUCTION . Transformer ( Vaswani et al. , 2017 ) has been widely used in many text generation tasks , which is first proposed in neural machine translation , achieving great success for its promising performance . Nevertheless , the auto-regressive property of Transformer has been a bottleneck . Specifically , the decoder of Transformer generates words sequentially , and the latter words are conditioned on previous ones in a sentence . Such bottleneck prevents the decoder from higher efficiency in parallel computation , and imposes strong constrains in text generation , with which the generation order has to be left to right ( or right to left ) ( Shaw et al. , 2018 ; Vaswani et al. , 2017 ) . Recently , many researches ( Gu et al. , 2018 ; Lee et al. , 2018 ; Wang et al. , 2019 ; Wei et al. , 2019 ) are devoted to break the auto-regressive bottleneck by introducing non-autoregressive Transformer ( NAT ) for neural machine translation , where the decoder generates all words simultaneously instead of sequentially . Intuitively , NAT abandons feeding previous predicted words into decoder state at the next time step , but directly copy encoded representation at source side to the decoder inputs ( Gu et al. , 2018 ) . However , without the auto-regressive constrain , the search space of the output sentence becomes larger ( Wei et al. , 2019 ) , which brings the performance gap ( Lee et al. , 2018 ) between NAT and auto-regressive Transformer ( AT ) . Related works propose to include some inductive priors or learning techniques to boost the performance of NAT . But most of previous work hardly consider explicitly modeling the position of output words during text generation . We argue that position prediction is an essential problem of NAT . Current NAT approaches do not explicitly model the position of output words , and may ignore the reordering issue in generating output sentences . Compared to machine translation , the reorder problem is much more severe in tasks such as table-to-text ( Liu et al. , 2018 ) and dialog generations ( Shen et al. , 2017 ) . Additionally , it is straightforward to explicitly model word positions in output sentences , as position embeddings are used in Transformer , which is natively non-autoregressive , to include the order information . Intuitively , if output positions are explicitly modeled , the predicted position combined with Transformer to realize non-autoregressive generation would become more natural . In this paper , we propose non-autoregressive transformer by position learning ( PNAT ) . PNAT is simple yet effective , which explicitly models positions of output words as latent variables in the text generation . Specifically , we introduce a heuristic search process to guide the position learning , and max sampling is adopted to inference the latent model . The proposed PNAT is motivated by learning syntax position ( also called syntax distance ) . Shen et al . ( 2018 ) show that syntax position of words in a sentence could be predicted by neural networks in a non-autoregressive fashion , which even obtains top parsing accuracy among strong parser baselines . Given the observations above , we try to directly predict the positions of output words to build a NAT model for text generation . Our proposed PNAT takes following advantages : • We propose PNAT , which first includes positions of output words as latent variables for text generation . Experiments show that PNAT achieves very top results in non-autoregressive NMT , outperforming many strong baselines . PNAT also obtains better results than AT in paraphrase generation task . • Further analysis shows that PNAT has great potentials . With the increase of position prediction accuracy , performances of PNAT could increase significantly . The observations may shed light on the future direction of NAT . • Thanks to the explicitly modeling of position , we could control the generation by facilitating the position latent variable , which may enable interesting applications such as controlling one special word left to another one . We leave this as future work . 2 BACKGROUND . 2.1 AUTOREGRESSIVE DECODING . A target sequence Y =y1 : M is decomposed into a series of conditional probabilities autoregressively , each of which is parameterized using neural networks . This approach has become a de facto standard in language modeling ( Sundermeyer et al. , 2012 ) , and has been also applied to conditional sequence modeling p ( Y |X ) by introducing an additional conditional variable X=x1 : N : p ( Y |X ) = M∏ t=1 p ( yt|y < t , X ; θ ) ( 1 ) With different choices of neural network architectures such as recurrent neural networks ( RNNs ) ( Bahdanau et al. , 2014 ; Cho et al. , 2014 ) , convolutional neural networks ( CNNs ) ( Krizhevsky et al. , 2012 ; Gehring et al. , 2017 ) , as well as self-attention based transformer ( Vaswani et al. , 2017 ) , the autoregressive decoding has achieved great success in tasks such as machine translation ( Bahdanau et al. , 2014 ) , paraphrase generation ( Gupta et al. , 2018 ) , speech recognition ( Graves et al. , 2013 ) , etc . 2.2 NON-AUTOREGRESSIVE DECODING . Autoregressive model suffers from the issue of slow decoding in inference , because tokens are generated sequentially and each of them depends on previous ones . As a solution to this issue , Gu et al . ( 2018 ) proposed Non-Autoregressive Transformer ( denoted as NAT ) for machine translation , breaking the dependency among the target tokens through time by decoding all the tokens simultaneously . Put simply , NAT ( Gu et al. , 2018 ) factorizes the conditional distribution over a target sequence into a series of conditionally independent distributions with respect to time : p ( Y |X ) = pL ( M |X : θ ) · M∏ t=1 p ( yt|X ) ( 2 ) which allows trivially finding the most likely target sequence by arg maxY p ( Y |X ) for each timestep t , effectively bypassing computational overhead and sub-optimality in decoding from an autoregressive model . Although non-autoregressive models achieves 15× speedup in machine translation compared with autoregressive models , it comes at the expense of potential performance degradation ( Gu et al. , 2018 ) . The degradation results from the removal of conditional dependencies within the decoding sentence ( yt depend on y < t ) . Without such dependencies , the decoder is hard to leverage the inherent sentence structure in prediction . 2.3 LATENT VARIABLES FOR NON-AUTOREGRESSIVE DECODING . A non-autoregressive model could be incorporated with conditional dependency as latent variable to alleviate the degradation resulted from the absence of dependency : P ( Y |X ) = ∫ z P ( z|X ) M∏ t=1 P ( yt|z , X ) dz ( 3 ) For example , NAT-FT ( Gu et al. , 2018 ) models the inherent sentence structure with a latent fertility variable , which represents how many target tokens that a source token would translate to . Lee et al . ( 2018 ) introduces L intermediate predictions Y 1 : L as random variables , and to refine the predictions from Y 1 to Y L in a iterative manner . 3 PNAT : POSITION-BASED NON-AUTOREGRESSIVE TRANSFORMER . We propose position-based non-autoregressive transformer ( PNAT ) , an extension to transformer incorporated with non auto-regressive decoding and position learning . 3.1 MODELING POSITION WITH LATENT VARIABLES . Languages are usually inconsistent with each other in word order . Thus reordering is usually required when translating a sentence from a language to another . In NAT family , words representations or encoder states at source side are copied to the target side to feed into decoder as its input . Previously , Gu et al . ( 2018 ) utilizes positional attention which incorporates positional encoding into decoder attention to perform local reordering . But such implicitly reordering mechanism by position attention may cause a repeated generation problem , because position learning module is not optimized directly , and is likely to be misguided by target supervision . To tackle with this problem , we propose to explicitly model the position as a latent variable . We rewrite the target sequence Y with its corresponding position latent variable z = z1 : M as a set Yz = yz1 : zM . The conditional probability P ( Y |X ) is factorized with respect to the position latent variable : P ( Y |X ) = ∑ z∈π ( M ) P ( z|X ) · P ( Y |z , X ) ( 4 ) where π ( M ) is a set consisting of permutations with M elements . At decoding time , the factorization allows us to decode sentences in parallel by pre-predicting the corresponding position variables z . 3.2 MODEL ARCHITECTURE Decoder Inputs Length Predictor Soft-copy Multi-Head Attention Add & Norm x M Positions Multi-Head Attention Add & Norm Input Embedding Figure 1 : The Transformer - model architecture . Decoder : The decoder is also composed of a stack of N = 6 identical layers . In addition to the two sub-layers in each encoder layer , the decoder inserts a third sub-layer , which performs multi-head attention over the output of the encoder stack . Similar to the encoder , we employ residual connections around each of the sub-layers , followed by layer normalization . We also modify the self-attention sub-layer in the decoder stack to prevent positions from attending to subsequent positions . This masking , combined with fact that the output embeddings are offset by one position , ensures that the predictions for position i can depend only on the known outputs at positions less than i . 3.2 Attention An attention function can be described as mapping a query and a set of key-value pairs to an output , where the query , keys , values , and output are all vectors . The output is computed as a weighted sum of the values , where the weight assigned to each value is computed by a compatibility function of the query with the corresponding key . 3.2.1 Scaled Dot-Product Attention We call our particular attention `` Scaled Dot-Product Attention '' ( Figure 2 ) . The input consists of queries and keys of dimension dk , and values of dimension dv . We compute the dot products of the query with all keys , divide each by p dk , and apply a softmax function to obtain the weights on the values . Inputs N x Feed Forward Add & Norm Figure 1 : The Transformer - model architecture . D c der : The decoder is also compo ed of a s ack of N = 6 identical layers . In addition to the two sub-layers in each encoder the decoder inserts a third sub-layer , which performs multi-head attention over the output f the encoder stack . Similar to the encoder , we employ residual connections around each f the sub-layers , foll wed b layer normalization . We also modify the self-attention sub-layer n the decod r stack to prevent positions from attending to subsequent positions . This masking , combined with ac that the utput embeddings are offset by one position , ensures that the predictions for position i c n depend only on the known outputs at positions less than i . 3.2 Attention An attention function can be described s mapping a query and a set of key-value pairs to an output , where the query , keys , values , and output are all v ctors . The output is computed as a weighted sum of he values , where the w ight assigned to each value is computed by a compatibility function of the query with the corresponding key . 3.2.1 Scaled Dot-Product Attention We call our particular attention `` Scaled Dot-Product At ention '' ( Figure 2 ) . The input consists of queri s and keys of dimension dk , and values of dimension dv . We compute the dot products of the query with all keys , divide each by p dk , and apply a oftmax function to obtain the weights on the values . 3 Feed Forward Add & Norm Multi-Head Attention Add & Norm Add & Norm Self-Masked Multi-Head Attention Linear Softmax Output Probabilities x N c Bridge Position Predictor Pointer Predictor Figure 1 : Illustr ti n of the pr posed model , where the black olid arrows represent diff re tiable connections and the dashed arrows are non-differentiable operations . As shown in Figure 1 , PNAT is composed of four modules : an encoder stack , a bridge block , a position predictor as well as a decoder stack . Before detailing each component of PNAT model , we overview the architecture for a brief understanding . 3 Like most sequence-to-sequence models , PNAT first encodes a source sequence X=x1 : N into its contextual word representations E=e1 : N with the encoder stack . With generated contextual word representation E at source side , the bridge block is leveraged to computed the target length M as well as the corresponding features D=d1 : M , which is fed into the decoder as its input . It is worth noting that the decoder inputs D is computed without reordering . Thus the position predictor is introduced to deal with this issue by predicting a permutation z=z1 : M over D. Finally , PNAT generates the target sequence from the decoder input D and its permutation z. Encoder and Decoder Given a source sentence X with length N , PNAT encoder produces its contextual word representations E. The contextual word representations E are further used in computing target length M and decoder initial states D , and are also used as memory of attention at decoder side . Generally , PNAT decoder can be considered as a transformer with a broader vision , because it leverages future word information that is blind to the autoregressive transformer . Intuitively , we use relative position encoding in self-attention ( Shaw et al. , 2018 ) , rather than absolute one that is more likely to cause position errors . Following Shaw et al . ( 2018 ) with a clipping distance d ( usually d ≥ 2 ) set for relative positions , we preserve d = 4 relations . Bridge The bridge module predicts the target length M , and initializes the decoder inputs D from the source representations E. The target length M could be estimated from the source encoder representation : M = N + arg maxφ ( E ) ( 5 ) where φ ( · ) produces a categorical distribution ranged in [ −B , B ] ( B = 20 ) . It is notable that we use the predicted length at inference stage , although during training , we simply use the length of each reference target sequence . Then , we adopt the method proposed by Li et al . ( 2019 ) to compute D. Given the source representation E and the estimated target length M , we linearly combine the embeddings of the neighboring source tokens to generate D as follows : dj = ∑ i wji · ei ( 6 ) wji = softmax ( −|j − i|/τ ) ( 7 ) where wji is a normalized weight that reflects the contribution of ei to dj , and τ is a hyperparameter indicating the sharpness of the weight distribution . Position Predictor For the proposed PNAT , we model position permutations with a position predictor . As shown in Figure 1 , the position predictor takes the decoder inputs D and the source representation E to predict a permutation z . The position predictor has a sub-encoder which stacks multiple layers of encoder units to predict its predicted input R=r1 : M . With the predicted inputsR , we conduct an autoregressive position predictor , denoted as AR-Predictor . The AR-Predictor searches a permutation z with : P ( z|D , E ) = M∏ t=1 p ( zt|z < t , D , E ; θ ) ( 8 ) where θ is the parameter of AR-Predictor , which includes a RNN-based model incorporated with a pointer network ( Vinyals et al. , 2015 ) . To purse the efficiency of decoding , we also explore a non-autoregressive version for the position predictor , denoted as NAR-Predictor , to model the position permutation probabilities with : P ( z|D , E ) = M∏ t=1 p ( zt|D , E ; θ ) ( 9 ) To obtain the permutation z , AR-Predictor performs greedy search whereas NAR-Predictor performs direct arg max . We chose the AR-Predictor as our mainly position module in PNAT , and we also analyze the effectiveness of position modeling in Sec . 4.4 .

Summary: This work builds on the non-autoregressive translation (NAT) by using position as a latent variable. Unlike the work by Gu et. al. 2018, where they assume the output word order to follow the word order of the input sentence, this work explores predicting word order supervision as an additional train signal. It shows that predicting the position of the words improves the performance of the translation and paraphrase task. This paper uses a heuristic that the inputs positions and output positions of the decoder with close by embeddings are more likely to represent the position mapping.