PNAT: Non-autoregressive Transformer by Position Learning

1 INTRODUCTION . Transformer ( Vaswani et al. , 2017 ) has been widely used in many text generation tasks , which is first proposed in neural machine translation , achieving great success for its promising performance . Nevertheless , the auto-regressive property of Transformer has been a bottleneck . Specifically , the decoder of Transformer generates words sequentially , and the latter words are conditioned on previous ones in a sentence . Such bottleneck prevents the decoder from higher efficiency in parallel computation , and imposes strong constrains in text generation , with which the generation order has to be left to right ( or right to left ) ( Shaw et al. , 2018 ; Vaswani et al. , 2017 ) . Recently , many researches ( Gu et al. , 2018 ; Lee et al. , 2018 ; Wang et al. , 2019 ; Wei et al. , 2019 ) are devoted to break the auto-regressive bottleneck by introducing non-autoregressive Transformer ( NAT ) for neural machine translation , where the decoder generates all words simultaneously instead of sequentially . Intuitively , NAT abandons feeding previous predicted words into decoder state at the next time step , but directly copy encoded representation at source side to the decoder inputs ( Gu et al. , 2018 ) . However , without the auto-regressive constrain , the search space of the output sentence becomes larger ( Wei et al. , 2019 ) , which brings the performance gap ( Lee et al. , 2018 ) between NAT and auto-regressive Transformer ( AT ) . Related works propose to include some inductive priors or learning techniques to boost the performance of NAT . But most of previous work hardly consider explicitly modeling the position of output words during text generation . We argue that position prediction is an essential problem of NAT . Current NAT approaches do not explicitly model the position of output words , and may ignore the reordering issue in generating output sentences . Compared to machine translation , the reorder problem is much more severe in tasks such as table-to-text ( Liu et al. , 2018 ) and dialog generations ( Shen et al. , 2017 ) . Additionally , it is straightforward to explicitly model word positions in output sentences , as position embeddings are used in Transformer , which is natively non-autoregressive , to include the order information . Intuitively , if output positions are explicitly modeled , the predicted position combined with Transformer to realize non-autoregressive generation would become more natural . In this paper , we propose non-autoregressive transformer by position learning ( PNAT ) . PNAT is simple yet effective , which explicitly models positions of output words as latent variables in the text generation . Specifically , we introduce a heuristic search process to guide the position learning , and max sampling is adopted to inference the latent model . The proposed PNAT is motivated by learning syntax position ( also called syntax distance ) . Shen et al . ( 2018 ) show that syntax position of words in a sentence could be predicted by neural networks in a non-autoregressive fashion , which even obtains top parsing accuracy among strong parser baselines . Given the observations above , we try to directly predict the positions of output words to build a NAT model for text generation . Our proposed PNAT takes following advantages : • We propose PNAT , which first includes positions of output words as latent variables for text generation . Experiments show that PNAT achieves very top results in non-autoregressive NMT , outperforming many strong baselines . PNAT also obtains better results than AT in paraphrase generation task . • Further analysis shows that PNAT has great potentials . With the increase of position prediction accuracy , performances of PNAT could increase significantly . The observations may shed light on the future direction of NAT . • Thanks to the explicitly modeling of position , we could control the generation by facilitating the position latent variable , which may enable interesting applications such as controlling one special word left to another one . We leave this as future work . 2 BACKGROUND . 2.1 AUTOREGRESSIVE DECODING . A target sequence Y =y1 : M is decomposed into a series of conditional probabilities autoregressively , each of which is parameterized using neural networks . This approach has become a de facto standard in language modeling ( Sundermeyer et al. , 2012 ) , and has been also applied to conditional sequence modeling p ( Y |X ) by introducing an additional conditional variable X=x1 : N : p ( Y |X ) = M∏ t=1 p ( yt|y < t , X ; θ ) ( 1 ) With different choices of neural network architectures such as recurrent neural networks ( RNNs ) ( Bahdanau et al. , 2014 ; Cho et al. , 2014 ) , convolutional neural networks ( CNNs ) ( Krizhevsky et al. , 2012 ; Gehring et al. , 2017 ) , as well as self-attention based transformer ( Vaswani et al. , 2017 ) , the autoregressive decoding has achieved great success in tasks such as machine translation ( Bahdanau et al. , 2014 ) , paraphrase generation ( Gupta et al. , 2018 ) , speech recognition ( Graves et al. , 2013 ) , etc . 2.2 NON-AUTOREGRESSIVE DECODING . Autoregressive model suffers from the issue of slow decoding in inference , because tokens are generated sequentially and each of them depends on previous ones . As a solution to this issue , Gu et al . ( 2018 ) proposed Non-Autoregressive Transformer ( denoted as NAT ) for machine translation , breaking the dependency among the target tokens through time by decoding all the tokens simultaneously . Put simply , NAT ( Gu et al. , 2018 ) factorizes the conditional distribution over a target sequence into a series of conditionally independent distributions with respect to time : p ( Y |X ) = pL ( M |X : θ ) · M∏ t=1 p ( yt|X ) ( 2 ) which allows trivially finding the most likely target sequence by arg maxY p ( Y |X ) for each timestep t , effectively bypassing computational overhead and sub-optimality in decoding from an autoregressive model . Although non-autoregressive models achieves 15× speedup in machine translation compared with autoregressive models , it comes at the expense of potential performance degradation ( Gu et al. , 2018 ) . The degradation results from the removal of conditional dependencies within the decoding sentence ( yt depend on y < t ) . Without such dependencies , the decoder is hard to leverage the inherent sentence structure in prediction . 2.3 LATENT VARIABLES FOR NON-AUTOREGRESSIVE DECODING . A non-autoregressive model could be incorporated with conditional dependency as latent variable to alleviate the degradation resulted from the absence of dependency : P ( Y |X ) = ∫ z P ( z|X ) M∏ t=1 P ( yt|z , X ) dz ( 3 ) For example , NAT-FT ( Gu et al. , 2018 ) models the inherent sentence structure with a latent fertility variable , which represents how many target tokens that a source token would translate to . Lee et al . ( 2018 ) introduces L intermediate predictions Y 1 : L as random variables , and to refine the predictions from Y 1 to Y L in a iterative manner . 3 PNAT : POSITION-BASED NON-AUTOREGRESSIVE TRANSFORMER . We propose position-based non-autoregressive transformer ( PNAT ) , an extension to transformer incorporated with non auto-regressive decoding and position learning . 3.1 MODELING POSITION WITH LATENT VARIABLES . Languages are usually inconsistent with each other in word order . Thus reordering is usually required when translating a sentence from a language to another . In NAT family , words representations or encoder states at source side are copied to the target side to feed into decoder as its input . Previously , Gu et al . ( 2018 ) utilizes positional attention which incorporates positional encoding into decoder attention to perform local reordering . But such implicitly reordering mechanism by position attention may cause a repeated generation problem , because position learning module is not optimized directly , and is likely to be misguided by target supervision . To tackle with this problem , we propose to explicitly model the position as a latent variable . We rewrite the target sequence Y with its corresponding position latent variable z = z1 : M as a set Yz = yz1 : zM . The conditional probability P ( Y |X ) is factorized with respect to the position latent variable : P ( Y |X ) = ∑ z∈π ( M ) P ( z|X ) · P ( Y |z , X ) ( 4 ) where π ( M ) is a set consisting of permutations with M elements . At decoding time , the factorization allows us to decode sentences in parallel by pre-predicting the corresponding position variables z . 3.2 MODEL ARCHITECTURE Decoder Inputs Length Predictor Soft-copy Multi-Head Attention Add & Norm x M Positions Multi-Head Attention Add & Norm Input Embedding Figure 1 : The Transformer - model architecture . Decoder : The decoder is also composed of a stack of N = 6 identical layers . In addition to the two sub-layers in each encoder layer , the decoder inserts a third sub-layer , which performs multi-head attention over the output of the encoder stack . Similar to the encoder , we employ residual connections around each of the sub-layers , followed by layer normalization . We also modify the self-attention sub-layer in the decoder stack to prevent positions from attending to subsequent positions . This masking , combined with fact that the output embeddings are offset by one position , ensures that the predictions for position i can depend only on the known outputs at positions less than i . 3.2 Attention An attention function can be described as mapping a query and a set of key-value pairs to an output , where the query , keys , values , and output are all vectors . The output is computed as a weighted sum of the values , where the weight assigned to each value is computed by a compatibility function of the query with the corresponding key . 3.2.1 Scaled Dot-Product Attention We call our particular attention `` Scaled Dot-Product Attention '' ( Figure 2 ) . The input consists of queries and keys of dimension dk , and values of dimension dv . We compute the dot products of the query with all keys , divide each by p dk , and apply a softmax function to obtain the weights on the values . Inputs N x Feed Forward Add & Norm Figure 1 : The Transformer - model architecture . D c der : The decoder is also compo ed of a s ack of N = 6 identical layers . In addition to the two sub-layers in each encoder the decoder inserts a third sub-layer , which performs multi-head attention over the output f the encoder stack . Similar to the encoder , we employ residual connections around each f the sub-layers , foll wed b layer normalization . We also modify the self-attention sub-layer n the decod r stack to prevent positions from attending to subsequent positions . This masking , combined with ac that the utput embeddings are offset by one position , ensures that the predictions for position i c n depend only on the known outputs at positions less than i . 3.2 Attention An attention function can be described s mapping a query and a set of key-value pairs to an output , where the query , keys , values , and output are all v ctors . The output is computed as a weighted sum of he values , where the w ight assigned to each value is computed by a compatibility function of the query with the corresponding key . 3.2.1 Scaled Dot-Product Attention We call our particular attention `` Scaled Dot-Product At ention '' ( Figure 2 ) . The input consists of queri s and keys of dimension dk , and values of dimension dv . We compute the dot products of the query with all keys , divide each by p dk , and apply a oftmax function to obtain the weights on the values . 3 Feed Forward Add & Norm Multi-Head Attention Add & Norm Add & Norm Self-Masked Multi-Head Attention Linear Softmax Output Probabilities x N c Bridge Position Predictor Pointer Predictor Figure 1 : Illustr ti n of the pr posed model , where the black olid arrows represent diff re tiable connections and the dashed arrows are non-differentiable operations . As shown in Figure 1 , PNAT is composed of four modules : an encoder stack , a bridge block , a position predictor as well as a decoder stack . Before detailing each component of PNAT model , we overview the architecture for a brief understanding . 3 Like most sequence-to-sequence models , PNAT first encodes a source sequence X=x1 : N into its contextual word representations E=e1 : N with the encoder stack . With generated contextual word representation E at source side , the bridge block is leveraged to computed the target length M as well as the corresponding features D=d1 : M , which is fed into the decoder as its input . It is worth noting that the decoder inputs D is computed without reordering . Thus the position predictor is introduced to deal with this issue by predicting a permutation z=z1 : M over D. Finally , PNAT generates the target sequence from the decoder input D and its permutation z. Encoder and Decoder Given a source sentence X with length N , PNAT encoder produces its contextual word representations E. The contextual word representations E are further used in computing target length M and decoder initial states D , and are also used as memory of attention at decoder side . Generally , PNAT decoder can be considered as a transformer with a broader vision , because it leverages future word information that is blind to the autoregressive transformer . Intuitively , we use relative position encoding in self-attention ( Shaw et al. , 2018 ) , rather than absolute one that is more likely to cause position errors . Following Shaw et al . ( 2018 ) with a clipping distance d ( usually d ≥ 2 ) set for relative positions , we preserve d = 4 relations . Bridge The bridge module predicts the target length M , and initializes the decoder inputs D from the source representations E. The target length M could be estimated from the source encoder representation : M = N + arg maxφ ( E ) ( 5 ) where φ ( · ) produces a categorical distribution ranged in [ −B , B ] ( B = 20 ) . It is notable that we use the predicted length at inference stage , although during training , we simply use the length of each reference target sequence . Then , we adopt the method proposed by Li et al . ( 2019 ) to compute D. Given the source representation E and the estimated target length M , we linearly combine the embeddings of the neighboring source tokens to generate D as follows : dj = ∑ i wji · ei ( 6 ) wji = softmax ( −|j − i|/τ ) ( 7 ) where wji is a normalized weight that reflects the contribution of ei to dj , and τ is a hyperparameter indicating the sharpness of the weight distribution . Position Predictor For the proposed PNAT , we model position permutations with a position predictor . As shown in Figure 1 , the position predictor takes the decoder inputs D and the source representation E to predict a permutation z . The position predictor has a sub-encoder which stacks multiple layers of encoder units to predict its predicted input R=r1 : M . With the predicted inputsR , we conduct an autoregressive position predictor , denoted as AR-Predictor . The AR-Predictor searches a permutation z with : P ( z|D , E ) = M∏ t=1 p ( zt|z < t , D , E ; θ ) ( 8 ) where θ is the parameter of AR-Predictor , which includes a RNN-based model incorporated with a pointer network ( Vinyals et al. , 2015 ) . To purse the efficiency of decoding , we also explore a non-autoregressive version for the position predictor , denoted as NAR-Predictor , to model the position permutation probabilities with : P ( z|D , E ) = M∏ t=1 p ( zt|D , E ; θ ) ( 9 ) To obtain the permutation z , AR-Predictor performs greedy search whereas NAR-Predictor performs direct arg max . We chose the AR-Predictor as our mainly position module in PNAT , and we also analyze the effectiveness of position modeling in Sec . 4.4 .

Summary: This work proposes a non-autoregressive model for conditioned text generation. The non-autoregressive decoder conditions on a sequence of discrete latent variables, which represent the generation order and can be autoregressively calculated. Instead of doing marginal inference, the paper takes the top 1 generation order that best match inputs. Experiments on machine translation and paraphrase generation show strong result in comparison to other non-autoregressive models.