PNAT: Non-autoregressive Transformer by Position Learning
Authors: Yu Bao
Hao Zhou
Jiangtao Feng
Mingxuan Wang
Shujian Huang
Jiajun Chen
Lei Li
Venue:  (2020)
Keywords: Text Generation

Abstract: Non-autoregressive generation is a new paradigm for text generation. Previous work hardly considers to explicitly model the positions of generated words. However, position modeling of output words is an essential problem in non-autoregressive text generation. In this paper, we propose PNAT, which explicitly models positions of output words as latent variables in text generation. The proposed PNATis simple yet effective. Experimental results show that PNATgives very promising results in machine translation and paraphrase generation tasks, outperforming many strong baselines.

Under review as a conference paper at ICLR 2020
PNAT: NON-AUTOREGRESSIVE TRANSFORMER BY POSITION LEARNING
Anonymous authors
Paper under double-blind review
ABSTRACT
Non-autoregressive models are promising on various text generation tasks, Previ- ous work hardly considers to explicitly model the positions of generated words. However, the position modeling is an essential problem in non non-autoregressive text generation. this study, we propose PNAT, which incorporates positions as a latent variable into the text generative process. Experimental results show that PNAT achieves top results on machine translation and paraphrase generation tasks, outperforming several strong baselines.
INTRODUCTION
Transformer Vaswani et al. 2017 has been widely used in many text generation tasks, which is first proposed in neural machine translation, achieving great success for its promising performance. Nevertheless, the auto-regressive property of Transformer has been a bottleneck. Specifically, the decoder of Transformer generates words sequentially, and the latter words are conditioned on previous ones in a sentence. Such bottleneck prevents the decoder from higher efficiency in parallel computation, and imposes strong constrains in text generation, with which the generation order has to be left to right (or right to left) (Shaw et al. 2018 Vaswani et al. 2017).
Recently, many researches eG 2018 Lee al. 2018 Wang et al. 2019 Wei al. 2019 are
devoted to break the auto-regressive bottleneck by introducing non non- -ar-ourrrrreesiveeeeeeee Transformer (NAT) for neural machine translation, where the decoder generates all words simultaneously instead of sequentially. Intuitively, NAT abandons feeding previous predicted words into decoder state at the next time step, but directly copy encoded representation at source side to the decoder inputs (Gu et al., 2018). However, without the auto-regressive constrain, the search space of the output sentence becomes larger (Wei et al., 2019), which brings the performance gap (Lee et al.) 2018) between NAT and auto-regressive Transformer (AT). Related works propose to include some inductive priors learning techniques to boost the performance of NAT. But most of previous work hardly consider explicitly modeling the position of output words during text generation.
We argue that position prediction is an essential problem of NAT. Current NAT approaches do not explicitly model the position of output words, and may ignore the reordering issue in generating output sentences. Compared to machine translation, the reorder problem is much more severe in tasks such table-to-text (Liu et al. 2018) and dialog generations (Shen al. 2017). Additionally, it is straightforward to explicitly model word positions in output sentences, as position embeddings are used in Transformer, which is natively non-autoregressive, to include the order information. Intuitively, if output positions are explicitly modeled, the predicted position combined with Transformer to realize non-autoregressive generation would become more natural.
In this paper, we propose non-autoregressive transformer by position learning (PNAT). PNAT is simple yet effective, which explicitly models positions of output words as latent variables in the text generation. Specifically, max sampling is adopted to inference the latent model. The proposed PNAT is motivated by learning syntax position (also called syntax distance). Shen et al. (2018) show that syntax position of words in a sentence could be predicted by neural networks in a non- non-autoregressive fashion, which even obtains top parsing accuracy among strong parser baselines. Given the observations above, try to directly predict the positions of output words to build a NAT model for text generation.
introduce a heuristic search process to guide the position learning, and
Our proposed PNAT takes following advantages:

---

Under review as a conference paper at ICLR 2020
We propose PNAT, which first includes positions of output words latent variables for text generation. Experiments show that PNAT achieves very top results in non- autorregrree NMT, outperforming many strong baselines. PNAT also obtains better results than AT in paraphrase generation task.
Further analysis shows that PNAT has great potentials. With the increase of position prediction accuracy. performances of PNAT could increase significantly. The observations may shed light the future direction of NAT.
Thanks to the explicitly modeling of position, we could control the generation by facilitating the position latent variable, which may enable interesting applications such as controlling one special word left to another one. We leave this as future work.
2 BACKGROUND
2.1
AUTOREGRESSIVE DECODING
target sequence Y=tn= is decomposed into a series of conditional probabilities autoregressivell, each of which is parameterized using neural networks. This approach has become de facto standard in language Modeling(Sundermeeye et al., 2012), and has been also applied to conditional sequence modeling pX) by introducing an additional conditional variable X=T1.N:
M
X
II P(ul
(1)
With different choices of neural network architectures such as recurrent neural networks (RNNs) (Bah- danau et al. 20114; Cho et al. 2014), convolutional neural networks (CNNs) (Krizhessky et al. 2012 Gehring et al. 201 7). as well as self-attention based transformer (Vaswani et al. 2017). the autore- gressive decoding has achieved great success in tasks such machine translation (Bahdanau et al.)
2014), paraphrase generation (Gupta et al. 2018), speech recognition (Graves et al. 2013). etc.
2.2
NON-AUTOREGGRESSIVE DECODING
Autoregressive model suffers from the issue of slow decoding inference, because tokens are generated sequentially and each of them depends on previous ones. As a solution to this issue, (uu et al. (2018) proposed con-Auttregreesive Transformer (denoted as NAT) for machine translation, breaking the dependency among the target tokens through time by decoding all the tokens simultaneously. Put simply, NAT (Gu et al.] 2018)) factorizes the conditional distribution over a target sequence into a series of conditionally independent distributions with respect to time:
p(Y[x)= = PL (M : 0) II p()
which allows trivially finding the most likely target sequence by arg maxy p(Y|X) for each timestep t, effectively bypassing computational overhead and sub-optimality in decoding from autoregres- sive model.
Although non-autoregressive models achieves 15x speedup in machine translation compared with autoregressive models, it comes at the expense of potential performance degradation (Gu et al. 2018). The degradation results from the removal of conditional dependencies within the decoding sentence(y), depend on yex). Without such dependencies, the decoder is hard to leverage the inherent sentence structure in prediction.
2.3 LATENT VARIABLES FOR MON--UUTORERESSSVVE DECODING
A aarn-uttreerrssvvveeeeeeeeeee model could be incorporated with conditional dependency as latent variable to alleviate the degradation resulted from the absence of dependency:
M
X = P(z[X) II
P(yl X X)dz
(3)
1=1
2

---

Under review as a conference paper at ICLR 2020
For example, NAT-FT (Gu et al. 2018) models the inherent sentence structure with latent fertility variable, which represents how many target tokens that a source token would translate to. Lee et al. (2018) introduces L intermediate predictions y random variables, and to refine the predictions from to Y in a iterative manner.
PNAT: POSITION-BASED NON-AUTOREGRESSIVE TRANSFORMER
We propose position-basee non-autorrerressive transformer (PNAT), extension to transformer incorporated with non auto-regressive decoding and position learning.
3.1 MODELING POSITION WITH LATENT VARIABLES
Languages are usually inconsistent with each other in word order. Thus reordering is usually required when translating a sentence from language to another. In NAT family, words representations or encoder states source side copied the target side to feed into decoder its input. Previously, Gu et al. (2018) utilizes positional attention which incorporates positional encoding into decoder attention perform local reordering. But such implicitly reordering mechanism by position attention may cause a repeated generation problem, because position learning module is not optimized directly, and is likely to be misguided by target supervision.
To tackle with this problem, we propose to explicitly model the position as latent variable. We rewrite the target sequence Y with its corresponding position latent variable 31:M as set Y2 U5 The conditional probability PXY is factorized with respect to the position latent variable:
P(Y[X)
Σ *ET(M)
where x (M) a consisting of permutations with M elements. At decoding time, the factorization allows us to decode sentences in parallel by pre-predicting the corresponding position variables 2.
3.2 MODEL ARCHITECTURE
Cottut
Sobrre
Addt Narm
Frid Fowwart
And & Taam
Nort
My Head
Aet Now
AN
Heoo
Mustias
You
Docoda
kprrs
Langth Predidd
Soll OCO
Position Predictor
Figure 1: Illustration of the proposed model, where the black solid arrows represent differentiable connections and the dashed arrows are non-differentiable operations.
As shown in Figure PNAT is composed of four modules: encoder stack. bridge block, position predictor as well as a decoder stack. Before detailing each component of PNAT model, we overview the architecture for brief understanding.

---

Under review as a conference paper at ICLR 2020
Like most seuuuence-t-sequeeee models, PNAT first encodes source sequence X=T1:N into its contextual word representations E= with the encoder stack. With generated contextual word representation E source side, the bridge block is leveraged to computed the target length M as well as the corresponding features D=d1:M. which is fed into the decoder as its input. It is worth noting that the decoder inputs D is computed without reordering. Thus the position predictor is introduced to deal with this issue by predicting permutation 2=21:M over D. Finally, PNAT generates the target sequence from the decoder input D and its permutation 2.
Encoder and Decoder Given contextual word representations E. The contextual word representations E are further used in computing target length M and decoder initial states D, and are also used as memory of attention at decoder side.
sentence X with length N, PNAT encoder produces its
Generally, PNAT decoder can be considered a transformer with a broader vision, because it leverages future word information that is blind to the autoregressive transformer. Intuitivelly we al. 2018), rather than absolute one that is
use relative position encoding in salf-attentionnShaae more likely to cause position errors. Following Shaw et al. (2018) with a clipping distance d (usually 2) set for relative positions, we preserve
relations.
Bridge The bridge module predicts the target length M, and initializes the decoder inputs D from the source representations E. The target length M could be estimated from the source encoder representation:
(5) where o(-) produces categorical distribution ranged in [-B, = 20). It is notable that use the predicted length at inference stage, although during training, we simply use the length of each reference target sequence. Then, adopt the method proposed by [i et al. (2019) compute D. Given the source representation E and the estimated target length M, we linearly combine the embeddings of the neighboring source tokens to generate D follows:
M N arg max ((E)
d, Σ Wij Cq
Wyz = softmax (-Ij -ij--11 i1/T)
(7)
where Wys is a normalized weight that reflects the contribution of e; to dj. and r is hyperparameter indicating the sharpness of the weight distribution.
Position Predictor For the proposed PNAT, model position permutations with position predictor. As shown in Figure [ the position predictor takes the decoder inputs D and the source representation E to predict permutation z. The position predictor has sub-encoder which stacks multiple layers of encoder units to predict its predicted input R=T1:M.
With the predicted inputs R, we conduct an autoregressive position predictor, denoted AR- AR-Predictor. The AR- AR-Predictor searches a permutation with:
P(x II Pealess D(E0))
where 0 is the parameter of AR AR-Preccc,or, which includes RNN-based model incorporated with pointer network (Vinyals et al. 2015).
To purse the efficiency of decoding, also explore non-autoregressive version for the position
predictor, denoted as NAR-Predictor, to model the position permutation probabilities with:
P(z I p...D.,E.0
To obtain the permutation z, AR-Predicte performs greedy search whereas NAR- MAR-Predictor performs direct arg max. We chose the AR-Predictor as our mainly position module in PNAT, and we also analyze the effectiveness of position modeling in Sec. 4.4
4

---

Under review as a conference paper at ICLR 2020
3.3 TRAINING
Training requires maximizing the marginalized likelihood in Egn. 4 However, this is intractable since we need enumerate all the M! permutations tokens. We therefore optimize this objective by Monte Carlo sampling method with a heuristic search algorithm.
Heuristic Search for Positions Intuitively, each target token should have corresponding decoder input, and meanwhile each decoder input should be assigned to a target token. Based on this idea, we design a heuristic search algorithm to allocate positions. Given the decoder inputs and its target tokens, we first estimate the similarity between each pair of the decoder input d; and the target token embedding V5- which is also the weights of the target word classifier:
sim = cosine (d,, M,)
(10)
Based on the cosine similarity matrix, HSP is designed to find perfect matching between decoder inputs and target tokens:
HSP(2) arg max
(sima,z;)
(11)
Here we apply greedy algorithm to select the pair with the highest similarity score iteratively until permutation ž is generated. More details are provided Appendix
The intuition behind is that, if the decoder input d; is already the most similar one to a target word, it would be easier to keep and even reinforce this association in learning model. also analyze the effectiveness of the HSP in the Sec.
Objective Function With the heuristically discovered positions reference positions Zief, the position predictor could be trained with a position loss:
Cp
log P( D)
(12)
Grounding on the referenced positions, the generative process of target sequences is optimized by:
Cg Σ log P(Y ( X)
(13)
Finally, combining two loss functions mentioned above, a full-fledged loss derived as
L Lg + aCp
(14)
The length predictor is classifier that follows the previous settings. We also follow the previous practice (Gu et al.] 2018 WWi et al., (019) and perform an extra training process for the length predictor after the model trained and do not tune the parameter of the encoder.
3.4 INFERENCE
We follow the common choice of approximating decoding algorithms (Gu et al. 2018 Lee et al. 2018 to reduce the search space latent variable model.
Argmax Decoding Following Gu et (018)) one simple and effective method to select best sequence by choosing the probabiiiyy latent sequence
arg max P(z|D,E)
Y arg max P(Y|=*,X'
where identifying Y only requires independently maximizing the local probability for each output position.

---

Under review as a conference paper at ICLR 2020
Length Parallel Decoding We also consider the common practice of noisy parallel decoding (Gu et al. 2018), which generates a number decoding candidates in parallel and selects the best via re-scoring using pre-trained autoregressive model. For PNAT, we first predict the target length M, then generate output sequence with argmax decoding for each target length candidate M [M A, M + AM (M 4 in our experiments), which called length parallel decoding (LPD). Then we use the pre-trained autoregressive model to rank these sequences and identify the best overall output as the final output.
4 EXPERIMENTS
We test PNAT on several benchmark sequence generation tasks. We first describe the experimental setting and implementation details and then present the main results, followed by some deep studies.
4.1 EXPERIMENTAL SETTING
To show the generation ability of PNAT, we conduct experiments on the popular machine translation and paraphrase generation tasks. These sequence generation task evaluation models from different perspectives. Translation tasks test the ability of semantic transforming across bilingual corpus. While paraphrase task focuses on substitution between the same languages while keeping the semantics.
Machine Translation We valid the effectiveness of PNAT on the most widely used benchmarks for machine translation WMT14 EN-DE(4.5M pairs) and IWSLT16 DE-EN(I96K pairs). The dataset is processed with Moses script (Koehn et al. 2007) and the words are segmented into subword units using byte-pair encoding (Sennrich et al. 2016 BPE). For both WMT datasets, the source and target languages share the same set of subword embeddings while for IWSLT we use separate embeddings.
Paraphrase Generation We conduct experiments following previous work (Miao et al. 2019) for paraphrase generation. We make use of the established Quora dataset I to evaluate on the paraphrase generation task. We consider the supervised paraphrase generation and split the Quora dataset in the standard setting. We sample 100k pairs sentence as training data, and holds out 3k, 30k for validation and testing, respectively.
4.2 IMPLEMENTATION DETAILS
Module Setting For machine translation, we follow the settings from Gu et al. (018)). In the case of IWSLT task, we use small setting (dmodel = 278, dhiddes 507, Pdropout 0.1, Taayer 5 and Theaad
2) suggested by Gu et al. (2018) 8) for Transformer and NAT models. For WMT task, we use the base setting of the Vaswani et al. (2017) 7) (dmostt 512, dnidden 512, Pdropout = 0.1, Tlaayer 6).
For paraphrase generation, we follow the settings from Miao et al. (2019). and set the 300-dimensional GRU with 2 layer for Seq-to-Seq (GRU). We empirically select Transformer and NAT models with hyperparameters (dmotel = 400, dridden = 800, Pamppout 0. Mayer = 3 and Thend 4).
Optimization We optimize the parameter with the Adam optimizer (Kingma & Ba 2014). The hyperparameter a used in Eqn. 14 was be set to 1.0 for WMT, 0.3 for IWSLT and Quora. We also use inverse square root learning rate scheduling Vaswani et al. 2017) for the WMT, and using linear annealing (from 3e - 4 to le 5, suggested Lee et al. (2018) for the IWSLT and Quora. Each mini-batch consists of approximately 2K tokens for IWSLT and Quora, 32K tokens for WMT.
Knowledge Distillation Sequence-level knowledge distillation is applied to alleviate multi- modality problem while training, using Transformer as teacher (Hinton al. 2015). Previous studies on non-autoregressive generation (Gu et al. 2018 [ee et al. [2018] Wei et al. 2019) have used translations produced by a pre-trained Transformer model as the training data, which significantly improves the performance. We follow this setting in translation tasks.

---

Under review as a conference paper at ICLR 2020
WMT
IWSLT16
Model
Speedup
EN-DE DE-EN
DE-EN
Autoregressive Methods
Transformer-base Vaswann et al.
2017)
27.30
Transformer(Beam=4
27.40
31.33
34.81
1.0x
Notoouttoeeeeesive Methods
Flowseq (Ma et al. 2019) "NAT-base
18.55
23.36
11.02
*PNAT
19.73
24.04
NAT w/ Knowledge Distillation
NAT-FT (Gu et al.
2018)
17.69
21.47
15.6x
LT (Kaiser et al. 2018)
19.80
/
5.8
IR-NAT (Lee et al. 2018
13.91
16.77
27.68
9.0x
ENAT (uo et al. 2019)
20.65
23.02
24.3x
NAT-REG Wang al.
2019
20.65
24.77
imitate-NAT (Wei et al. 2019
22.44
25.67
18.6x
Flowseq (Ma et al. 2019)
21.45
26.16
1.1x
*NAT-base
16.69
13.5x
*PNAT
23.05
27.18
31.23
7.3 x
NAT w/ Reranking or Iterative Refinments
NAT-FT (rescoring 10 candidates)
18.66
22.42
7.7x
LT (rescoring 10 candidates)
22.50
/
IR-NAT (refinement 10)
21.61
25.48
32.31
1.3x
ENAT (rescoring candidates)
24.28
26.10
12.4x
NAT-REG (rescoring 9 candidates)
24.61
28.90
imitate-NAT (rescoring candidates)
24.15
27.28
9.7x
Flowseq (rescoring 30 candidates)
23.48
28.40
*PNT (LPD n= AM=4)
24.48
29.16
32.60
3.7x
Table Performance on the newstest-2014 for WMT14 EN-DE and test2013 for IWSLT EN-DE. denotes same numbers above. **' indicates our implementation. The decoding speed is measured sentence-by-sentenn and the speedup computed by comparing with Transformer.
4.3
MAIN RESULTS
compare the PNAT with strong NAT baselines, including the NAT with fertility (Gu et al. [2018] NAT-FT), the NAT with iterative refinement (Lee et al. 2018 IR-NAT), the
Machine Translation
NAT with regularization (Wang et al.] 2019 NAT-REG), the NAT with enhanced decoder input (Guo)
al. 2019, ENAT), the NAT with learning from auto-regressive model (Wei al.. 2019 imitate-
NAT), the NAT build on latent variables (Kaiser et al. 2018 LT), and the flow-based NAT model (Mal
al.
2019 Flowseq).
The results are shown in Table We basically compare the proposed PNAT against the autoregressive counterpart both in terms of generation quality, which is measured with BLBU (Papineni al. 2002) and inference speedup. For all our tasks, we obtain the performance of competitors by either directly using the performance figures reported in the previous works if they are available or producing them by using the open source implementation of baseline algorithms on our datasets 2 Clearly, PNAT achieves a comparable or better result to previous NAT models on both WMT and TWSLT tasks.
We list the result of the NAT models trained without using knowledge distillation in the second block of the Table []] The PNAT achieves significant improvements (more than 13.0 BLEU points) over the naive baselines, which indicate that position learning greatly contributes to improve the model capability of NAT model. The PNAT also achieves a better result than the Flowseq around 1.0 BLEU, which demonstrates the effectiveness of PNAT in modeling dependencies between the target outputs.
As shown in the third block of the Table | all the competitors with large margin, achieves balance between performance and efficiency. In
without using reranking techniques, the PNAT outperforms
For the sake of fairness, have chosen the hase setting for all competitors.
7

---

Under review as a conference paper at ICLR 2020
particular, the previous state-of-the-arie thth-aarttWMT14 DE-EN) Flowseq achieves good performance with the slow speed(1.1 x), while PNAT goes beyond Flowseq in both respects.
Our best results obtained with length parallel decoding which employ autoregressive model to rerank the multiple generation candidates of different target length. Specifically, on the large scale WMT14 DE-EN task, PNAT (+LPD) surpass the NAT-REG by 0.76 BLEU score. Without reranking, the gap has increased to 2.4 BLEU score (27.18 18 v.s. 24.77). The experiments shows the power of explicitly position modeling which reduces the gap between non-autoregressive and the autoregressive models.
Paraphrase Generation Given a sentence, paraphrase generation aims to synthesize another sentence that is different from the given one, but conveys the same meaning. Comparing with translation task, paraphrase generation prefers a more similar order between source and target sentence, which possibly learn trivial position model. PNAT can potentially yield better results with the position model to infer the relatively ordered alignment relationship.
Paraphrase(BLEU)
Model
Valid
Test
(eQ-)o-seq(GRRU)
24.68
24.75
Transformer
25.88
25.46
NAT-hase
19.80
20.34
PNAT
29.30
29.00
Table 2: Results validation and test of Quora.
The results of the paraphrase generation are shown Table 2 In consist with our intuition, PNAT achieves the best result this task and even surpass Transformer around 3.5 BLEU. The NAT model not powerful enough to capture the latent position relationship. The comparison between NAT-base and PNAT shows that explicit position modeling in PNAT plays crucial role in generating sentences.
4.4 ANALYSIS
Effectiveness of Heuristic Searched Position First, we analyze whether the position derived from the heuristic search suitable for use as supervision the position predictor. We evaluate the effectiveness of the searched position by training PNAT as before and testing with the heuristic searched position instead the predicted position. As shown easier noticed that as PNAT w/ HSP achieves a significant improvement over the NAT-base and the Transformer, which demonstrates that the heuristic search for the position is effective.
second block of the Table
Position Accuracy( (%)
DM
Model
Speed Up
permutation-ace relative-aco(r=4)
BLBU
Transforme(beemm-4)
30.68
1.0x
NAT-base
16.71
13.5x
PNAT HSP
100.00
100.00
46.03
12.5x
PNAT w/ AR-Predictor
25.30"
59.27
27.11
77.x
PNAT w/ CAR--Prdiccoor
23.11
55.57
20.81
11.7x
Table 3: Results on validation set of WMT14 DE-EN with different position strategy. "HSP" means the reference position sequence derived from the heuristic position searching.
Effectiveness and Efficiency of Position Modeling We are also analysis the accuracy of our po- sition modeling and its influence on the quality of generation on the WMT14 DE-EN task. For evaluating the position accuracy, we adopt the heuristic searched position as the position reference (de- noted as "HSP"). which is the training target of the position predictor. PNAT requires the position information at two places. The first is the mutual relative relationship between the states that will be used during decoding. And the second is to reorder the decoded output after decoding. We
8

---

Under review as a conference paper at ICLR 2020
then propose the corresponding metrics for evaluation, which is the relative position accuracy (with relation threshold
4) and the permutation accuracy.
As shown in Table 3 better position accuracy always yields better generation performance. The non-autoregressive position model less effective than the current autoregressive position model, both in the accuracy of the permutation and the relative position. Even though the current PNAT with a simple AR-Predictor has surpassed the previous NAT model, the position accuracy is still less desirable (say, less than 30%) and has great exploration space. We provide few examples in Appendix [ There is also trade-off between the effectiveness and efficiency, the choice of the non-auvoreeressive means the efficiency and the choice of autoregressive means the effectiveness.
Repeated Generation Analysis Previous NAT often suffers from the repeated generation problem due to the lack of sequential position information. NAT is less effective to distinguish adjacent decoder hidden states, which is copied from the adjacent source representation. To further study this problem, we proposed to evaluate the gains of simply remove the repeated tokens. As shown in T perform the repeated generation analysis the paraphrase generation tasks. Removing repeated tokens has little impact for PNAT model, with only 0.05 BLEU differences. However for the NAT-base model, the gap comes with almost BLEU (0.89). The results clearly demonstrate that the explicitly position model essentially learns the sequential information for sequence generation.
Paraphrase(Test-BLaa
Model
remove repeats w/o remove repeats
BLEU
NAT-base PNAT
20.34
19.45
0.89
29.00
28.95
0.05
Table 4: Results on test set of Quora.
Convergence Efficiency We also perform the training efficiency analysis in IWSLTI6 DE-EN Translation task. The learning curves are shown in 2 The curve of the PNAT is on the top-left corner. Remarkably, PNAT has the best convergence speed compared with the NAT competitors and even strong autoregressive model. The results are in line with our intuition, that the position learning brings meaningful information of position relationship and benefits the generation of the target sentence.
30
My
25
20
BIFU
15
10
Transformer
NAT
5-
1R-
PMAT
100000
200000
250000
300000
Train Steps
Figure 2: The learning curves from training of models on evaluation set of IWSLT-16 DE-EN. Mini-batch size 2048 tokens.
5
RELATED WORK
Gu et al. 2018) first develops a aoo-erttreeeessvvve Transformer for neural machine translation (NMT) tasks, which produces the outputs in parallel and the inference speed is thus significantly boosted.

---

Under review as a conference paper at ICLR 2020
Due to the removal of the dependencies between the target outputs, it comes at the cost that the translation quality is largely sacrificed. A line of work has been proposed to mitigate such performance degradation. Some previous work is focused words as inputs, such as Guo et al. (019) and Lee et al. (2018). [ee et al. (2018) proposed a method of iterative refinement based on the latent variable model and denoising autoencoder. [Guo al. (2019) enhances decoder input by introducing the phrase table in statistical machine translation and embedding transformation. Another part of previous work focuses on improving the supervision of NAT's decoder states, including imitation learning from autoregressive models (Wei et al. 2019) or regularizing the decoder state with backward reconstruction error ( (Wang et al.) 2019). There is also line studies build upon latent variables, such as Kaiser et al. (2018) and Roy et al. al.] (2018) utilize discrete latent variables for making decoding more parallelizable. Moreover, /Saao et al. (2019) also proposed a method to retrieve the targer sequential information for NAT models. Unlike previous work, we explicitly model the position, which has shown its importance to the autoregressive model and can well model the dependence between states. To the best of our knowledge, PNAT is the first work to explicitly model position information for -on-autoreeressive text generation.
enhancing the decoder inputs by replacing the target
6 CONCLUSION
We proposed PNAT, -on-autoreeressive transformer by explicitly modeled positions, which bridge the performance gap between the non-autoregressive decoding and autoregressive decoding. Specifically, model the position latent variables, and training with heuristic searched positions with MC algorithms. As a result, PNAT leads to significant improvement and move more close to the performance gap between the NAT and AT machine translation tasks. Besides, the experimental results of the paraphrase generation task show that the performance of the PNAT can exceed that of the autoregressive model, and the same time, it also has a large improvement space, According to our further analysis on effectiveness of position modeling, in future work, we can still enhance the performance of the NAT model by strengthening position learning.
REFERENCES
Dzmitry Bahdanau. Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv: 1409.043, 2014.
Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn incoder-decoder for statistical machine translation. In EMNLP, pp. 1724-1734, 2014.
Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann Dauphin. Convolutional sequence sequence learning. In ICML, pp. 1243-1252, 2017.
Alex Graves, Abdel-rahman Mohamed, and Geoffrey Hinton. Speech recognition with deep recurrent neural networks. 2013 IEEE international conference on acoustics, speech and signal processing. pp. 6646-664444 IEEE, 2013.
Jiatao Gu, James Bradbury, Caiming Xiong, Victor O.K. Li, and Richard Socher. Non-autorregressive neural machine translation. In ICLR, 2018
Junliang Guo, Xu Tan, Di He, Tao Qin, Linli Xu, and Tie- Yan Liu. Non-autoregreessive neural
machine translation with enhanced decoder input. In AAAI, volume 33, pp. 3723-3730, 2019.
Ankush Gupta, Arvind Agarwal, Prawaan Singh, and Piyush Rai. A deep generative framework for paraphrase generation. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv: 02531, 2015.
Lukasz Kaiser, Samy Bengio, Aurko Roy, Ashish Vaswani, Niki Parmar, Jakob Uszkoreit, and Noam Shazeer. Fast decoding in sequence models using discrete latent variables. In ICML, pp. 2395-2404, 2018.
10

---

Under review as a conference paper at ICLR 2020
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6900, 2014.
Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondřej Bojar, Alexandra Constantin, and Evan Herbst. Moses: Open source toolkit for statistical machine translation. In ACL, pp. 177-180, 2007.
Alex Krizhevsky, lyy Sutskever, and Geoffrey Hinton. Imagenet classification with deep convolu- tional neural networks. In NIPS, pp. 1097-1105, 2012.
Jason Lee, Elman Mansimov, and Kyunghyun Cho. Deterministic non-autoregressive neural sequence modeling by iterative refinement. In EMNLP, pp. 1173-1182, 2018.
Zhuohan Li, Di He, Fei Tian, Tao Qin, Liwei Wang, and Tie- Tie-Yan Yan Liu. Hint-based training for non-autoregressive translation. In NeuralIPS (to appear), 2019.
Tianyu Liu, Kexiang Wang, Lei Sha, Baobao Chang, and Zhifang Sui. Table-to-text generation by structure-aware seq2seq learning. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
Xuezhe Ma, Chunting Zhou, Xian Li, Graham Neubig, and Eduard Hovy. FlowSeq: Non- autoregressive conditional sequence generation with generative flow. In EMNIP-IJCNLPP pp. 4273-4283, Hong Kong, China, November 2019. doi: 10.1655/11/D1191433. URL https: www. wwweb....b..o.. ozgg/anthoooogy/DI1---
Ning Miao, Hao Zhou, Lili Mou, Rui Yan, and Lei Li. CGMH: Constrained sentence generation by Metropplis- Hastings sampling. In AAAI, 2019.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. BLEU: A method for automatic evaluation of machine translation. In ACL, pp- 311-318. 2002.
Aurko Roy, Ashish Vaswani, Niki Parmar, and Arvind Neelakantan. Towards a better understanding of vector quantized autoencoders. 2018
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In ACL, pp. 1715-1725, 2016.
Chenze Shao, Yang Feng, Jinchao Zhang, Fandong Meng, Xilin Chen, and Jie Zhou. Retrieving sequential information for non-autoregress con-autoregressive neural machine translation. In ACL, 2019.
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position representations. In NAACL-HLT, pp. 464-468, 2018
Xiaoyu Shen, Hui Su, Yanran Li, Wenjie Li, Shuzi Niu, Yang Zhao, Akiko Aizawa, and Guoping Long. A conditional variational framework for dialog generation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp- 504-509, 2017.
Yikang Shen, Zhouhan Lin, Athul Paul Jacob, Alessandro Sordoni, Aaron Courville, and Yoshua Bengio. Straight to the tree: Constituency parsing with neural syntactic distance. In ACL, pp. 1171-1180, 2018.
Martin Sundermeyer, Ralf Schlüter, and Hermann Ney. Lstm neural networks for language modeling. In Thirleenth annual conference of the international speech communication association, 2012.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones. Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, pp. 9098-6008, 2017.
Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. Pointer networks. In NIPS, pp. 2692-2700, 2015.
Yiren Wang, Fei Tian, Di He, Tao Qin, ChengXiang Zhai, and Tie- Yan Liu. Non-autorrerrssiiee machine translation with auxiliary regularization. In AAAI, 2019.
Bingzhen Wei, Mingxuan Wang, Hao Zhou, Junyang Lin, and Xu Sun. Imitation learning for non-autoregressive neural machine translation. In ACL, 2019.
11

---

Under review as a conference paper at ICLR 2020
A HEURISTIC SEARCH FOR POSITIONS
Algorithm Heuristic Search for Positions
Input: The candidates set of decoder inputs: = {di,"., das} and target embeddings:
{y1,- yar); }:
Output: The position of the decoder inputs 2.
1: initial A (). D D, =Y:
2: compute the similarity matrix Simp,y: the Sim[i, the matrix is the similarity between the d; and yjj computing with sima, cosine (di, Uj);
3: repeat
4: extract the similarity matrix Simp,y from the Simp,y:
5: select: (i,j) arg (ax().) Simb.y
6: update: A + A ((1,5)) - [d)), y t y [8i1:
7: until D = {} and y 0
8: for each pair (i.j) in A do
set 2; =
9: end for;
10: return
As shown in Algorithm we perform a greedy algorithm to select the pair with the highest similarity score iteratively until the permutation is generated.
The complexity of this algorithm is ((M (M is the length of output sentence). Specifically, the complexity to select the maximum from the similarity matrix is of 2) for each loop. We need M loops of greedy search allocate positions for all decoder inputs.
B CASE STUDY OF PREDICTED POSITIONS
We also provide few examples in Table For each source sentence, we first analyze the generation quality of the PNAT with heuristic searched position. Besides, we also show the translation with the predicted position. We have the following observations: First, the output generated by the PNAT using the heuristic searched position always keeps the high consistency with the reference, shows the effectiveness of the heuristic searched position. Second, better position accuracy always yields better generation performance (Case 1,2 against Case 3). Third, permutation accuracy is lower, still generates a good result, the reason why we chose to use the relative self-attention instead of absolute self-attention.
in case though the
12

---

Under review as a conference paper at ICLR 2020
Source
bei dem deutschen Gesetz geht es um die
Zuweisung bei der Geburt
Reference
German law about assigning at birth.
Heuristic Searched Position(HSPP)
3, 6. 1, 2, 10, 0, 5, 4, 8,
PNAT w/ HSP
German law is about assigning them at birth.
Predicted Position
3, 6. 1, 2, 10, 0, 5, 7, 8,
PNAT w/ Predicted Postion
German law is about assigning them at birth..
Source
weiß er über das Telefon @-@ Hacking
Bescheid
Reference
does he know about phone hacking ?
Heuristic Searched Position(HSP)
2, 1, 3, 4, 8, 5, 6, 0, 7.9
PNAT w/ HSP
does he know the telephone hacking
Predicted Position
1. 0, 3, 4, 8, 5, 6, 2. 9
PNAT w/ Predicted Postion
know about the telephone hacking
Source
CCAA bedeutet, möchte eine Besucherin wissen
Reference
one visitor wants know what CCAA means.
Heuristic Searched Position(HSP)
5, 6, 7, 8, 9, 3, 2. 0. 1, 11, 4, 10 visitor wants to know what CCAA means.
PNAT w/ HSP
Predicted Position
5. 0. 1, 2, 3, 7, 4, 8, 9, 11, 6, 10
PNAT w/ Predicted Postion
CCAA means wants know
visitor.
Source
eines von 20 Kindern in den Vereinigten Staaten hat inzwischen eine Lebensmittelal- lergie
Reference
one in 20 children in the United States now have food allergies
Heuristic Searched Position(HSP) PNAT w/ HSP
14, 1, 2, 3, 4. 5, 6, 7, 9, 8, 0, 10, 11, 12, 13 one of 20 children in the United States now has food allergy.
Predicted Position PNAT w/ Predicted Postion
14, 0, 1, 2, 3, 4, 5, 6, 8, 7, 9, 10, 11, 12, 13 of 20 children in the United States now has food allergy
Table Examples translation outputs from PNAT with different setting on WMT14 DE-EN. is should be noted that the length is different between the position sequence and the output sequence because we keep the origin position output and combine the BPE sequence to word sequence.
13


Reviews:
[{"review_id": "Sklk_sZaYH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work proposes a non-autoregressive model for conditioned text generation. The non-autoregressive decoder conditions on a sequence of discrete latent variables, which represent the generation order and can be autoregressively calculated. Instead of doing marginal inference, the paper takes the top 1 generation order that best match inputs. Experiments on machine translation and paraphrase generation show strong result in comparison to other non-autoregressive models.\n\nThe idea of delegating generation order to the latent variables seems interesting, and the paper takes a reasonable approach when fleshing it out. Most of the experiments seem solid to me. The presentation does a good job in giving a very brief description of the model to readers familiar with non-autoregressive generation; but for those who are not (like myself), much content needs to be clarified.\n\nDetails:\n\n- How is the model trained? Eq.13--14 present the loss, does it ever include the length predictor? Is the computation graph fully-diiferentiable? If it is, how do the authors backprop through the argmax in the length? If not, is reinforcement learning used?\n\n- Can the authors specify B around Eq.5? Also \\Delta M above section 4.3.\n\n- Is the length predictor basically a classifier? Have the authors considered doing regression, which preserves the order relation (e.g., 3 is less than 5)?\n\n- HSP. Can the authors describe the matching algorithm in detail? If it is some well-known algo like the Hungarian algorithm, please specify. If the authors come up with their own algo, please typeset it and analyze the complexity.\n\n- Adding onto the above: is the algorithm exact or approximate? If the later, Eq.11 should not be \\argmax.\n\n- Table 1 is really confusing: what do NPD, WT, i_dec stand for?\n\n- 3.2 discusses two position predictors. Which of them is actually used?\n\n- Tabl 2: what does `Searched-Position` stand for? I thought the position search is only used to derived the reference position sequence.\n\n- It is a bit sad that the paper started with marginal inference but ends up taking the top 1. Have the authors considered taking top K of the predicted orders to have a better approximation to the likelihood?\n\n\nMinor:\n\n- The second paragraph in intro. I'm not sure why non-autoregressive model has a larger search space. Both of them search over \\Sigma^\\ast, with \\Sigma being the vocabulary.\n\n- Typo above section 4.4: ... not powerful enough....", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "SkgebGyeqS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work proposes an alternative approach to non-autoregressive translation (NAT) by predicting positions in addition to the word identities, such that the word order in the final prediction doesn't matter as long as the positions are correct. The length of the translation is predicted similar to Gu et al 2017, as well as smoothly copying the source sequence to decoder input. However, since the positions are unknown, this paper employs a heuristic search method to find the nearest neighbors in the embedding space to obtain position supervision.\n\nExperiments are conducted on a machine translation task and a paraphrase generation task. For WMT14 DE-EN and IWSLT16 DE-EN, this method gets superior performance compared to baselines. For paraphrase generation, this approach with an AR position predictor gets even better performance than a normal AR transformer. Further analysis shows that this approach degrades less without knowledge distillation, or without removing repetitions.\n\nPros:\n1. This approach gets better performance compared to baselines.\n2. The idea of modeling positions as a latent variable is interesting and might generalize to other tasks beyond NAT.\n\nCons:\n1. This work should compare to later baselines such as FlowSeq (https://arxiv.org/pdf/1909.02480.pdf) which gets better performance with flow.\n2. In table 1, although the proposed approach outperforms imitate-NAT, the speedup is lower, making it hard to judge which is actually better.\n3. In table 2, why is AR predictor used? What's the performance of NAR predictor? (in general why consider NAR position predictor at all?)\n4. It is not clear why heuristic search would work here. Is any pretraining required? Otherwise, since there's no gradient signal for the positions, I'm not sure how the model figures it out.\n\nQuestions:\n1. How many samples are used in table 1 LPD? Or is it argmax decoding for each length?\n2. Is it possible to include a few examples showing predicted positions in an appendix to help better understand the model's behavior?\n3. Why do you think positions can be predicted in a NAR manner? Isn't it just shifting the burdens to the position predictor? (Since in transformers if it's able to learn positions then it should be trivial to reorder based on those positions)\n\nMinor details:\nSome typos need to be fixed.\n\nOverall, it is an interesting idea to predict the positions and word identities separately, and with Gumbel-Softmax and VAE we might be able to optimize the true marginals instead of relying on heuristic search. However, empirically there's been better performance achieved by flow-based models, so I am inclined to reject this work.\n\n\n--------updates after reading the authors' rebuttal-----\nI really appreciate the substantial experiments the authors conducted post-review. However, I still have some concerns regarding the baseline flowseq. Flowseq-large gets better results but is not reported in the main paper. Besides, it is unclear how is the speedup computed since Ma et al 2019 reported that the speedup depends on batch size, and at batch size 120 the speedup is ~X6 for flowseq-base.\n\nMinor issue: in table 1 which dataset are the speedup numbers based on?", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "BylAtfRPjB", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work builds on the non-autoregressive translation (NAT) by using position as a latent variable. Unlike the work by Gu et. al. 2018, where they assume the output word order to follow the word order of the input sentence, this work explores predicting word order supervision as an additional train signal. It shows that predicting the position of the words improves the performance of the translation and paraphrase task. This paper uses a heuristic that the inputs positions and output positions of the decoder with close by embeddings are more likely to represent the position mapping. \n\nSince word order might change across languages the idea of using position based supervision seems promising. The results from the experiments on translation and paraphrasing tasks seem promising as they beat previously established baselines. With techniques like length parallel decoding from Gu et. al. 2018, PNAT performs much better than the baselines. For the paraphrasing task, it is interesting to observe that PNAT beats the autoregressive transformer based model. \n\nPros:\n- This work gives a convincing argument to model word position prediction as a latent variable.\n- Experiments show PNAT beats baseline models for Translation and Paraphrasing task. \n- It also shows that using position supervision increases the convergence speed of the model.\n\nQuestions:\n- While position prediction seems like a good idea, I am not fully convinced of the heuristic used -  similarity between input (d_i) and output (y_j) is used to determine the position supervision. Since (d_i)s undergo many transformations to produce (y_j)s, the embedding vectors don't necessarily have to be similar for, some d_i to have greater influence on a particular y_j. It would be nice to verify this assumption using some gold data or some manual checks. \n\n- In addition to the previous point, is the model pertained before this heuristic is used? Since, starting with random initialization might just reinforce random position mappings based on initial conditions.\n\n-  In describing the HSP, could you please make it more clear how the z_i are decided? Is it that the iteratively best (d_i, y_j) is selected as the z_i and then d_i & y_j are removed from the corresponding sides?\n\n- The tables assume that the reader knows about the abbreviations. Could you please add what NPD, WT, etc. mean?\n\n- It would be nice to see the respective results with NAR position predictor since the discussion is about building a Non Autoregressive model.\n\n- Are the gains from NAT lost by using AR position predictor since autoregressive prediction is added indirectly to the whole model?\n\n- In Table 2 PNAT w/HSP seems to have amazing performance compared to other models. Could the authors shed some light on why this cannot be used directly? Is it because of delays due to the iterative process in extracting z_i?\n\n\n\nMinor:\n\n- There are a couple of typos.\n\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a non-autoregressive NMT model which predicts the positions of the words to be produced as a latent variable in addition to predicting the words. This is a novel idea in the field of several other papers which are trying to do similar things, and obtains good results on benchmark tasks. The major concerns are systematic comparisons with the FlowSeq paper which seems to have been published before the ICLR submission deadline. The reviewers are still not convinced by the empirical performance comparison as well as speed comparisons. With some more work this could be a good contribution. As of now, I am recommending a Rejection.
Decision: Reject