Step-unrolled Denoising Autoencoders for Text Generation
Authors: Nikolay Savinov
Junyoung Chung
Mikolaj Binkowski
Erich Elsen
Aaron van den Oord
Venue: ICLR 2022 Poster (2022)
Keywords: generative models
text generation
denoising autoencoders

Abstract: In this paper we propose a new generative model of text, Step-unrolled Denoising Autoencoder (SUNDAE), that does not rely on autoregressive models. Similarly to denoising diffusion techniques, SUNDAE is repeatedly applied on a sequence of tokens, starting from random inputs and improving them each time until convergence. We present a simple new improvement operator that converges in fewer iterations than diffusion methods, while qualitatively producing better samples on natural language datasets. SUNDAE achieves state-of-the-art results (among non-autoregressive methods) on the WMT'14 English-to-German translation task and good qualitative results on unconditional language modeling on the Colossal Cleaned Common Crawl dataset and a dataset of Python code from GitHub. The non-autoregressive nature of SUNDAE opens up possibilities beyond left-to-right prompted generation, by filling in arbitrary blank patterns in a template.

Published as a conference paper at ICLR 2022
STEP-UNROLLED DENOISING AUTOENCODERS FOR TEXT GENERATION
Nikolay Savinov' Erich Elsen Aäron van den Oord
Junyoung Chung"
Mikotaj Binkowskii
DeepMind, London, UK
ABSTRACT
In this paper we propose new generative model of text, Step-unrolled Denoising Autoencoder (SUNDAE), that does not rely on autoregressive models. Similarly to denoising diffusion techniques, SUNDAE is repeatedly applied on sequence tokens, starting from random inputs and improving them each time until con- vergence. We present simple improvement operator that converges in fewer iterations than diffusion methods, while qualitatively producing better samples on natural language datasets. SUNDAE achieves state-of-the-art results (among non-autoregressive methods) on the WMT' 14 English-to-German translation task and good qualitative results on unconditional language modeling on the Colossal Cleaned Common Crawl dataset and a dataset of Python code from GitHub. The non-autoregreesive nature of SUNDAE opens up possibilities beyond left-to-right prompted generation, by filling in arbitrary blank patterns in a template.
INTRODUCTION
Autoregressive (AR) models have shown excellent results in generating text (c.g., GPT-3, Brown et al., 2020). However, while their training scales very well, sampling is prohibitively slow for many practical applications. Moreover, there are limitations the kinds of conditioning AR models can seamlessly handle: the left-to-right restriction makes it hard to "fill in the gaps" in partially written text draft. Even more importantly, this prohibits iterative refinement of complete text drafts make them more self-consistent, which is a common task for human writers. Finally, AR models require network architectures be causal, severely limiting the kinds of neural network architectures that can be used for rext-moeeling All of these motivated the machine learning community to make extensive efforts to propose alternatives AR models.
Machine translation (MT) was perhaps one of the first tasks where non-AR approaches were shown seriously rival the AR-based state of the art: methods like CMLM (Ghazvininejad et al., 2019) and DisCo (Kasai et al., 2020) show promising results and their decoding speed is excellent compared to AR. However, while their performance is competitive, they are still behind the AR benchmark and actually require distillation of larger AR model without which, performance drops considerably.
Non-AR methods have proven hard to apply to the general unconditional language modeling (LM) task. When there is conditioning, the multi-modality problem becomes paramount, shown by Gu et al. (2017), which likely makes it problematic to use methods like CMLM and DisCo be- cause their decoding mechanism is deterministic and does not model uncertainty. Yet, recently the community has seen promising results from non-AR models like Multinomial Diffusion (Hooge- boom al., 2021) and D3PM (Austin et al., 2021). These methods optimize lower bound (ELBO) on likelihoods and have shown negative log-likelin -igliiklldood (NLL) results approaching AR models on several benchmarks like text8 (Mahoney, 201 and LMIB (Chelba al., 2013). However, major gap in NLL persists, and samples from those models lack coherence.
In this paper we propose a novel non-autoregressive method which shows state-of-the-art results in machine translation on WMT' 14 EN-DE raw data (without distillation from AR) amongst non-AR methods and good qualitative results on unconditional language modeling on the Colossal Clean Common Crawl (C4) dataset (Raffel et al., 2019) and a dataset of Python code from GitHub. Our model operates as a homogenoogss Markov Chain similar to that of Lee et al. (2018): condi- tioned on the corrupted data, it tries to approximate the original uncorrupted samples by per-token
Shared first authorship.

---

Published as a conference paper at ICLR 2022
sundae is an ice cream dessert that typically consists of one or
comuup
devoising
sund loop Ga genes ice greatly photograp that76fen $30 oneFrench
unrolled devolaing
generative unrol!
sunday of optical cream piece that may at as one
Figure I: The difference between denoising and unrolled denoising. Original text (top) is randomly corrupted, producing text (middle) where some tokens are original (green) and others are corrupted (red). This text is denoised by sampling from the generative model to produce another noisy text (bottom). While standard denoising autoencoders only learn mapping from the middle text to the top text, Step-unrolled Denoising Autoencoder learns a mapping from bottom to top (including middle special case of zero unroll steps). This has an intuitive meaning: during generation time, the network will mostly (right after the first step) encounter texts like the bottom one, not like the middle one so unrolls prepare the model during training for inputs will get generation time.
factorised distribution. During generation, we unroll the chain by sampling from the transition dis- tribution and feeding samples back into the input. We propose a new training mechanism called un- rolled denoising, which uses unrolls of the chain during training well, and which we empirically show to be crucial for practical performance of the algorithm in ablation studies. This feature mo- tivates the name for the model, Step-unrolled Denoising Autoencoder (SUNDAE). The difference between usual denoising and unrolled denoising is illustrated in Figure 1.
To summarize our contributions:
We present SUNDAE, generative model of text that unrolls the denoising process during training.
SUNDAE achieves state-of-the-art results on WMT 14 Engrimhh translation task among non-AR methods.
We demonstrate good qualitative results for unconditional generation and inpainting on Colossal Clean Common Crawl dataset and dataset of Python code from GitHub.
We carefully ablate and analyze the properties of the proposed method, and show that unrolls during training are crucial for the model's performance.
METHOD
We approach the problem of generative modelling of discrete sequences by bringing together the framework of denoising autoencoders and Markov chain models. In this section, we first discuss the definition of the generative model, and then describe the training process, which includes our main contribution, unrolled denoising.
For fixed prior distribution Po on some space consider a process x; ~ fo(-(x;11), where fo is parametric transition function. Then {xe)e is time-homogeneous Markov chain, and transition
1-steps ahead has the following form!
Σ
II s=1
folxx
LEX
For a fixed number steps T,, the prior Po and decoder pr together determine our model distribution Pr (x²) Pr (x Xx (Xp).
We assume X = {l...., w} to be the space of sequences of length with values at all positions coming from vocabulary of size and the prior po to be uniform over X. Let Pasta be distribution of the data and assume that fo induces a conditional probability distribution fo x') that can be
for formal derivation, Appendix A.
2

---

Published as a conference paper at ICLR 2022
factorised into conditionalll--ndeeeeee product as follows:
fo(z)<=') = fo (z) (zx²) ...
for x. x' E X.
(2)
Note that although for / = the distribution pi (-120) is limited by product structure (Eq. 2), the subsequent P&S are not restricted the same way, and after each step they belong to potentially more and more expressive family.
2.1 TRAINING WITH UNROLLED DENOISING
Due to intractability of the likelihood of the model pr as well as the computational requirements of optimising the whole chain {pe)r, we propose an efficient two-step training method which we term unrolled demosing and visualise in Figure 1. We consider a smaller number of steps than T that intend to use at sampling, but compensate for that, we unroll the chain starting from corrupted data samples. rather than the prior Po- This way, the model learns to denoise the samples it is likely encounter during the full unroll used at sample time. While using a single step would resemble the training strategy of BERT (Devlin et al., 2018), using at least two steps is essential for the performance of our model.
Consider corruption distribution ((=*|x) that replaces random proportion of tokens in a sentence with ones randomly sampled from a uniform distribution. Our objective is given by (1:2)
(L + (2)) where
(() -E x--Péaes xo-=q(-1x)
[log fo(x(xx-1))
xi~foll (x))
ie. the reconstruction loss of the chain after steps, starting from corrupted sample Xo- In Ap- pendix A.1 we show that (() is an upper bound on the actual negative log- ligeliidld from the model distribution P::
(1) == E x-Paatt [log [x(x)(((x))] x" x'>gq-1xx)
(4)
Note that L(T) is the actual term we would like minimise pr were tractable, and in fact closely related to evidence lower bound (ELBO) via a fixed encoder (x*])x) (Appendix A.2). However,
due to the inherent non-differentiable nature of discrete Markov chain models, optimisation of L(T) would not only be costly, but also limit the gradient flow to the last step of the chain. We found instead that optimising several steps together without propagating the gradient through sampling is sufficient for obtaining good results. Averaging more [(() terms can lead to minor improvements in performance (see Figure 4d in Appendíx C), but it considerably slows down the training speed. We discuss these and other loss variants further in Section 3.1.
Since function fo modelled neural network, the logarithm under expectation on the right hand side of Eq. 3 is obtained from predicted logits, thus we term ()) and (2)( logits loss and unrolled logits loss, respectively.
Corruption function. To corrupt texts, we first sample a proportion of tokens uniformly from [0, 1]. randomly select positions according to this proportion and then change tokens at those posi- tions to random tokens sampled uniformly from the vocabulary. This way, we want to ensure that samples seen at various steps during generative unroll (including the first one, sampled from the uniform prior) are well-represented corruptions applied during training. More details and relations to the forward diffusion process (Hoogehoom et al., 2021) ) are presented in Appendix A.3.
2.2 SAMPLING
At sampling time we follow the structure of the Markov process and sample sequentially 2, ~ [n(#:/0r-1) for some fixed number of steps T, beginning from a random sequence wo (potentially starting with a prompt a template for conditional generation). To control the speed of convergence, we propose three improved strategies which allow the use of a much smaller number of steps:
Low-temperature sampling. For temperature and trained network fo we consider modified function fī such that log tures lower than leads to much faster convergence than with the original logits, and allowed generation of high-quality samples in matter of 10-16 steps.
fT f x log fe(- We found that sampling with tempera-
3

---

Published as a conference paper at ICLR 2022
Raw BLEU
AR-distilled BLEU
Model
Steps (7)
EN-DD
DE-+EN
EN-DE DE-+EN
AR Models
Transformer Base (65M) (Vaswani et al., 2017) (n 4)
27.3
31.78"
Non-AR Models
NAT (Gu al., 2017) (n 100)
19.17
23.20
LVM-DAF (Lee et al., 2018)
21.54
25.43
NAT-REG (Wang et al., 2019) (1=
1
24.61
28.90
LV-NAR (Shu ct al., 2020) (x=50)
11.8
25.
NART w/ hints (Li et al., (019)(n
25. 25.20
29.52
FlowSeq (Ma et al., 2019) (0=30)
23.64
28.29
25.31
30.68
ReorderNAT (Ran et al., 2019)
26.49
31.13
NART (Sun
2019) (9)
26.80
30.04
CMLM (Ghuzvininejad al. 2019) Mask-Predict
22.25
25.94
29.90
CMLM (Chazvininejad al., 2019) Mask-Predict (n=
10
24.61
27.03
30.53
DisCo (Kasai et al., 2020) + Mask-Predict (n=5)
4
25.83
30.15
DisCo (Kasai ct al., 2020) + Mask-Predict (n=
10
27.06
30.89
DisCo (Kasai et al., 2020) Fasy-First (x=5)
4-51
24.8
27.34
31.31
NARLVM (Lee ct al., 2020) (x (5)
4
27. 27.40
JM-NAT (Guo et al., 2020) (n=3)
4
27.05
31.51
JM-NAT (Guo et al., 2020) (n=3)
10
27.69
32.54
SMART (Ghazvininejad
2020) (m=5)
27.03
30.87
SMART (Ghazvininejad
2020) (n=
10
27.65
31.27
Imputer (Saharia al., 2020) (x=
4
24.7
28.0
31.0
Imputer (Saharia et ol., 2020) (n 1)
8
25.2
28.2
31.3
SUNDAE (ours 63M)
Detenninistic (n= 16)
4
25.01
29.53
28.33
32.25
Deterministic (n 16)
8
25.53
30.01
28. 38.32
32.27
Detenministic (n= 16)
10
25.54
30.11
28.32
32.27
Stochastic (n 16)
4 23.05
28.13
27.94
32.10
Stochastic (n=18 16)
26.08
30.48
28.23
32.33
Stochastic (n 16)
10
26.25
30.80
28.33
32.29
Stochastic (n=16)
16
26.24
30.76
28.46
32.30
Table 1: Test BLEU scores of AR and non-AR systems on the 1MT Engriah-to-German (EN->DE) and German-to-Eil tnn-lsh (DE++EN translation tasks. The number of reranked candi- dates is denoted n. SUNDAE does not use an AR model for re-ranking. We highlight BLEU scores of the best non-AR systems in bold font. All the entries are ordered based on the EN-DE BLEU score. "indicates the results of the Transformer baselines implemented by the authors of DisCo. DisCo has dynamic inference mode for which we report the average number of steps.
A Argmax-unrulde decoding. At the limit 7 - sampling with temperature reduces to deter- ministic argmax decoding where the most probable token chosen each step. Relatedly to our unrolled logits loss, we modify this strategy by resampling the low-certainty tokens in ac- cordance with unrolled logits. This heuristic allowed further improvements sampling speed while maintaining the high quality of the samples. We discuss it in detail in Section 3.1.
Updating fewer tokens. In tasks where diversity is paramount (like unconditional text gener- ation), we found that updating random subset of tokens at each decoding step leads to faster convergence. This likely happens because independent sampling of all tokens might create un- coordinated changes which could take some time to fix the subsequent steps. use this strategy in Section 3.2.
EXPERIMENTS
3.1 MACHINE TRANSLATION
We first evaluate SUNDAE on Machine Translation (MT) benchmarks. We compare SUNDAE against AR and non-AR models in terms of the translation quality using BLEU (Papineni et al., 2002) as the metric. We demonstrate that without using techniques like sequence-level knowledge distillation (Kim & Rush, 2016), SUNDAE performs almost well as the AR model and outper- forms all other methods that do not rely on AR models.
Target Length Prediction. Unlike their AR counterparts, non-AR models do not explicitly learn predict end of sequence, but instead have been shown benefit from auxiliary target length
4

---

Published as a conference paper at ICLR 2022
prediction (Lee et al., 2018; Ghazvininejad et al., 2019; Kasai et al., 2020). A common approach is to treat as a classification task, predicting either the exact length or the difference between the source and target lengths. The decoding process has to commit to the predicted length, and often multiple length beams are used to get the best translation results. In our models, we use a separate network that receives source encodings length embedding vector is prepended to source embeddings so that decoder can attend to it. During training, we use the reference target length, while sampling, a predicted one, although the model does not need to fully commit to the predicted length like some previous non-AR models (Saharia et al.. 2020). Note that the length classification loss does not affect the encoder parameters. Our models are also trained to predict the padding tokens at the end of the text. For more details, see Appendix B.
input and predicts corresponding target length. Target
Experiment Settings. We conduct experiments on WMT 14 parallel corpora using EN (4.5M pairs) and EN-FFR (36M pairs) translation tasks. The raw texts are encoded using BPE (Sen- nrich et al., 2015) as the subword units, and we use the same preprocessed data as in Vaswani et al. (2017) for fair comparisons. We evaluate the performance by measuring BLEU (Papineni et al., 2002: Post, 2018) on the test split of cach translation task?. We use the encoder-decoder Trans- former architecture for MT (Vaswani et al., 2017), but remove the causality masking in the decoder. There are 6 attention layers for both encoder and decoder, 8 attention heads, 512 model dimen- sion and 2048 feedforward hidden dimension. The total number of parameters is 63M including the target length prediction module described in 3.1. We use dropout (p = 0.1) and label smooth- ing (c10.1) during training for all tasks, except for AR-distilled EN EN-DE, where we found lower dropout (p=00. 0.05) and no label smoothing produces better results validation. The training batch size is 4096, and we use Adam (Kingma & Ba. 2014) with 3| = 0.9, 3g 0.999. e < 10 6 and weight decay of 0.1 (Loshchilov & Hutter, 2017). We warm up the learning rate from 10-7 to 10-1 in the first 5K steps and decay it to 10-5 using cosine annealing (Loshchilov & Hutter, 2016). Our models were trained for 10 steps using 16 TPU accelerators using 6 precision. We crop sequences that have more than 128 tokens (this occurs less than 0.2% in training data). Finally, we average the last 10 checkpoints to obtain a single model for the evaluation. All hyperparameter tuning is performed on held-out validation set.
Decoding. We decode translations from SUNDAE using two different types of heuristic decoding methods in MT experiments. The decoding process always begins with an array of random integers sampled from the discrete uniform prior Yo Po- while the encoder input the source sentence remains the same throughout the process. The first decoding method is low-temperature sampling, where we iteratively sample y: ~ f (-1 (1)-) for T iterations (with the decoder output logits di- vided by т, see Section 2.2). with T < 16 and T E [0.1, 0.6] determined based on the validation performance. The second method argmax-unrolle unrolled decoding, which requires smaller number of iterations compared to its stochastic counterpart. In argmax- unrolledd decoding, we first compute the logits A = fo of the initial sample array yo and obtain the samples yı, pass a tuple (3)1 A1) to the next iteration. At each step / 2 2, the method finds top-p share of the tokens that are sorted by the log-probabiliyy in descending order (i.e. uncertain tokens) from A 1. where p E [0.1, 0.6] is another hyperparameter searched for in the validation stage. We compute unrolled logits for those top-p tokens and then apply arg max to obtain unrolled tokens. For the rest of the input tokens of y1-1 we compute logits once and obtain X fe(-1 and then apply arg max to obtain predicted tokens. Finally, we combine unrolled tokens for uncertain input positions with predicted tokens for remaining positions to obtain yt. and the tuple (y), A₂) is provided as the input to the next iteration. This procedure is repeated over a fixed number of iterations T'. We always decode n samples in parallel and rerank them based on the model score. We show the relative speed gain of SUNDAE in Table 2, the Transformer base is used as the AR baseline, for which we perform incremental sampling by caching the previous attention states.
Baselines. We compare SUNDAE with Transformer base (Vaswani et al., 2017) as the AR baseline and several non-AR models including CMLM (Ghazvininejad et al., 2019) and DisCo (Kasai al., 2020). The last two are both iterative non-AR models that share a substantial amount of common ground with SUNDAE. However, their best results come from the variants distilled from Trans- former large ( (Vaswani et al., 2017), with only few test results available without distillation. As we aim remove any kinds of dependencies on AR approaches, we would like to make a distinction
Due limited space, we report BLEU using SacreBL in Appendix G.
5

---

Published as a conference paper at ICLR 2022
Steps (T)
Relative Speed Improvement
RanoOAN VaIDAY
Thiking Site
4.7x
8
2.6x
10
2.2x
SIRING jaun)
16
1.4x
Negative BLEUS iqua Ity)
BLEU scores on EMNLP2017 News, Left is better, lower is better. Quality/variation controlled by changing the temperature.
Figure 2:
Table 2: Relative speed gain of SUNDAE over AR Transformer base (greedy decoding) on WMT' 14 EN-DE validation set.
between raw, fully-non-AR models, and AR-distilled ones, i.e. those trained via distillation from large AR teachers (which typically perform better than the student models), or via AR reranking.
Results and Ablation Studies. In Table 1, we show test BLEU scores from SUNDAE and other baselines. On EN DDE, SUNDAE achieved 26.25 26 BLEU, and it performed the best among the raw non-AR models. To the best of our knowledge, SUNDAE achieved the closest result (A 1.05) to the score of Transformer base (Vaswani et al., 2017) without the aid of AR models.² In the raw setting, augma--unrllees decoding outperforms mask-predic (Ghazvininejad et al., 2019) and easy- first (Kasai et al., 2020) by 0.74 BLEU when using the same amount of the worst-case compute budget. At T = 8, SUNDAE performs better than Imputer (Saharia et al., 2020), which is strong baseline that can generate high-quality translation within few (4-8) iterations. We discovered that as I' increases, low-temperature sampling outperforms argmax-unrolled decoding, and at T'= 10, low- temperature sampling outperforms Mask-Predich (Ghazvininejad et al., 2019) and Easy-First (Kasai al., 2020) by 1.5 BLEU score. SUNDAE achieved 30. 80 BLEU on DE- where there are not that many raw baselines to compare with. The difference between SUNDAE and the AR baseline on DE-EN is 0.98. Finally, SUNDAE achieved 37.53 BLEU on EN-FR at T= 10, while the Transformer base (Vaswani et al., 2017) reported 38.1 BLEU on this task. Thus, SUNDAE without distillation is only 0.57 BLEU behind the standard AR baseline on EN- We present more details on EN scores in Table 7 in Appendix G.
While knowledge distillation was outside our main focus, we also report scores for AR-distilled SUNDAE. It again performed well, establishing new SotA on EN-DE translation among AR- distilled models, and being second only to JM-NAT (Guo et al., 2020) in DE-EN task.
We ablate the number of steps of unrolled denoising EN-DE test split. We observe that having at least one unrolled denoising step is crucial to obtain good performance in translation. With only the (()) loss, SUNDAE achieves 11.19 BLEU, whereas using (1::),, the score improves to 26.57 BLEU. Using L (1:3), i.e. averaging more unrolled denoising loss terms, did not improve the perfor- mance further, scoring 26.25 BLEU. The target length prediction also turned out to be an important component of our model: without it, the translation performance degrades on average by 2 BLEU score points on EN (see Appendix C). Finally, we qualitatively show how translation improves along the sampling steps by elimination of incoherent/repeated tokens in Table 5 in Appendix E.
3.2 TEXT GENERATION ATTON
Unconditional Text Generation (Qualitative). We train our method on a large high-quality pub- licly available Colossal Clean Common Crawl (C4) dataset (Raffel et al., 2019) to demonstrate samples. We tokenize the data using SentencePiece (Kudo & Richardson, 2018) 5 with vocabulary size 32K and train randomly cropped sequences of length 32. The network is the same as the Transformer decoder part for MT in Section 3.1 but is much larger 335M parameters: 24 layers,
3 As shown in Table 7 in Appendix G, allowing larger compute reduces the gap even further, to 0.73..
*For distillation, we follow the protocol used by Saharia et al. (2020). Scores in Table 1 were ohtained with distillation from Transformer-Bip to allow direct comparisons with CMLM (Gharviningjad et al., 2019) and SMART Ghazvininejad et al. (2020); for Baaefforme--Bs-ddsitiee https: //gi thub. com/ googggoolleee//////oo e sensenceteeeeeeeeeeee ice
scores see Table 6 in Appendix F.

---

Published as a conference paper at ICLR 2022
Indox
Type
Text
Pronge
Secing si e any national pack. Whill exciling mosacut,
Completion
Seeing
park. exciligg whe wwww
intimidutingg will come
Prompe
Seeing th w e for any national park. While excliin
Completion
Seeing unique experience for any vistorr national pack, While exciting
havigate way througts forest
Table 3: Inpainting from our model trained on C4 (cherry-picked).
1024 embedding size, 4096 hidden size, 16 attention heads. Same as in the MT setup, we remove the causal mask since our model is non-autoreerressive We train up to 400K steps with Adam optimizer with batch size 4096 and use cosine annealing for scheduling the learning rate with minimum 10 maximum 2 10 3 and linear warm-up up of 10K steps from starting value of 10-7.
The trained model is then used for unconditional sampling. Starting with 32 random tokens, we iter- atively perform up to 1K steps from our model, stopping earlier samples do not change anymore. To make convergence faster, we use temperature 0.8 and update only 30% of all tokens at each step, chosen randomly- - as explained in Section 2.2. We show 10 non-cherry-pic model in Table 8 of Appendix H. Of those samples, all but one resemble reasonable internet texts.
samples from our
Unconditional Text Generation (Quantitative). While C4 is one of the largest high-qualit datasets publicly available, it does not have an established benchmark among non-AR methods. To quantitatively evaluate our algorithm, we train it on the EMNLP2017 News dataset, which has numerous text GAN baselines, including a strong ScratchGAN method (d' Autume et al., 2019). We also compare to several other baselines reported in (d' Autume et al., 2019; Caccia et al., 2020); Se- qGAN (Yu et al., 2017), LeakGAN (Guo et al., 2017). RankGAN (Lin et al., 2017), MaliGAN (Che et al., 2017) and autoregressive (AR) language models.
We use the same architecture and optimizer as in qualitative experiments and train with batch size 1024 for 8K steps (chosen to achieve the lowest validation loss). We follow the same tokenization strategy as d' Autume et al. (2019), with vocabulary size of 5.7K and maximum length 52 tokens, padding shorter sequences to this maximum length. During sampling, update all the tokens at each step (which we found beneficial for the chosen metrics). The quality/- variation trade-off' is controlled by changing the sampling temperature in the range [0.1, 2.0), using 60 values overall. For every temperature, we sample 10K texts for evaluating quality and another 10K texts for evaluating diversity, as done by d' Autume al. (2019).
perform 1K steps and
The results are shown in Figure 2. SUNDAE demonstrates good quality/variation trade-off, as measured by BLEU/se/f-BLEU metries, outperforming 4 out of GAN baselines, and competing with ScratchGAN and AR model. In the higher quality region, SUNDAE outperforms ScratchGAN, while in higher diversity region it slightly underperforms. The advantage of ScratchGAN in higher diversity region could be explained by its autoregressive decoding (it is hybrid between AR and GAN), Interestingly, the curve for SUNDAE has a more complicated shape than the baselines - possibly because of its Markov Chain sampling, which can empirically exhibit something akin to "phase transitions". We show samples from SUNDAE in Table 9 of Appendix I.
Text In-painting. The non-autoregressive nature of our model opens up possibilities beyond left- to-right prompted generation, such as filling in arbitrary blank patterns in a template. In this section, we qualitatively investigate this new capability, using the C4-trained model described in previous sections, and another one, trained on a dataset of Python code from GitHub. All the architecture/traiing settings remain the same, except we train twice longer: up to S00K steps. Sampling settings are also the same, except update all the tokens at each step as diversity is less of an issue with stronger conditioning. We construct the code dataset by ex- tracting files ending in . py from open source GitHub repositories with licenses that are one of apache-2.00 0, mit, asd-3-clause, bsd- 2 (il---ccauss, unlicense, cc0-1.0 C, isc, artistic-2. O. We perform document level de-duplication checking for exact matches.
For the C4 dataset, we present a model with a particular type of prompt [CTX1] [MSK]] [CTX2] [MSK2], where [CTX] stands for "context" and [MSK] for "masked", in Table 3, which shows how information can be "teleported" from arbitrary location to another arbitrary location, while taking into account bidirectional context. We change a species of the animal in [CTX2] and observe that the model actually uses the right word while in-painting
http:: hett:://ww..r statmt. wwt//mtt//.//....o..o////.... 17/
7

---

Published as a conference paper at ICLR 2022
Index
Type
Text
Prompt
def trunc_len (x, t)
return min (n, t)
def trunc.len (x, t)
Completion
(x) return min (n, L1
isewen ambda x: is.odd lambda x: not
Prompt
reuse
Completion
iseeven 1ambda x
is.odd lambda x: is.even (x)
Table 4: Inpainting from our model trained on GitHub (cherry-pickeet
[MSK1] and plausibly continues [MSK2] according to the species. This kind of task would be impossible perform for an AR model: with left-to-right causality would be unable to correctly guess the name of the animal, while with a right-to-left one it would have to start filling [MSK2] tokens first, so it would just ignore the conditioning. By contrast, our model succeeds in this task.
For GitHub Python data, we use two different kinds of prompts for our model in Table 4: [CTX1] [MSK1] [CTX2] and [CTX1] [MSK1] [CTX2] [MSK2] [CTX3]. the first example, our model guesses that should be the length of the input sequence and correctly in-paints [MSK1] region. This would again be impossible to do for AR models. Looking from left-to-right, it is im- possible guess that will be the name of the length variable and is meant to be threshold. Looking from right-to-left, it is unclear what variables n and t. represent and what is the function being computed. Only bidirectional model perform well in this task. In the second example, our model correctly follows the suggestion to reuse the function which it also has to in-paint.
RELATED WORK
In this section we provide an overview of generative models, focusing on applications in text domain.
4.1 AUTOREGRESSIVE MODELS
Early attempts to perform autoregressive modelling of text using neural networks began with Bengio al. (2003); later Sutskever et (2011) extended this approach with RNNs. Since then, improve- ments in neural network architecture, such as the Transformer ( (Vaswani et al., 2017), and scaling up as with GPT-3 (Brown et al., 2020) dramatically improved the performance AR models. While the architectures evolved, the loss function remained the same with minor modifications.
Insertion (Stern al., 2019) and Levenshtein (Gu et al., 2019) Transformers are deviations from the standard paradigm of training and could potentially overcome some of AR problems. However, in parallel decoding regime, they have to rely on AR distillation for obtaining competitive results.
4.2 NON-AUTOREGRESSSIVE MODELS IN GENERAL
-on-eetoreeeessvve models evolved in parallel with autoregressive ones but have far struggled to address the same spectrum of tasks with the same quality as AR models.
Diffusion for generative models was originally introduced by Sohl-Diekstein et al. (2015). Re- cently Ho et al. (2020) proposed a different parametrization: instead of modeling one-step tran- sitions, one could instead model p2=)(2:) and later convert those to p(x;-1/xx; by probabilistic reasoning. This was recently up-scaled for image modeling by Nichol & Dhariwal (2021) and ex- tended to continuous time by Song et (2020).
terms of applications to text generation, two recent works addressed this: Hoogeboom et al. (2021) and Austin et al. (2021). Likelihoods ob- tained in those works are promising, but still behind AR and lacking sample quality as well.
Some diffusion works like Mittal al. (2021) also approached modeling discrete sequences in an- other way: first encode sequences with a VAE into a continuous space and then apply continuous diffusion approaches commonly used for image generation.
Variational Autoencoders (VAEs) were proposed by Rezende et al. (2014) and Kingma & Welling (2013) likelihood- baaeed extension of autoencoders. Application of such methods to text has been problematic empirically: as was first noted in Bowman et al. (2015), good results come from using strong AR decoder instead of simple factorised one, but the latents of-
8

---

Published as a conference paper at ICLR 2022
ten ignored - which named "posterior collapse problem". This problem was later tackled by Delta-VAEs (Razavi et al., 2019), with limited success in the text domain (Bosc & Vincent, 2020).
Normalising Flows (Dinh et al., 2014; Rezende Mohamed, 2015; Dinh et al., 2016) are an- other prominent family of generative models, originally proposed for real-valued data and recently revisited for text generation by Hoogeboom et al. (2021). While conceptually interesting, the text samples of such methods lack in fidelity.
Generative Adversarial Networks (GANs) were originally introduced by Goodfellow et al. (2014) and remain one of the dominant methods for image generation. They were later adapted to text generation in Yu al. (2017); Guo al. (2017); Lin et al. (2017); Che et al. (2017). However, text generation with such methods is still a challenge, with a more recent ScratchGAN (d' Autume al., 2019) showing relatively low-quality samples. At least part of the problem with applying GANs to text generation comes from non-differentiabiity of discrete text samples, which requires usage of zeroth order optimization methods.
Energy-Based models have a long history dating back Hopfield (1982); Hinton (2002); LeCun et al. (2006); Ranzato et al. (2007). Recently, Deng et al. (2020) shows promising results for text.
4.3 NON-AUTOREGRESSII MODELS FOR MACHINE TRANSLATION
There have been many attempts to apply non-autoregressive methods to machine translation. Latent transformer (Kaiser et al., 2018) generated latent variables autoregressively and then decoded feed- forwardly. NAT (Gu et al., 2017) was the first to characterize a problem with nor-autoreeressive generation - "mdaltimmddality". FlowSeq (Ma et al., 2019) applied Normalising Flows to machine translation. LVM-DAEs (Lee et al., 2018) are perhaps most related to method, but these models do not have unrolled denoising and their decoding method is deterministic. The leading methods in this area are CMLM (Ghazvininejad et al., 2019) and DisCo (Kasai et al., 2020). While conceptually similar to LVM-DAEs (Lee et al., 2018), these works have introduced significant improvements to the training and decoding procedure, and, as result, shown strong competition Both of them also do not have unrolled denoising and sampling like we do, but they are still powerful baselines. Later, there were attempts combine CMLM with local AR (Kong al., 2020). Perhaps the most related to our work is the SMART (Ghazvininejad et al.. 2020) follow-up of the CMLM. However, this method is not probabilistic and it does not show unconditional generation capanilitio Finally, recent line of work called Imputer (Chan et al., 2020; Saharia et al., 2020) achieves strong results in machine translation by aligning target to source via dynamic programming.
AR methods.
capabilities.
4.4 DENOISING OBJECTIVE IN GENERAL
With the advent of BERT (Devlin et al., 2018), the denoising objective became very popular for text representation learning. The original idea later simplified in RoBERTa (Liu et al., 2019) by removing the unnecessary sexxt-sentenc--reeee (the cloze task). This is similar our work, however, we do not apply masking corrupt the input tokens, and instead only switch them to random ones. More importantly, RoBERTa does not use unrolled denoising. Another work, Electra (Clark al., 2020a), deals with the same kind of corruption that we use, but instead of predicting the original token before the corruption, Electra predicts whether it was corrupted or not. Electric (Clark et al., 2020b) does not perform masking, but instead uses contrasvnt loss to learn a representation of text. BART (Lewis et al., 2019) uses both BERT-style denoising and autoregressive modeling for learning representations. All of these methods were originally motivated by improving representation learning of text. There were a few attempts to heuristically decode models like BERT and turn them into text generative models such in Wang & Cho (2019), however, samples from these models lack coherence.
task and only retaining the denoising task
CONCLUSION
In this paper we have proposed a novel non-autoreeeessive method that operates within the frame- work of denoising autoencoders and, crucially, unrolls the denoising process during training. Un- rolled denoising allows us to achieve state-of-the art results in WMT' 14 English-to-Germal transla- tion task amongst non-AR methods (without distillation from large AR models) and good qualitative results in unconditional text modeling on C4 dataset. We have also qualitatively demonstrated new inpainting capabilities on C4 and GitHub Python data, which opens up avenues for creative text editing where a human could more naturally collaborate with a machine on writing and even programming.

---

Published as a conference paper at ICLR 2022
AUTHOR CONTRIBUTIONS
Nikolay Savinov came up with the idea of unrolled denoising, wrote the first prototype of uncondi- tional generation, contributed to the machine translation codebase and co-led the project. Junyoung Chung wrote most of the machine translation codebase, came up with the length prediction con- ditioning, suggested argmax-unrolled decoding and co-led the project. Mikolaj Binkowski came up with theoretical insights about our model, wrote most of the evaluation codebase, implemented model improvements and co-led the project. All three first authors contributed equally writing the paper. Erich Elsen gave high-level scientific guidance, suggested text in-painting experiments and made substantial edits to the paper draft. Aäron den Oord originally suggested to look into de- noising generative models for text, gave high-level scientific guidance, proposed machine translation experiments and made substantial edits to the paper draft.
ACKNOWLEDGMENTS
We would like to thank Jean- Baptiste Alayrac and Miruna Pislar for valuable discussions on the machine translation experiments, William Chan and Chitwan Saharia for sharing distilled translation datasets, Mihaela Rosca and Cyprien de Masson d' Autume for sharing text GAN evaluation code, Yujia Li and Jack Rae for sharing dataset of Python code from GitHub, Sander Dieleman and Oriol Vinyals for in-depth feedback about our work, Jungo Kasai for responses to our questions about DisCo, Dani Yogatama for sharing knowledge on machine translation evaluation, and Jeff Donahue for feedback and discussions.
REFERENCES
Jacob Austin, Daniel Johnson, Jonathan Ho. Danny Tarlow, and Rianne van den Berg. Structured denoising diffusion models in discrete state-spaces arXiv preprint arXiv:2 (00.::200700000 2021.
Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Janvin. A neural probabilistic language model. The journal of machine learning research, 3:1137-1155, 2003.
Tom Bosc and Pascal Vincent. Do sequence-to-sequeece vaes learn global features of sentences? arXiv preprint arXiv:2004.00000 006683, 2020.
Samuel R Bowman, Luke Vilnis, Oriol Vinyals, Andrew M Dai, Rafal Jozefowicz, and Samy Ben- gio. Generating sentences from a continuous space. arXiv preprint arXiv:1511.06349 2015.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005. 14165, 2020.
Massimo Caccia, Lucas Caccia, William Fedus, Hugo Larochelle, Joelle Pineau, and Laurent Char- lin. Language gans falling short. In 8th International Conference on Learning Representa- tions, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL htttps://ppenreee wwwp//ppeeeeeeeeeeeeeeeeeeee.eee net.tt/wwweeeeeeeeeee......ee httpp//eeweeeeeeeeeeooooooaaaaa/
William Chan, Chitwan Saharia, Geoffrey Hinton, Mohammad Norouzi, and Navdeep Jaitly. Im- puter: Sequence modelling via imputation and dynamic programming. In International Confer- ence on Machine Learning, pp. 1403-1413. PMLR, 2020.
Tong Che, Yanran Li, Ruixiang Zhang, R Devon Hjelm, Wenjie Li, Yangqiu Song, and Yoshua augmented discrete generative adversarial networks. arXiv preprint
Bengio. arXiv:1702.07988. 2017.
Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, and Phillipp Koehn. One billion word benchmark for measuring progress in statistical language modeling. CoRR, abs/1312.3005, 2013. URL
3005.
Kevin Clark, Minh- Tinh-Thang Luong, Quoc V Le, and Christopher D Manning. Electra: Pre-training text encoders discriminators rather than generators. arXiv preprint arXiv:2003. 10555, 2020a.
10

---

Published as a conference paper at ICLR 2022
Kevin Clark, Minh-Thame Thang Luong, Quoe V Le, and Christopher D Manning. Pre-training transform- ers as baaegy cloze models. arXiv preprint arXiv:2012.08561, 2020b.
Cyprien de Masson Autume, Mihaela Rosca, Jack Rae, and Shakir Mohamed. Training language gans from scratch. arXiv preprint arXiv: 099222 2019.
Yuntian Deng, Anton Bakhtin, Myle Ott, Arthur Szlam, and Marc' Aurelio Ranzato. Residual energy-based models for text generation. arXiv preprint arXiv:2004. 1714, 2020.
Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 1810.0080 2018.
Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components esti-
mation. arXiv preprint 0rXiv:1440.85/11 2014.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. arXiv
preprint 160iv:1605. 105.00000000,033,, 2016.
Marjan Ghazvininejad, Omer Levy, Yinhan Liu. and Luke Zettlemoyer. Mask-predict: Parallel decoding of conditional masked language models. arXiv preprint arXiv:1904.09324, 2019.
Marjan Ghazvininejad, Omer Levy, and Luke Zettlemoyer. Semi-autoregressive training improves mask-predict decoding. arXiv preprint 2xXx:2001.087855 2020.
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
Jiatao Gu, James Bradbury, Caiming Xiong, Victor OK Li, and Richard Socher. Non-autorrerresive neural machine translation. arXiv preprint wrXiv:1711.02286 2017.
Jiatao Gu. Changhan Wang, and Jake Zhao. arXiv:190S.11006, 2019.
Levenshtein transformer.
arXiv preprint
Jiaxian Guo, Sidi Lu, Han Cai, Weinan Zhang, Yong Yu, and Jun Wang. Long text generation via adversarial training with leaked information. arxiv e-prints, art arXiv preprint arXiv: 2017.
Junliang Guo, Linli Xu, and Enhong Chen. Jointly masked sequence-to-sequenee model for non- autoregressive neural machine translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 376-385, 2020.
Geoffrey E Hinton. Training products of experts by minimizing contrastive divergence. Neural
computation, 14(8):1771-180% 2002.
Jonathan Ho. Ajay Jain, and Picter Abbeel. Denoising diffusion probabilistic models. arXiv preprint crXiv:2006.11230 2020.
Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, and Max Welling, Argmax flows and multinomial diffusion: Towards aanoatorrerreiive language models. arXiv preprint arXiv:2/02.053379 2021.
John J Hopfield. Neural networks and physical systems with emergent collective computational abilities. Proceedings of the national academy of sciences, 25554225442 1982.
Lukasz Kaiser, Samy Bengio, Aurko Roy, Ashish Vaswani, Niki Parmar, Jakob Uszkoreit, and Noam Shazeer. Fast decoding in sequence models using discrete latent variables. In International Con- ference on Machine Learning, pp. 2390-2399. PMLR, 2018.
Jungo Kasai, James Cross, Marjan Ghazvininejad, and Jiatao Gu. Non-autoregressive machine trans- lation with disentangled context transformer. In International Conference Machine Learning, pp. 5144-5155 PMLR, 2020.
Yoon Kim and Alexander M Rush. Sequence-level knowledge distillation. Mxxv:1606.009.. 2016.
arXiv preprint
11

---

Published as a conference paper at ICLR 2022
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv:1312.61 2013.
arXiv preprint
Xiang Kong, Zhisong Zhang, and Eduard Hovy. Incorporating a local translation mechanism into non-autoreegressive translation. arXiv preprint arXiv:2011.06132 2020.
Taku Kudo and John Richardson. Sentencepicce: simple and language independent subword tokenizer and detokenizer for neural text processing. arXiv preprint arXiv: 2018.
Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, and F Huang. A tutorial on energy-based learning. Predicting structured data, 1(0), 2006.
Jason Lee, Elman Mansimov, and Kyunghyun Cho. Deterministic non-autooeegressive neural se-
quence modeling by iterative refinement. arXiv preprint arXiv: 2018.
Jason Lee, Raphael Shu, and Kyunghyun Cho. Iterative refinement the continuous space for non-autoregressive neural machine translation. arXiv preprint arXiv:2009.07177 2020.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to-sequeene pre- training for natural language generation, translation, and comprehension. arXiv preprint 1:Xiv:1910.13661 2019.
Zhuohan Li, Zi Lin, Di He, Fei Tian, Tao Qin, Liwei Wang, and Tie- Yan Liu. Hint-based training for non-autoregressive machine translation. arXiv preprint (.XXv:1999.067000 2019.
Kevin Lin, Dianqi Li, Xiaodong He, Zhengyou Zhang, and Ming- Ting Sun. Adversarial ranking for language generation. arXiv preprint (0ff:::770.100000 2017.
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining 2019.
approach. arXiv preprint
Iyya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv
preprint 1r0iv:1608. 0tt.i-6688.0398,,, 2016
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 171 101, 2017.
Xuezhe Ma, Chunting Zhou, Xian Li, Graham Neubig, and Eduard Hovy. Flowseq: Non-autoregreesivve conditional sequence generation with generative flow. arXiv preprint arXiv:1909.02488) 2019.
Matt Mahoney. Text 8 dataset. http: htwp://tttaaanneeeee/e/eeeeeeeee 2021-09-29.
2011. Accessed:
Gautam Mittal, Jesse Engel, Curtis Hawthorne, and Ian Simon. Symbolic music generation with diffusion models. arXiv preprint trXiv:2103.1609) 2021.
Alex Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. arXiv preprint NXV::2102.00672 2021.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311-318, 2002.
Matt Post. A call for clarity in reporting bleu scores. arXiv preprint arXiv: 1804.0877,, 2018.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv e-prints, 2019.
12

---

Published as a conference paper at ICLR 2022
Qiu Ran, Yankai Lin, Peng Li, and Jie Zhou. Guiding non-autoreegressive neural machine translation decoding with reordering information. arXiv preprint arXiv:1911.02215, 2019.
Marc' Aurelio Ranzato, Y-Lan Boureau, Sumit Chopra, and Yann LeCun. A unified energy-based framework for unsupervised learning. In Artificial Intelligence and Statistics, pp. 371-379. PMLR, 2007.
Ali Razavi, Aäron van den Oord, Ben Poole, and Oriol Vinyals. Preventing posterior collapse with valsa-vaes. arXiv preprint arXiv:/901.03416, 2019.
Danilo Rezende and Shakir Mohamed, Variational inference with normalizing flows. Interna- fional conference machine learning, pp. 1530-1538. PMLR, 2015.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and ap- proximate inference in deep generative models. In International conference on machine learning, pp. 1278-1286. PMLR, 2014.
Chitwan Saharia, William Chan, Saurabh Saxena, and Mohammad Norouzi. Non-autoregressive machine translation with latent alignments. arXiv preprint arXiv:2004.07337, 2020.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. arXiv preprint Xrxiv:5088.
2015.
Raphael Shu, Jason Lee, Hideki Nakayama, and Kyunghyun Cho. Latent-variable non- autoregressive neural machine translation with deterministic inference using delta posterior. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 8846-8853, 2020.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrimm thermodynamics. In International Conference on Machine Learn- ing, pp. 2256-2265, PMLR, 2015.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011 1..4466, 2020.
Mitchell Stern, William Chan, Jamie Kiros, and Jakob Uszkoreit. Insertion transformer: Flexible sequence generation via insertion operations. In International Conference on Machine Learning, pp. 1975-5985. PMLR, 2019.
Zhiqing Sun, Zhuohan Li, Haoqing Wang, Zi Lin, He, and Zhi-Hong Deng. Fast structured decoding for sequence models. arXiv preprint 1rX1v:1910.11555, 2019.
Ilya Sutskever, James Martens, and Geoffrey E Hinton. Generating text with recurrent neural net- works. In ICML, 2011.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998 6008, 2017.
Alex Wang and Kyunghyun Cho. Bert has mouth, and it must speak: Bert as markov random field language model. arXiv preprint NXXi:1992000094 2019.
Yiren Wang, Fei Tian, Di He, Tao Qin, ChengXiang Zhai, and Tie- Yan Liu. Non-autorregressive machine translation with auxiliary regularization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 5377-5384, 2019.
Lantao Yu, Weinan Zhang, Jun Wang, and Yong Yu. Seggan: Sequence generative adversarial nets with policy gradient. In Proceedings of the AAAI conference on artificial intelligence, volume 31, 2017.
13

---

Published as a conference paper at ICLR 2022
A
METHOD DETAILS
Using Markov property and the form of = fo(-1x=-1), for any > 0 we obtain
(x( = Σ Pt- (2-1111-1110 (0)
(5)
E1-1EX
Exx 10) (x-1))
(6)
By induction, can keep unrolling the chain for all steps t 1,t ...... eventually obtaining
Pe = Σ
1 aml
Ex
= E
x1mp1 (ea)
(x=1)]
(7)
X ---1-p--1 (-Xx--
which proves the from the model distribution (Eq. 1).
A.1 UPPER BOUND ON THE LOSS.
Using Jensen's 's inequality we can swap log and expectation Eq. 7 to obtain [log (x((xx=))))
-log [logo((((xo) -E (t(oo)))
(8)
x--1mp((xxx-)))
hence
(0) = -E xccpasta [oog(((((xx) S -E
x xeePaata ((b-g(-)2))
[log so([)xx-1))] (0)
(9)
xgxr((1x3)
xi fol-poo))
Xe-1 Sol fo(-(xx-2)
A.2 RELATION TO VAE
Variational Autoencoders (Kingma & Welling, 2013; Rezende et al., 2014) optimise the expected evidence lower bound (ELBO) on the marginal likelihood
log (xx 2 DDLL (x() + Ex)0((1)) [log p(x)))
(10)
with respect to parameters of both the variational distribution and generative distribution p, where Po remains the prior for for the latent variables. Denosing autoencoders limit the optimisation to the second term, assuming fixed encoder q, which also the case with SUNDAE: identifying encoder with our corruption distribution and generative distribution p with the distribution of our chain at the last step pr, expected ELBO becomes
Ex-pusta log Pr (x)]
Exxopaata (
[log (x x-~g(1x)
x~Pasta
Excopaat ( - (T).
(11)
where the second inequality is consequence of the upper bound 3. Hence, minimizing I(T) would lead to maximisation of the lower bound on the model likelihood.
A.3 CORRUPTION FUNCTION
Our corruption function is obtained in stages:
sample uniform expected corruption proportion a ~4([0, 1)).
sample Bernoulli mask, independently for each token m (Beroolli(aa)*
sample noise ~ (M((1,......u))
14

---

Published as a conference paper at ICLR 2022
Predicted langths
True longth
True transition
A sundac on crcam dessert
Locs
Langth predicter nutworr
sundlyy
Stop gracient
Enceder network
Decoder network
guewe greatly.
serthecion
Noley franudetion
Figure 3: Overview of target length prediction. During SUNDAE training, we simultaneously train the length predictor with cross-entropy loss and give ground-truth target length as input to the de- coder (green dashed arrow), thus teacher-forcing it. During sampling. we give the most likely target length prediction from the network the decoder (red dashed arrow).
compute corruption x" = Ca (2)) =(1 -11 -m) x + m (with multiplications taken element-wise)
The distribution (1(x obtained in this process covers the entire input space X, with probability skewed towards samples whose parts come from Paata- Note that except for corruption proportion, all steps are independent for all individual tokens. Given a, peo (y/(az) can be factorised as
II
RM (
x (1)
II (Qa(zz())))
where Q. aI
al +1 a transition matrix, V is matrix with all entries equal to 1. and h(a) denotes one-hot vector representation of E {1,2..... v}. In this context, corruption with rate a corresponds to a calliioomil-diiffssiii forward process with probability a of remaining in the same state, as proposed by (Hoogeboom et al., 2021).
B DETAILS OF TARGET LENGTH PREDICTION
We provide a simplified high-level overview of the target length predictor in Figure 3. In the rest of this section we focus on implementation details.
The target length prediction module consists of 6 residual blocks. The length prediction module takes the source encodings h = encoder (a:) as the input, where a sequence of source tokens. We first project h into a vector length prediction module. We add source length embedding vector Vien = S(Naauree xd (Ix) e and obtain the final encoder state representation e'z U2 + Ulens where la is the number of the source tokens, and Nsource = 128 is the maximum number of the source tokens. The target length prediction module then takes Va as the input and predicts the (downsampled) target length la- We found beneficial downsampling the target lengths for the length prediction by a factor of 2, ie. la [1/2] as compared to exact prediction of I; since the maximum number of target tokens is = 128, the maximum size of the outcome of length prediction is N N.=64.. 64.
E RdL., where d, 128 is the hidden size of the target
We prepend an embedding hen E Reane of target length to the source encodings h Nowere x dene for each source example æ and allow the decoder to attend to (dene here denotes the dimensionality of the encoder). heen is obtained by applying an embedding matrix V V(N,xdene)) to ground truth (downsampled) length la at training time, or to predicted one is during sampling. Note that optimisation of the length classification loss does not affect the encoder parameters.
C EFFECTS OF TARGET LENGTH PREDICTION
It is shown empirically that the length of the translated results affect BLEU score. Approximate search methods such as beam search usually incorporate a length normalization term order to pre- vent the search algorithm to prefer shorter sentences. This due to the fact that the scoring function
15

---

Published as a conference paper at ICLR 2022
BLEU
step
step
EN-DE with and without length prediction
(b) DE-+NN with and without length prediction
(15)
with lngén prodedion leuth oedicion
114)
119)
step
step
(c) EN >FR with and without length prediction
(d) EN DEE with increasing number of unrolls
Figure 4: Validation BLEU curves during training with and without target length prediction
WMT 14 EN-DE, DE-EN and EN-FR tasks shown in Figure 4a, 4b and 4c, respectively. Fig- ure 4d shows the effect of unrolled denoising terms on translation quality, here L(k) = [11kk) All models trained with batch size of 256 using 20 different random seeds display uncertainty.
Source
Ich habe dort Dinge gesehen, aus denen ich gestärkt hervorgegangen bin.
Initialization
Thir Martcl moderatopposed frameworks solidierweitert [...] saw things there that which stronger.
Step
Step
saw things there from which stronger.
Step
saw things there from which emerged stronger. saw things there from which emerged stronger.
Step
Reference saw some things there and came out stronger.
Table 5: German-te to- no-English translation process. Since initialization is quite long, we substitute the trailing tokens with [...]. Tokens changed from previous step are highlighted in gray. The process converges after 3 steps, while AR would take 10 steps (one for each token in the translation).
usually chosen as the likelihood modelled by the translation system, and beams containing shorter sequences tend to score higher likelihoods. In non-AR models, the end of sequence is not explicitly learned by the models, thus they can benefit by knowing the target sequence length in advance to generating the translation. However, the length of the target sequence cannot be known at inference time, therefore, model should predict instead. We show how BLEU scores can differ in WMT" 14 translation tasks with and without the target length prediction in Figure 4a-4c.
D EFFECTS OF UNROLLED LOGITS LOSSES
We show how varying the number of unrolled logits affect the MT performance in Figure 4d.
16

---

Published as a conference paper at ICLR 2022
ISTONECE
14000 Score
Sampling
(a) Model score evolution throughout sampling.
(b) BLEU cvolution throughout sampling.
Figure 5: Model score (left) and BLEU (right) evolution throughout sampling for various tempera- tures. The reference score on the left hand side denotes the model score obtained for ground truth inputs. Overall, medium-low temperatures give the best results.
E TRANSLATION ANALYSIS
We show how our algorithm works step-by-step for translation in Table 5. is interesting to ob- serve how the multimodality problem (Gu et al., 2017) gets resolved with more steps. Multimodality usually manifests in repeated tokens in translation, which intrinsically comes from inability to coor- dinate decisions when independently sampling multiple tokens once. However, after few steps all repetitions are corrected, because the model conditions on the previous step and hence can better coordinate further changes.
Figure shows how the model and the BLEU scores evolve during sampling for various tempera- tures. The model score here refers to the cross-entropy between translation and model logits pro- duced by giving this translation as the input. While for very low temperatures the scores initially improve faster, they are eventually outperformed by scores for higher temperatures, possibly due to low diversity induced by -etermeeeriiiiiicc sampling. For very high temperatures, on the other hand, the scores improve slowly, and hence using such temperatures is rather impractical.
F DISTILLATION SCORES OBTAINED WITH TRANSFORMER -BASE
In Table 6 we also present results for SUNDAE trained on data distilled from autoregressive Transformer- Baaee model, which allows direct comparison with similarly obtained Imputer scores (Saharia et al., 2020). The results show that SUNDAE noticeably outperforms Imputer 8 steps for both EN-DE and DE-EN language pairs.
AR-distileed BLEU
Model
Steps (7)
EN-
DE ENN
Imputer (Saharia et al., 2020) (n= 1) Imputer (Saharia et al., 2020) (n (n=1)
27.9
30.9
27.9
31.1
SUNDAE (63M)
Deterministic (n = 16)
4
28.00
31.82
Deterministic (n 16)
28.01
31.84
Deterministic (n = 16)
10
28.01
31.84
Stochastic (n = 16)
27.78
31.42
Stochastic (n 16)
8
28.14
31.78
Stochastic (n = 16)
10
28.11
31.87
Stochastic (n 16)
16
28.11
31.87
Stochastic (n 16)
32
28.02
31.92
Table 6: Test BLEU scores of SUNDAE and Imputer (Saharia et al., 2020) distilled from AR Transformer-Bass model on Baglish-to-Germee (EN+DDE) and German-to-Eal Enngish (DE~>EN) translation tasks.
17

---

Published as a conference paper at ICLR 2022
G SACREBLEU SCORES FOR WMT 14 EXPERIMENTS
SacreBLEU (Post, 2018) is library for computing BLEU score' that does not require the user to manually tokenize the reference and candidate translations. We report BLEU scores measured with SacreBLEU library in order to provide reference for future studies. We show all the BLEU scores Table 7.
EN-DE
DE~EN
EN-FR
Model
Steps (T)
BLEU
BLEU
BLEU
BLEU
BLEU
BLEU
Deterministic (n (n=10 16)
2
20.29
20.13
25.28
24.83
31.60
30.51
Deterministic (n= 16)
3
24.07
23.84
28.64
28.12
35.88
34.68
Deterministic (n (n=10 16)
4
25.01
24.75
29.53
28.99
36.85
35.61
Deterministic (n (n= (n=16)
10
25.54
25.25
30.11
29.54
37.15
35.92
Stochastic (n 16)
4
23.05
22.75
28.13
27.62
35.27
34.03
Stochastic (n 16)
8
26.22
26.08
30.48
30.00
37.45
36.16
Stochastic (n 16)
10
26.25
25.99
30.80
30.24
27.53
36.23
Stochastic (n 16)
32
26.57
26.31
30.74
30.11
37.60
36.20
Table 7: Test BLEU without AR distillation (raw). Scores computed with SacreBLEU library are
denoted as BLEU
H UNCONDITIONAL SAMPLES FOR C4
We provide samples for our model trained on C4 dataset as described in the main text (Section 3.2). All of them resemble reasonable-quality internet texts, except perhaps sample #4.
Index
froes model
had bock and forth und what had happened, what working and all in the right places.
fossed delicious
Beet beisket
thick "Wild Herring', although very often achieve
make lot
expect
"Heming".
cousidered
Delocah Chicnnn 365 days, hesdlsches, stemach pain,
priming photo postors.
Jeneyy, Prilsdelphhi and Washington This conferance attended by residents, physicions, graduste maliccl students, public
This researh was suppocted by grast National Association Advancenentt Science thouugh Depestment Energy
you piece our shop!
Keep eye on Gallery you have do
from 5883 ropected by Zacka.com Februsay.
Brad Brickley; genemal monager Trvine, Calif-bbsed
10
to writing with the Autarnss Together, some of the chase spend time che classroom to read a poems from the original book. From thene
Table 8: Unconditional samples from our model trained C4 (without cherry-picking). Since C4 was crawled from the web, newline symbols are abundant both in the training data and the samples.
18

---

Published as a conference paper at ICLR 2022
I UNCONDITIONAL SAMPLES FOR EMNLP2017 NEWS
We provide samples from our model trained on EMNLP2017 News dataset Table 9, accompany- ing the quantitative results in the main text (Section 3.2). These samples obtained at temperature 0.8. None of them appear in the training set, so the model does not merely memorize the data it is given. To provide point of reference, also include samples from ScratchGAN (d' Autume al., 2019).
Index
Sample
ScratdGGAN
We pleased for the trist and it incrediblle joh quickly leamn the shope and that puys, maybe Melhourne. players that Keith also held 2005 and
ahvionsly
quiite
govenmmet saccessfal campaign spokeswoman for students and young brothers has took advantage of operator,
suciety to secure
ecunomy
Police sai how Democratic police officer, would choose the honor of alcohal and reduce his defense and foundation.
We de not the Blues because that spent in ten months and so didn have great job big revolution
The 28 old Dr Price said she would have been invited to Beiaiin her friend " lovely family.
And long
about families coming
friend
SUNDAE
by They been around long time. but too natural to ighore what has hoppened.
Starpoon was quoted as saying
prepared quit
next year..
"I a busy roud, and we ve got get through it. he said.
Thar means some voters will docide they II change their minds before the first day of the dehote " We don leam much from
view because
" said..
"I spent whole
stage
Whether your phone near online, you can check your phone
to your Instagrmm account,
is only too early to know why the incident occurred shortly after happened. "he told the Associated Press
The websise will be updatted on day in day basss as well as other special services. So too early to know exactly what each condicate has say terms the Amenica election.
10
Table Unconditional samples from our model SUNDAE trained on EMNLP201 News (without son.
We also provide samples from ScratchGAN (d' Autume et al., 2019) for compari-
J PSEUDOCODE OF SUNDAE
We provide Python-like pseudocode for our method in Listing 1. The main functions are build.loss.fn, which should be used for training, and sampl ing_fn which should be used for sampling. For simplicity, we consider the decoder-only setup as in unconditional generation experiments (Section 3.2).
def get_randcm_tte (shape, vocab_size): random_text rand_int (shape, minval-0, mxxaal-vccabsiiee return random_text.
def corrupt_text (batched_ vccab_size)
rand prrrutiion_rrb_per__see mask rand corruption_proo_pr__eeee random_text. get_randone text (batched_teex) ttt.ttetttt......eeeee vocab _ocab_size)
rand__niform ([batched_toxt) shape [0], 11) shape)
return corrupted 10
n corrupted mask random_text (1 mask) batched_text.
14
15
def Quila_logite fn (vocab, (vocab size, a unrolled Saable_sampling]
16
def logits (fattss_f(natthheeteee
18
model Transformer vvocab_size, use_cuusal_mskraalsse
19
20
def fn (input___m www.baacchheeeeeeee
logits = model return logits
text)
23
24
def unrolled
25
samples corrupt_text (anput vocab_size)
al ali_logits - []
for in range (n
19

---

Published as a conference paper at ICLR 2022
logits En (samples)
29
samples stop_grad (rand_categoricall (logits))
30
all_logits += [logits]
final_logite concatenate (all_logits) axis=0)
return final_logite
34
enable_sampling:
35
return in (bat else:
36
return unrolled (patched_text)
39
return logits_fn
40
def build loss fn (vocab size, Lunoolle_ssteps_ logits_fn fn build_logits fn vocab_size, n_unrolled_stepes mnable_sampling-ral...
43
44
def loss fn (batched_text) 45
logits logits_fn (batcced_etch)))
48
repeat batched_text to fit unrolled logits
49
targets - concatenate ([batched_texte one one hoa_gtetttgssst = one one_het (targets, sizeb Loss_per_tokee pekeeetteeeeneeeeeeeeeeeeeeeeeee -sum
L_unrolled_step axis-0)
50
52
one hot targets hlg_ssotttee (logits), axis=-1) loss
53
mean plassppos_perrtooee) return loss
54
return lcss_fn
57
58
59
def fampling__nnoogiiss__ooo steps, temperature, batch_size,
batched_text sequence__ength,, vocab_size)
61
62
shape- (batch_size, sequence_length), [ocab_size)
63
for in range (steps))
logits logits_ (batched__eex) _tetted___ette______
samples rand_categorical (legits temperature)
66
batched text samples
67
return batched text
Listing 1: Pseudocode of SUNDAE.
20


Reviews:
[{"review_id": "oQ-db_w1Qhs", "reviewer": "Reviewer_7uVK", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Non-AR generation is a hot topic recently, this paper proposes a new NAR method from the perspective of denoising autoencoder. The related works are well-organized and discussed thoroughly.\n\nBut the following flaws prevent me from giving a higher score.\n-  The contribution seems incremental. The one-step denoising resembles the training of BERT and the related NAR method mask-predict (Ghazvininejad et al., 2019). And the two-step denoising, which is the main contribution of the paper, also resembles an improvement of mask-predict [1] which conducts one-step generation while training and takes the result as the model output. Can you compare it and claim the difference and the contribution of this paper?\n- For NAR models, the decoding speed (or latency) is a crucial evaluation metric, which is missed in this paper. From Table 2, we can observe that the proposed method needs 32 decoding steps to obtain good results. In my experience, the average length of sentences in WMT14 En-De lies around 20-30, which means that the proposed method is slower than the AR counterpart in a majority of cases. Can you report the decoding speed in Table 2 to construct a complete comparison?\n- Obtaining good results w/o kd is an advantage of the method. But I think it is worthy to utilize kd if it can provide better results. Does kd brings benefits to your method and why? KD alleviates the multi-modality problem in the dataset so that the NAR model can be more easily trained. Your method may also achieve a similar effect. Therefore I suggest authors add an investigation w.r.t the method's capacity of alleviating the multi-modality problem.\n- As for the experiments of unconditional text generation, I suggest the authors provide quantitive evaluation scores to provide a clear comparison. \n\n\n[1] Semi-Autoregressive Training Improves Mask-Predict Decoding, Marjan Ghazvininejad et.al., 2020", "summary_of_the_paper": "The paper proposes an unrolled denoising method for non-autoregressive text generation. Specifically, the method first corrupts the original input and then re-generate uncertain tokens in it, and the result is taken as the output of the denoising autoencoder to compute loss with the target. Experiments are conducted on machine translation and unconditional text generation. The method achieves good results on MT without utilizing knowledge distillation.", "summary_of_the_review": "The paper studies NAR generation, which is an interesting topic. The proposed method achieves good results, but the flaws on contribution/evaluation prevent me from giving a higher score.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Ukvo3GeM-lw", "reviewer": "Reviewer_Rdkb", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\nThe present approach (and approximations) are well motivated and show decent improvements over baselines on machine translation experiments. \n\nWeaknesses/Questions:\n1. While the improvements on \"raw BLEU\" are good, it is not clear why raw BLEU is interesting. If distilled models are able to provide good quality translation, why are methods to train from scratch required for conditional generation tasks like MT? \n2. The motivation in the introduction does say that for unconditional generation methods, distillation is not possible where this method can be applied. To support this claim, some quantitative results should be provided (quality/diversity of the generated text) similar to what text GANs work report.\n3. How sensitive is this approach to the heuristics and their related hyperparameter values (such as \\rho, temperature etc)? Do they need to be tuned for every dataset/language pair for example? \n4. The authors should also present an analysis of decoding speed of this method compared to baselines.", "summary_of_the_paper": "This paper presents a training method of non-autoregressive text generation that is trained from scratch without distillation. The method is motivated as an unrolled denoising approach, which starts with a sequence of random tokens and transforms it in multiple stages to a valid sequence. To make training feasible, the proposed approach starts with a corrupted target sentence instead of a totally random sequence and does only two steps of unrolling. The training loss is an approximation of this process and is computed as a sum of loss on direct denoising and two steps of denoising. During inference, it starts with a random sequence and applies several steps of denoising to arrive at the desired sequence. Several heuristics are also applied to make generation feasible. The main result of the paper on non-autoregressive machine translation on which the authors show decent improvements over baselines on non-distilled settings however slightly underperforms AR distilled models. The authors also show examples for the case of unconditional generation and text in-painting", "summary_of_the_review": "The paper shows good improvements on non-autoregressive translation without distilling from trained models, however, it is not clear why that is something to strive towards. The approach uses several heuristics and it is again not clear how sensitive this approach is to their values.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "TV06z_Jex32", "reviewer": "Reviewer_D77g", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "#### **Strengths**:\n1. The proposed method achieves new state-of-the-art performance in non-autoregressive machine translation tasks.\n2. The paper is written well and easy to understand.\n\n**My concerns concentrate on the experiments**.\n1. Missing several important results. For example, a) efficiency analysis in machine translation tasks, we all know that deeper iterations improve the performance at the cost of the decoding efficiency, b) mere quantitative results in unconditional text generation tasks are helpless to understand whether it works well.\n2. Some comparisons could not be fair—for example, the comparison with the Imputer model in Table 2. Although the performance improvement over the Imputer model is impressive, the proposed method (SUNDAE) jointly performs iterative decoding and parallel rerank techniques. However, the Imputer model does not use reranking techniques. In addition, we also notice that the Imputer model only needs fewer iterations.\n\n#### **Questions**:\n1. What do $x^{(N)}$ and $f_{\\theta}^{(N)}$ in equation (2) mean?\n2. Can the author use a picture to illustrate the target length prediction? It is also not clear to me why a different (to previous works) design is needed for length prediction.\n3. Table 2 does not include the results of SUNDAE trained without knowledge distillation (KD). However, most baselines do not include performance without the KD. Why not report the SUNDAE's performances trained with the knowledge distillation for a comprehensive comparison?\n\n**After reading the author's response, most of my concerns have been resolved in the revision. Therefore, I would like to increase my score by 6.**", "summary_of_the_paper": "This paper proposes a new training technique for an iterative non-autoregressive (NAR) model and achieves significant improvements over the previous model. Inspired by the success of the diffusion model, the author formalizes the iterative NAR decoding process with a denoising autoencoder framework, then unrolls the denoising process during training. The proposed method (SUNDAE) achieves new state-of-art performances in machine translation by deep iteration decoding and parallel reranking. More experiments on unconditional text generation tasks further show its promising capability.", "summary_of_the_review": "This paper proposes a new state-of-the-art performance in non-autoregressive translation without the help of an autoregressive teacher for training. However, some concerns about the experiment still need to be addressed to understand the contribution better.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "B_Qr5Wth00e", "reviewer": "Reviewer_D7fL", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed method is well-motivated and looks reasonable: it provides better noisy samples in addition to the random corrupted ones for training the denoising autoencoder. It has promising performance on the EN->DE translation compared to several baselines. The effectiveness is confirmed by ablation studies.\n\nBesides, my concerns have two-fold. \n\nOne is regarding the evaluation in this paper. \n(1) I found the table that reports the main results is remarkably sparse. Although it lists a great number of baselines in comparison, almost all these baselines were evaluated only on AR-distilled BLEU, whereas the proposed models were only evaluated with Raw BLEU. This makes the many scores in Table 2 do not comparable. For baselines that have Raw BLEU score, only one baseline is available for DE->EN translation;\n(2) The results of the unconditional text generation experiment have never been quantitatively evaluated. This makes me feel that this experiment is only for illustrating \"how can the proposed model apply to language modeling tasks\" rather than \"how well is the proposed model on language modeling tasks\".\n(3) In the experiment section, the authors only talk about what they have done and what they have seen from the table. More explanations, interpretations, and in-depth analyses are welcome.\n(4) As the evaluation has only been done on only MT on a single language pair, it would be interesting to see how and how well the proposed model handles other language generation tasks. I personally view that MT is not a typical LG task as its outputs highly depend on its inputs. I wonder, for tasks that require planning and require the generated text to be diverse (in line with most text generation tasks), will such a denoising technique still work?\n\nThe other is that I found the present paper is a bit hard to follow. This is probably because, for example, variables are never defined before use, and, for example, many terms (e.g., step) refer to multiple concepts but sometimes can not be disambiguated given their contexts.", "summary_of_the_paper": "The present paper proposes a Step-untolled denoising autoencoder (SUNDAE), a NAR text generation model which marries de-noising auto-encoder and Markov chain models. The major contribution of this paper is the unrolled denoising training scheme, which is a training step method. In the first step, the original text is randomly corrupted in the same way as what the denoising autoencoder did. Subsequently, in the second step, the noisy text from the first step is first denoised by sampling from the model, and the denoised text is then used as a sample for training the denoising autoencoder. \n\nThe proposed model has been tested on two tasks that involve text generation: machine translation as well as language modeling. ", "summary_of_the_review": "The proposed model is interesting, reasonable, and well-motivated. It has promising performance in ablation studies. Nevertheless, the evaluation part has clear flaws and the paper itself is sometimes hard to follow.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces a simple technique to improve non-autoregressive generation by training the model to reconstruct model-perturbed inputs in addition to inputs perturbed by a fixed noise source. 

Despite interest in the paper, we were worried about a number of aspects missing from section 3. During the rebuttal phase, however, the authors addressed most if not all comments and the section is now rather complete.

For its clarity, and the interesting results, we are recommending this version for acceptance. 

A comment on presentation:

The paper attempts to establish a connection with variational diffusion models, but the connection does not seem strong enough at this point. In a variational diffusion approach, the forward view would not involve $f_\theta$, for example. Also, given that the distribution of $\mathbf x_t$ depends on $\theta$, the gradient estimator used in the paper is a heuristic, and I'd like to ask that the authors emphasise this clearly and early on in the draft.
Decision: Accept (Poster)