Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper presents a training method of non-autoregressive text generation that is trained from scratch without distillation. The method is motivated as an unrolled denoising approach, which starts with a sequence of random tokens and transforms it in multiple stages to a valid sequence. To make training feasible, the proposed approach starts with a corrupted target sentence instead of a totally random sequence and does only two steps of unrolling. The training loss is an approximation of this process and is computed as a sum of loss on direct denoising and two steps of denoising. During inference, it starts with a random sequence and applies several steps of denoising to arrive at the desired sequence. Several heuristics are also applied to make generation feasible. The main result of the paper on non-autoregressive machine translation on which the authors show decent improvements over baselines on non-distilled settings however slightly underperforms AR distilled models. The authors also show examples for the case of unconditional generation and text in-painting
main_review: Strengths:

The present approach (and approximations) are well motivated and show decent improvements over baselines on machine translation experiments. 

Weaknesses/Questions:
1. While the improvements on "raw BLEU" are good, it is not clear why raw BLEU is interesting. If distilled models are able to provide good quality translation, why are methods to train from scratch required for conditional generation tasks like MT? 
2. The motivation in the introduction does say that for unconditional generation methods, distillation is not possible where this method can be applied. To support this claim, some quantitative results should be provided (quality/diversity of the generated text) similar to what text GANs work report.
3. How sensitive is this approach to the heuristics and their related hyperparameter values (such as \rho, temperature etc)? Do they need to be tuned for every dataset/language pair for example? 
4. The authors should also present an analysis of decoding speed of this method compared to baselines.
summary_of_the_review: The paper shows good improvements on non-autoregressive translation without distilling from trained models, however, it is not clear why that is something to strive towards. The approach uses several heuristics and it is again not clear how sensitive this approach is to their values.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new training technique for an iterative non-autoregressive (NAR) model and achieves significant improvements over the previous model. Inspired by the success of the diffusion model, the author formalizes the iterative NAR decoding process with a denoising autoencoder framework, then unrolls the denoising process during training. The proposed method (SUNDAE) achieves new state-of-art performances in machine translation by deep iteration decoding and parallel reranking. More experiments on unconditional text generation tasks further show its promising capability.
main_review: #### **Strengths**:
1. The proposed method achieves new state-of-the-art performance in non-autoregressive machine translation tasks.
2. The paper is written well and easy to understand.

**My concerns concentrate on the experiments**.
1. Missing several important results. For example, a) efficiency analysis in machine translation tasks, we all know that deeper iterations improve the performance at the cost of the decoding efficiency, b) mere quantitative results in unconditional text generation tasks are helpless to understand whether it works well.
2. Some comparisons could not be fair—for example, the comparison with the Imputer model in Table 2. Although the performance improvement over the Imputer model is impressive, the proposed method (SUNDAE) jointly performs iterative decoding and parallel rerank techniques. However, the Imputer model does not use reranking techniques. In addition, we also notice that the Imputer model only needs fewer iterations.

#### **Questions**:
1. What do $x^{(N)}$ and $f_{\theta}^{(N)}$ in equation (2) mean?
2. Can the author use a picture to illustrate the target length prediction? It is also not clear to me why a different (to previous works) design is needed for length prediction.
3. Table 2 does not include the results of SUNDAE trained without knowledge distillation (KD). However, most baselines do not include performance without the KD. Why not report the SUNDAE's performances trained with the knowledge distillation for a comprehensive comparison?

**After reading the author's response, most of my concerns have been resolved in the revision. Therefore, I would like to increase my score by 6.**
summary_of_the_review: This paper proposes a new state-of-the-art performance in non-autoregressive translation without the help of an autoregressive teacher for training. However, some concerns about the experiment still need to be addressed to understand the contribution better.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes an unrolled denoising method for non-autoregressive text generation. Specifically, the method first corrupts the original input and then re-generate uncertain tokens in it, and the result is taken as the output of the denoising autoencoder to compute loss with the target. Experiments are conducted on machine translation and unconditional text generation. The method achieves good results on MT without utilizing knowledge distillation.
main_review: Non-AR generation is a hot topic recently, this paper proposes a new NAR method from the perspective of denoising autoencoder. The related works are well-organized and discussed thoroughly.

But the following flaws prevent me from giving a higher score.
-  The contribution seems incremental. The one-step denoising resembles the training of BERT and the related NAR method mask-predict (Ghazvininejad et al., 2019). And the two-step denoising, which is the main contribution of the paper, also resembles an improvement of mask-predict [1] which conducts one-step generation while training and takes the result as the model output. Can you compare it and claim the difference and the contribution of this paper?
- For NAR models, the decoding speed (or latency) is a crucial evaluation metric, which is missed in this paper. From Table 2, we can observe that the proposed method needs 32 decoding steps to obtain good results. In my experience, the average length of sentences in WMT14 En-De lies around 20-30, which means that the proposed method is slower than the AR counterpart in a majority of cases. Can you report the decoding speed in Table 2 to construct a complete comparison?
- Obtaining good results w/o kd is an advantage of the method. But I think it is worthy to utilize kd if it can provide better results. Does kd brings benefits to your method and why? KD alleviates the multi-modality problem in the dataset so that the NAR model can be more easily trained. Your method may also achieve a similar effect. Therefore I suggest authors add an investigation w.r.t the method's capacity of alleviating the multi-modality problem.
- As for the experiments of unconditional text generation, I suggest the authors provide quantitive evaluation scores to provide a clear comparison. 


[1] Semi-Autoregressive Training Improves Mask-Predict Decoding, Marjan Ghazvininejad et.al., 2020
summary_of_the_review: The paper studies NAR generation, which is an interesting topic. The proposed method achieves good results, but the flaws on contribution/evaluation prevent me from giving a higher score.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The present paper proposes a Step-untolled denoising autoencoder (SUNDAE), a NAR text generation model which marries de-noising auto-encoder and Markov chain models. The major contribution of this paper is the unrolled denoising training scheme, which is a training step method. In the first step, the original text is randomly corrupted in the same way as what the denoising autoencoder did. Subsequently, in the second step, the noisy text from the first step is first denoised by sampling from the model, and the denoised text is then used as a sample for training the denoising autoencoder. 

The proposed model has been tested on two tasks that involve text generation: machine translation as well as language modeling. 
main_review: The proposed method is well-motivated and looks reasonable: it provides better noisy samples in addition to the random corrupted ones for training the denoising autoencoder. It has promising performance on the EN->DE translation compared to several baselines. The effectiveness is confirmed by ablation studies.

Besides, my concerns have two-fold. 

One is regarding the evaluation in this paper. 
(1) I found the table that reports the main results is remarkably sparse. Although it lists a great number of baselines in comparison, almost all these baselines were evaluated only on AR-distilled BLEU, whereas the proposed models were only evaluated with Raw BLEU. This makes the many scores in Table 2 do not comparable. For baselines that have Raw BLEU score, only one baseline is available for DE->EN translation;
(2) The results of the unconditional text generation experiment have never been quantitatively evaluated. This makes me feel that this experiment is only for illustrating "how can the proposed model apply to language modeling tasks" rather than "how well is the proposed model on language modeling tasks".
(3) In the experiment section, the authors only talk about what they have done and what they have seen from the table. More explanations, interpretations, and in-depth analyses are welcome.
(4) As the evaluation has only been done on only MT on a single language pair, it would be interesting to see how and how well the proposed model handles other language generation tasks. I personally view that MT is not a typical LG task as its outputs highly depend on its inputs. I wonder, for tasks that require planning and require the generated text to be diverse (in line with most text generation tasks), will such a denoising technique still work?

The other is that I found the present paper is a bit hard to follow. This is probably because, for example, variables are never defined before use, and, for example, many terms (e.g., step) refer to multiple concepts but sometimes can not be disambiguated given their contexts.
summary_of_the_review: The proposed model is interesting, reasonable, and well-motivated. It has promising performance in ablation studies. Nevertheless, the evaluation part has clear flaws and the paper itself is sometimes hard to follow.
review_title: null

Meta-review: The paper introduces a simple technique to improve non-autoregressive generation by training the model to reconstruct model-perturbed inputs in addition to inputs perturbed by a fixed noise source. 

Despite interest in the paper, we were worried about a number of aspects missing from section 3. During the rebuttal phase, however, the authors addressed most if not all comments and the section is now rather complete.

For its clarity, and the interesting results, we are recommending this version for acceptance. 

A comment on presentation:

The paper attempts to establish a connection with variational diffusion models, but the connection does not seem strong enough at this point. In a variational diffusion approach, the forward view would not involve $f_\theta$, for example. Also, given that the distribution of $\mathbf x_t$ depends on $\theta$, the gradient estimator used in the paper is a heuristic, and I'd like to ask that the authors emphasise this clearly and early on in the draft.
Decision: Accept (Poster)