Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper studies an edit-based generative text model that starts with a complete noise distribution (random gibberish) as input and then produces a series of edits to reach high-quality output. Inspired by diffusion models in CV, this Diffuser model rivals or outperforms standard autoregressive models on various generation tasks (MT, summarization), while also providing additional editing-based functionality.
strength_and_weaknesses: 
Strengths:
* The objective and model design is quite novel and refreshing. I appreciate the careful design of the architecture---rather than focusing on a pure end-to-end system is decomposes the task into edit tagging and generation.
* The training objective seems sensible, and the overall evaluation is pretty solid. I specifically appreciate using the model as a sort of "general-purpose post-editing" model (as shown in Diffuser + AR bootstrap). It would be nice to consider more post-editing-like baselines, e.g., just adding some off-the-shelf post editors or grammatical-error correction models to the MT output.

Weaknesses:
* While sensible, the training objective is a bit limited in that it denoises pure random tokens. Concretely, to get good performance at test time, the model needs to denoise generations that have various errors spanning semantic errors, syntactic inconsistencies, etc. These types of errors are quite far from swapping in random tokens in the input.
* The editing-based evaluation is a bit limited. It would have been great to explore more some of the capabilities/failures of the model. For example, conditioning on various keywords on the target side, conditioning on a target syntactic style, or similar evaluations.
clarity,_quality,_novelty_and_reproducibility: The paper is very clearly written. The work is quite original. The overall experiment evaluations are quite high quality and comprehensive.
summary_of_the_review: I think methods for non-traditional forms of generating text are widely underexplored, and this paper proposes a method that rivals typical autoregressive generation methods that are commonplace today. The overall writing and paper quality is solid.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: Motivated by how humans revise content and the success of diffusion models with continuous inputs, the authors propose a generative model of text based on multiple editing steps, with each step based on one or more text span editing operations (insert, delete, replace, keep). The generative model is trained to invert, step by step, the random edits of a "text diffusion process", which is specified by a prior over edits and edit length. Generation at each step is further decomposed into a model for generating the edit operations to apply at each position of the current input text, and a model for generating text given the input and the selected edit operations, similarly to many existing edit-based generation approaches.

Results on machine translation (WMT 14') and summarization (CNN Daily Mail) show competitive performance (Table 1). An advantage of the proposed Diffuser model can refine text and so can be boostrapped by an autoregressive translation or the source to summarize, which boosts performance (Table 1). The authors also show that by simply training 2 models for positive and negative sentiment Yelp reviews and then using them to transform the input text to the target sentiment, performance is competitive with SOTA task specific models, which is impressive. Some ablations around decoding method vs. speed and performance and seed text type are also included.
strength_and_weaknesses: Strengths

- The model is intuitively appealing, and the paper in general is well written.
- The experiments demonstrate competitive performance, and the advantages of being able to bootstrap and refine results.

Limitations

- While the paper well written, and establishes the approach well, it could benefit tremendously from some additional supplementary material, to give the paper more depth, as discussed in the following points.
- The work is well executed and the model intuitive, but in context, an obvious next step given current work on iterative text generation models, and the recent success of diffusion models. More credit should be given to previous related work that establishes similar reconstruction and corruption processes, similar two-stage editing, and similar decoding processes. These are not novel components of this paper as suggested by the current manuscript. Related, I'd like the authors, to the extent possible, to discuss the similarities/differences advantages/disadvantages over similar text editing models, such as the Levenshtien transformer. Why would it not perform as well as Diffuser?
- More flexible control over generation is claimed as an advantage of the model, but this is really never demonstrated or exercised. While I can imagine variations of the model that could deliver this, the models investigated are trained and utilized end-to-end, with intermediate generations that have no notion of validity associated with them, and internal decisions that, if manipulated, would likely degrade performance.
- Related, investigating the generalization of and effects of manipulating the editing priors at test time to exercise some control and/or diversify outputs would be an interesting ablation/demonstration.
- Statements like "editing processes can also be used to calculate the probability of only the final document while taking into account previous revisions, which is not possible in the traditional text generation setup", are misleading. Under the model intermediate revisions are meaningless, and conventional LMs can evaluate likelihood in a single pass (i.e., having to go through a revision process for likelihood evaluation is a disadvantage of the model). Generally speaking, a more balanced and frank presentation of the strengths and limitations of the approach, properly situated within context with previous work, would improve the paper signficantly. 
- Equation 5 has errors, please correct.
clarity,_quality,_novelty_and_reproducibility: See S&W section.
summary_of_the_review: Overall, the presented text-edit based diffusion model is much anticipated, well motivated, and has been well executed. However, as detailed in the limitations section, more credit and context as it relates to previous work should be included when presenting the elements of the model, and both the strengths and limitations of the approach should be frankly discussed, in supplementary material if necessary, to give the paper more context and depth. If feasible, additional experiments that investigate the extent that generations can be flexibly controlled would further strengthen the paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes DIFFUSER, a denoising diffusion model for text generative tasks. It treats text generation as a Markov chain of Levenshtein edit steps to denoise from the initial text. An editing step is modeled as an editing process in existing work (Reid & Neubig). 

The contribution of the paper is mainly 1) using the more flexible edit operations in diffusion models; 2) more thoughtful bootstrapping from autoregressive output or source text as initial text sequence; 3) employing 2D-beam search during decoding. 

strength_and_weaknesses: Pros:
- Well motivated, provided valuable exploration in the recent popular direction of diffusion models in NLP area.
- Very effective diffusion approach in NLP by adopting flexible editing operations.
- Strong results in experiments on multiple tasks and datasets, compared against strong baselines including recent literature on non-auto-regressive generation models. 
- Effective techniques improved on top of the diffusion approach, including bootstrapping and 2D-beam. Ablation and analysis are also provided.

Cons:
- There is novelty in the approach, but it is not considered significant, since there is already existing work on NLP diffusion models as described in Sec 2.3.
- The description of models and experiment settings is not clear enough. Table 4 could be moved to Appendix to make room for clearer and more detailed descriptions and explanations.
  1. Since the tasks are sequence to sequence generation, the Diffuser model can be looked on as the decoder, should there also be an encoder to take in the input text, so that the decoder can condition the output on the input? This was not described anywhere in the paper.
  1. How many diffusion steps, and what b and r in Summarization and Text Style Transfer tasks? 
  1. Equation (3) is not explained clearly, including some symbols not annotated.
  1. There is no description of the Accuracy metric in Table 2.
- It would be helpful if some experiments and analysis are included for the number of diffusion steps in *training*. 
- Numbers of training time and decoding time compared to baselines would be helpful, together with some discussion on the trade-off of efficiency and efficacy.

Minor edit:
Page 8: "comparsion" -> "comparison"

clarity,_quality,_novelty_and_reproducibility: The main approaches and contributions of the paper is clearly presented and well organized. However, more clarity is needed on some details in the descriptions of models and experiments. It did not affect the claim of the paper, though.

The approaches and claims are solid and sound, backed up by experiment results.

There is novelty in the approach, but it is not considered significant, since there are already existing work on NLP diffusion models as described in Sec 2.3.

The authors stated that "Code and data to reproduce experiments will be released".

summary_of_the_review: The paper is well-motivated and provided valuable exploration in the recent popular direction of diffusion models in NLP area. The approach, although not strong in novelty, effectively employed various techniques to obtain strong experiment results on multiple tasks. The presentation is mostly clear and well organized, although there are multiple places that need to be clarified or further explained and discussed.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The idea and the method proposed in this paper is novel and all reviewers appreciated the model design. The method can be a general purpose edit method. The evaluation seems solid although some reviewers found it limited and could be compared with other edit-based approaches like GEC (MpCU). Reviewer Gnic also recommended to describe its similarity/dissimilarity and advantages/disadvantages with other similar method. Overall, considering its strength, I am in favour of accepting the paper. 
justification_for_why_not_lower_score: It's a good paper
Decision: Accept: poster