Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper seems to be the first application of diffusion models for (discrete) language data. To deal with the discrete nature, they propose, to my understanding, a softmax layer that maps dense vectors into a distribution over the vocabulary. Another proposed technique is to force the neural network to predict x_0 in every term. The authors test the proposed diffusion models on several datasets and show better results than a set of baseline methods.
strengths_and_weaknesses: Strengths:
1. The proposed methods seems effective compared to previous plug-and-play controllable generation approaches.
2. People may be interested in such a successful application of diffusion methods in NLP.
3. The proposed methods are simple and well-examined by ablation studies.

Weaknesses:
1. The report of computational cost is missing (though the authors mentioned that the decoding is substantially slower).
2. I feel it is better to show some human evaluation results regarding the results in Table 1. 
3. Might not be a (major) weakness. But I have a hard time reading this paper. The writing is very abstract. I feel important deduction and concrete design are always missing.
questions: 1. Lots of important details are given in Appendix, including the modeling choices for realizing all functions and I really do not know how the classifiers for these control tasks are implemented. I also don't know how infilling is completed.
2. Although this work makes a successful use of diffusion methods. A question is: why would one study diffusion model in the first place? The authors mainly compare diffusion models with autoregressive models to illustrate their advantages. But how about non-autoregressive  models such as https://aclanthology.org/D19-1633.pdf ? I feel non-autoregressive models also (1) enjoy continuous latent representations,  (2) generate all tokens in parallel. and (3) induce a coarse-to-fine hierarchy. In fact, the first and last points are rather vague since any multi-layer neural network can be interpreted in such ways. I feel the discussion with iterative non-autoregressive generation is insufficient.
limitations: n/a
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: Recent studies on controllable text generation often fall short of complex controls like syntactic structure. This paper proposes Diffusion-LM that adopts the idea of denoising diffusion probabilistic models, which have been proven effective, in the vision domain, to naturally incorporate the gradient information from an external classifier for the controls of interest. The main challenge of applying diffusion models to text generation is the discrete nature of language data. The authors propose to introduce an embedding step and its reverse rounding step as the solution. Based on such a remedy, the rest of the model is constructed and trained very similarly to the original DDPM model. Plus, the authors also propose an alternative objective to reduce rounding errors by reparameterizing the model to explicitly predict $x_0$ at every timestep. 

Experiments on two datasets and six different control tasks show that the proposed diffusion-LM yields acceptable perplexity and superior performance for controllable generation.
strengths_and_weaknesses: Strengths:

1. Very novel methodology contribution that introduces continuous diffusion probabilistic models into text domain with domain-specific adaption. To the best of my knowledge, this is one of the first (or early) successful attempts to study diffusion models for language generation.
2. Impressive performance on controllable text generation, which may motivate further studies on this research problem.
3. The paper is well-organized and mostly clearly described.

Weaknesses:

1. Yet inefficiency in decoding. 
2. The model runs upon sequences with short and fixed lengths (64 tokens), which may prevent the approach from generalizing to more practical cases. Are there any results about longer or more flexible/dynamic length choices?
questions: - Inaccurate terminology. For controls like syntactic structures, the control signal comes from an external parser. I hence suggest using another term rather than “classifier” for sources of control signals.
limitations: Most limitations of this paper have been discussed in the paper.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper introduces a new non-autoregressive language model based on continuous diffusion models, which have been quite successful in the vision area. Specifically, this paper proposes a novel training objective to train an embedding function that can convert the discrete text tokens into a continuous space. Correspondingly, this paper also introduces methods to improve the rounding from a continuous latent variable to discrete text. The abovementioned techniques facilitate training a Diffusion-LM. Besides, this paper shows that the control over text generation can be done on continuous latent space and the proposed approach yields successful control of Diffusion-LM for six fine-grained control tasks compared to prior work.
strengths_and_weaknesses: **Strengths**
- Very well-written, I enjoyed reading the paper
- There have been few works exploring diffusion models on NLP and this is the first work to explore continuous diffusion models on NLP. It looks definitely interesting and could potentially inspire others to keep working along this line.
- The proposed end-to-end training looks effective in learning embeddings.
- Empirical results demonstrate the success of text control for *fine-grained* control tasks.


**Weakness**
- My primary concern is the scalability of Diffusion-LM. Though under the current experiment setup, i.e., 80M parameters for Diffusion-LM and GPT-2 Small, the proposed approach outperforms existing methods based on autoregressive LM on text control. One of the advantages of existing works is that they can use arbitrary large pre-trained LMs that are available. It is not clear to me how the scaling low works for the current proposed Diffusion-LM and text control approach.
- Though Diffusion-LM achieves better performance on text control, I wonder what could be the most appealing reason to choose a Diffusion-LM over an autoregressive LM generally? As I understand, autoregressive LMs can benefit many downstream NLP tasks in addition to the text control. I am curious about what justifies the application of Diffusion-LM here.



questions: - Line 77-78, I don't think the cited paper has shown that current non-autoregressive approaches fail for language modeling. Would you provide some more details here?
- Line 174, I do not quite understand the following sentence:
" a mixture of Dirac delta distribution will only appear in the terms with $t$ near zero". Could you elaborate more?
- In addition to the fine-grained text control tasks, I wonder how the proposed model performs on coarse-grained tasks, e.g., controlling simple sentence attributes like sentiment?

limitations: Yes
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposed Diffusion-LM, a non-autoregressive language model based on continuous diffusions, for controllable text generation. The non-autoregressive model would iteratively denoise a sequence of Gaussian vectors into word vectors, and the control information is injected by running gradient updates on intermediate latent variables. The main contribution of this paper is that it demonstrated how to combine non-autoregressive model and continuous diffusion model and use it for controllable text generation. The experiment result shows that Diffusion-LM significantly outperforms prior work.
strengths_and_weaknesses: Strengths:
1. The writing is good and the paper is well presented.
2. Detailed derivations are included, and the experiments are adequate and demonstrate good performance.
3. Combining the diffusion model from CV and non-autoregressive model is not a trivial task, this paper explores the changes and improvements needed to apply Diffusion-LM on controllable text generation.

Weakness:
1. The training speed is much slower and the generated text suffers from higher PPL.
2. Some downstream tasks still need three classifiers and the cost for these classifiers is high.
3. Some downstream tasks may lack practical application.
questions: 1. I'm not sure about the training process, is this understanding right: the output vectors of each decoding pass of the non-autoregressive model are called latent variables, and these variables are used to predict the control information. After the diffusion process, the final output  would compute a cross entropy loss with the reference.

2. I don't see the reason why evaluating on syntax-related tasks. Are these tasks useful in real scenario? Or they are just deigned tasks to highlight the strengths of Diffusion-LM?
limitations: The authors had mentioned their limitations in the paper: higher PPL and slower training, and the training process itself might need large GPU memory. The fewer diffusion steps might speed up decoding process but needs further exploration to avoid performance loss.
review_title: null

Meta-review: metareview: This paper introduces a new non-autoregressive language model based on continuous diffusion models, which have been quite successful in the vision area. All reviewers agree that the method is novel, this paper is well-written, and the experiments are convincing. Although there are concerns about the efficiency, this paper is worthy of being accepted by NeuRIPS.
confidence: Certain
Decision: Accept