Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper constructs a conditional (sequence-to-sequence) generative diffusion model for text: DIFFUSEQ. Methodologically, the work is quite similar to Diffusion-LM (Li et al., 2022) with the notable adoption of importance sampling (following Nichol and Dhariwal, 2021) to improve training efficiency. Experimentally, this work evaluates DIFFUSEQ using automatic quantitative metrics against a strong collection of baseline methods in four conditional generation tasks: dialog, question generation, text simplification, and paraphrasing.
strength_and_weaknesses: Methodologically, this work is a minor extension of Diffusion-LM (Li et al., 2022). Nevertheless, this style of text generation model is quite new, so the empirical work evaluating these models for seq2seq modeling tasks is both novel and interesting as it helps to build the community's emerging understanding of these moels. The chosen collection of seq2seq modeling tasks and datasets are interesting, and the choice of baseline methods for comparison are both strong and thorough.

The empirical evaluation compares DIFFUSEQ's performance in each of 4 tasks to (1) encoder-decoder autoregressive models (both GRU and Transformer) (2) fine-tuned LLM's (GPT2 and T5) and (3) the non-autoregressive Levenshtein transformer. DIFFUSEQ's quantitative performance is strong compared to all baselines, and is even competitive with finetuned pre-trained LLM's: this is impressive given that DIFFUSEQ is a smaller model, trained on much smaller datasets.

I also appreciate the architectural ablation of the use of joint versus separate embeddings for sequence pairs (Table 3). One additional ablation that could be interesting to see is a comparison of DIFFUSEQ trained with and without importance sampling. I believe this is the first paper to apply importance sampling to text diffusion models, and so it would be quite interesting to see the effect of this adjustment.
clarity,_quality,_novelty_and_reproducibility: I found the details of this paper difficult to parse. There is a lot of notation, much of which is introduced informally. I had difficulty tracking the meaning and use of various symbols. I think I more or less understand the method that has been implemented in this work, but I gave up on understanding the discussion on Page 4: this discussion is tangential to the methods proposed and evaluated in the rest of the paper, however, I do imagine that this discussion could be interesting and valuable with some improvements to the writing.

Notable examples of notational and writing issues (non-exhaustive):

  - The framing of the first sentence of the introduction is weird. Two issues with GAN training (instability and mode collapse) are attributed collectively to broader generative modelling techniques (GAN + VAE + Flow). And the mention of surrogate objectives here is also a little odd, because diffusion models also rely on surrogates (the variational lower bound). Furthermore, autoregressive models are not mentioned. So overall, this initial framing could really use an overhaul.

  - In Section 2 we are introduced to the latent diffusion states using variable "x". However, in Section 3 "x" is used to denote the conditioning sequence, and "z" is used to denote the latent states. Not a big deal, but redefining notation like this is easily avoidable.

  - Given z_t = x_t + y_t, we are told that "we only impose noising on y_t" and that "In terms of implementation, we execute an anchoring function that replaces the corrupted x_t with the original x_0." This anchoring operation is never defined, and I am pretty confused about what is actually meant. I think this means that we don't noise x, so x_t = x_0 for all values of t: this makes sense to me, but it is weird to describe this as adding noise and then removing it.

  - I do not understand the new subscripts "K" on "w" introduced for Equation 5 (the equality given in the preceding sentence does not constitute a meaningful definition.

  - In Equation 6, shouldn't the sum be an integral? This is marginalization over continuous Gaussian random variables.

  - In Equation 7 and 8, we are introduced to variables "w_t" (with subscripts "t"). I'm not sure what this is supposed to mean, because "w" have been defined as the noiseless, discrete input sequences. And again I think that perhaps these sums ought to be integrals.
summary_of_the_review: This paper is an interesting empirical study of seq2seq text diffusion models, with some (fixable) issues in the writing.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes diffuseq – a diffusion model for sequence-to-sequence text generation tasks. The x and y pairs are concatenated together and sent to the forward diffusion process. Different with diffusion-lm’s classifier-guided diffusion, diffuseq here uses classifier-free diffusion guided by points in space.  Similar with diffusion-lm, an embedding on words is earned jointly with the diffusion/reverse processes.
Experiments on rich text generation tasks show that the proposed diffuseq could achieve better results than strong pretrained language model baselines or vae variant baselines.

strength_and_weaknesses: Strong:

1. Diffuseq framework, with new objective for training and concatenates x (given) and y (target) for diffusion model training and consequent inferencing;

2. Better results compared with a list of strong baselines.

Weak:

1. Different with the paper’s arguments, it is a bit hard to find the significant difference in terms of technical novelty with existing diffusion-LM – concatenating x and y actually almost doubles the diffusion space and possibly makes the training/inferencing speed to be much slower than diffusion-lm – a detailed experimental comparison with diffusion-lm is required. 

2. Some essential parts need to be enriched – such as equation 2. And most equations in page 4 are less important to be included – give the spaces to equation 2 – which is your novel objective function. (appendix A’s extension is far from enough, such as how EMB’s item square was removed).


Detailed questions and comments

1. Figure 1, the t=T/2 figures are real examples? Frequently in my experiments, the figures are clear only at the last several steps and did not achieve the clear results at t=T/2 steps.
2. Also in figure 1, can you make it clearer the difference of “guidance” in diffuseq vs. “condition” in diffusion-LM – align these differences in figures with the specific equations’ differences.
3. Equation 2, more details are preferred and even appendix A is not clear enough of from the two equations: such as how the EMB included item changed from square to without square. I could not find the details in appendix A.
4. Page 4’s equations are less important (most are not novel points of this paper) and I think they can be reduced or part of them can be moved to appendix – leaving more spaces for explaining equation 2 – which is your own objectives.
5. The “1” involved in equation 2 is a bit ambiguous. Basing on current description, I think it is t=1, not a simple 1 and prefer to see how exactly you implemented this in your code – did you use t=[0, 1] or t=[0, T]?
6. Table 1, can you also give the results of diffusion-LM? More than half baselines selected here are less comparable than existing diffusion-based generation models.
7. In “inference speed”, “1000 diffusion steps” a similar inference speed. I am not able to understand this part. Any detailed reasons on this? How about the speed comparison on other tasks besides “question generation”? does the same speech comparison results hold?
8. Table 3, right-bottom item, “0.9376” is larger than “0.8103” and why this happen? Any detailed error analysis on this? If you say diffusion-seq is strong at diversity, then this result actually did not align with that.




clarity,_quality,_novelty_and_reproducibility: Most parts of the paper are clear and the novelty is good. No code provided and the reproducing ability is average.
summary_of_the_review: The concatenate of x and y together for diffusion is the major novel part with a well-designed objective function. Most parts of the paper are well written. The score will for sure be improved if equation 2 can be enriched and more comparison with strong baselines such as diffusion-lm can be given here.

This paper is above the average level and higher scores are deserved with the updating of the current unclear parts.

-- 

after reading the authors' feedbacks, I decide to rank higher this paper. Basically, the general idea of seq2seq with diffusion is an interesting direction and many related seq2seq tasks should benefit from it. It is good to see that the authors enriched their core ideas and replace some less important parts of page 4 to the appendix, hopefully, this makes this paper easier to be followed.

[still, a bit doubt about that diffusion-LM can not deal with seq2seq problems since in Diffusion-LM's paper, a number of controllable text-to-text generation tasks were reported... anyway, this only influence the comparison part of this paper, not the central idea of this seq2seq-diffusion models.]
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper introduces conditional diffusion models in the NLP setting, demonstrates its empirical advantages, and draws theoretical connections with autoregressive and iterative-nonautoregressive models.
strength_and_weaknesses: The paper leaves good overall impression due to high-quality writing, propose positioning against state of the art, and good empirical study. Story-telling could be improved with proper background/notation introduction, clearing any abuse of notation, offloading any derivations to supplementary, and focusing on key findings and theoretical conclusions being drawn. Particularly, "connections" paragraph seems to be overly detailed, yet Eq. 1 and 2 are neither too detailed nor distilled to the barebones. I appreciate the author's usage of equation annotations, yet I found them hard to follow anyway.
clarity,_quality,_novelty_and_reproducibility: The paper has an overall clear structure. The problem solved by the papers is important; The method is sufficiently novel and the results can be clearly reproduced. It would be better to state computational requirements in the main text.
Writing:
- "Partially noising" -> "partial noising"
- Notation at the beginning of Sec.3 is not self-contained and needs refinement. Particularly, z was never introduced. The admitted abuse of notation in the absence of properly introduced notation does not look good.
- Fig. 2 was never referred to for method illustration purposes (the first sentence of Sec. 3 doesn't count as it doesn't list any details)
- "cope" misused around the same paragraph
- "standard variation" sounds like an exotic term, I'm not sure if it is any better than what already exists, standard deviation, variance. For the case of multivariate Gaussian distribution, diagonal covariance would also be an appropriate term.

summary_of_the_review: Overall a good paper with few loose ends, but writing, clarity, and story-telling could be improved.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper studies the application of denoising diffusion probabilistic models to seq2seq text generation. Authors propose DiffuSeq - a diffusion-based model for seq2seq task using a technique they call "partially noising". Authors also show the theoretical connection between DiffuSeq and standard AR / NAR  language models. They evaluate DiffuSeq on a number of seq2seq tasks such as dialogue generation and paraphrasing, against several autoregressive and non-autoregressive baselines, achieving comparable generation quality. On top of that, authors note that DiffuSeq generates more diverse outputs. Authors analyze the behavior of the proposed model, report inference throughput and ablate several design decisions.
strength_and_weaknesses: 
### Strengths

- the proposed method is evaluated in multiple seq2seq tasks, supporting the claim for general seq2seq modeling
- authors attempt to show the connection between DiffuSeq and existing seq2seq strategies
- the paper is reasonably well-written and easy to follow

### Weaknesses

I have a single main concern, justifying my current (negative) recommendation -- the theoretical need for a new type of denoising diffusion specific to Seq2Seq problems. The other weaknesses i found are fairly minor and are listed at the bottom.

As authors note in the paper, prior works have already adapted denoising diffusion to unconditional language modeling. To extend their results to seq2seq problems, one must find a way to incorporate additional inputs to the diffusion model. In continuous "text-to-image" diffusion, this can be done by simply feeding the extra inputs to the model.
Modern diffusion models for conditional image generation, such as Imagen[1] or DALLE-2 [2] simply attend to the input text inside the diffusion model __without any changes to the underlying diffusion or noise distribution__.

Hence, my main question: __does DiffuSeq really need the extra complexity of "partially noising", instead of just treating x as inputs?__ To the best of my understanding, the x part is not generated by the model, but frozen throughout the diffusion process. As a result, the model does not learn to denoise it, and can effectively treat it as an extra input to the existing (unconditional) text diffusion methods. Please correct me if my understanding is flawed in any way.

If DiffuSeq does *not* need the extra math, then this paper effectively becomes an empirical study of existing diffusion models for seq2seq applications, which leads to my __second concern: non-standard evaluation tasks__. As authors state in the paper, seq2seq is a very popular and competitive area of research with many popular datasets. However, the datasets/tasks chosen in the paper are not among the popular ones (see [3]), which makes it difficult to assess whether the baseline models are state-of-the-art for those tasks.



[1] https://arxiv.org/abs/2205.11487

[2] https://cdn.openai.com/papers/dall-e-2.pdf

[3] https://paperswithcode.com/area/natural-language-processing
clarity,_quality,_novelty_and_reproducibility: The paper is generally clear and easy to follow, except for a few unconventional word choices (see example below).
As for the reproducibility, it is difficult to assess without the supplementary code; assuming it contains the proper instructions and dependencies, it should be fairly easy to make all experiments reproducible. The two exceptions are (1) missing hardware specifications for inference speed and (2) the type of BLEU used for evaluation (see below).

> This modification (termed partially noising) allows us to cope diffusion models with conditional language modeling.

To the best of my understanding, this is an unconventional use of "cope". One alternative would be consider "adapt __ for __". This is a minor concern that does not affect my score.


> we use the standard metric BLEU (Papineni et al., 2002)

For the sake of reproducibility, please specify the exact version of BLEU you used. For instance, SacreBLEU [4] can be different from NLTK's BLEU [5], which is in turn different from the implementation in MOSES[6]. This is a minor concern that does not affect my overall recommendation.


### Questions


> Diversity Source : Temperature (Table 4)

- did you use sampling with temperature when reporting the quality of autoregressive models? If so, how does it change when using beam search? (if needed, see FairSeq or Hugging Face Transformers for fairly standard beam search implementations).
- how does DiffuseSeq compare against diversity methods based on diverse beam search[9]?


> We first generate a set of candidate samples S from different random seeds of DIFFUSEQ and select the best output sequence that achieves the minimum expected risk under a meaningful loss function (e.g. BLEU or other cheaper metrics like precision). In practice, we use the negative BLEU score in our implementation

Please clarify: does this reranking strategy require knowing target sequence? If not, please explain how exactly is BLEU computed (assuming this is a sentence-level BLEU, what is the target?). Do you apply this strategy for AR / NAR baselines as well - and if so, how do you do that?



[4] https://github.com/mjpost/sacrebleu

[5] https://www.nltk.org/

[6] http://www2.statmt.org/moses/

[7] https://github.com/facebookresearch/fairseq

[8] https://github.com/huggingface/transformers

[9] https://arxiv.org/abs/1610.02424
summary_of_the_review: I left a generally negative review, mainly because I do not see the need for the extra complexity of DiffuSeq, compared to simply treating the source text x as an extra input. My concern is based on how input text is treated in existing diffusion-based text-to-image models[1,2]. 
Outside of that, the paper appears to be generally decent, with both strong and weak points as described above. I am open to discussion and, _should authors prove that my main concern is wrong_, increasing my score.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper constructs a conditional (sequence-to-sequence) generative diffusion model for text: DIFFUSEQ. The paper evaluates the method on four tasks in comparison with six baselines, demonstrating the empirical improvement of Diff-Seq2seq. They also draw theoretical connections with autoregressive and iterative-nonautoregressive models.

**Strengths:**

The reviewers generally had positive comments about the paper, highlighting the quality of the writing, the rigor of the empirical study, and the impressive results.

**Weaknesses:**
Most reviewers felt that the technological novelty w.r.t. Diffusion-LM was minimal, though they acknowledged the differences. One reviewer had specific concerns about the justification for the need for a new type of diffusion specific to Seq2Seq problems. Another concern was that the authors had evaluated using non-standard tasks.

I’m inclined to accept the paper. I think the tasks in the empirical study are fine and may even provide promising indicators for applicability of the method beyond MT and summarization. With regards to whether the seq2seq-specific diffusion is necessary, the point was not deemed crucial by the reviewer who participated in the discussion, who felt the remaining contributions of the work made it suitable for acceptance.

justification_for_why_not_higher_score: Lingering issues surrounding whether the methodological contribution is as strong as presented.
justification_for_why_not_lower_score: The empirical study demonstrates clear improvement over strong baselines.
Decision: Accept: poster