DiffuSeq: Sequence to Sequence Text Generation with Diffusion Models
Authors: Shansan Gong
Mukai Li
Jiangtao Feng
Zhiyong Wu
Lingpeng Kong
Venue: ICLR 2023 poster (2023)
Keywords: diffusion model
sequence to sequence
text generation
diveristy

Abstract: Recently, diffusion models have emerged as a new paradigm for generative models. Despite the success in domains using continuous signals such as vision and audio, adapting diffusion models to natural language is under-explored due to the discrete nature of texts, especially for conditional generation. We tackle this challenge by proposing DiffuSeq: a diffusion model designed for sequence-to-sequence (Seq2Seq) text generation tasks. Upon extensive evaluation over a wide range of Seq2Seq tasks, we find DiffuSeq achieving comparable or even better performance than six established baselines, including a state-of-the-art model that is based on pre-trained language models. Apart from quality, an intriguing property of DiffuSeq is its high diversity during generation, which is desired in many Seq2Seq tasks. We further include a theoretical analysis revealing the connection between DiffuSeq and autoregressive/non-autoregressive models. Bringing together theoretical analysis and empirical evidence, we demonstrate the great potential of diffusion models in complex conditional language generation tasks. Code is available at https://github.com/Shark-NLP/DiffuSeq

Published as a conference paper at ICLR 2023
DIFFUSEQ: SEQUENCE TO SEQUENCE TEXT GENERATION WITH DIFFUSION MODELS
Shansan Gong', Mukai Li', Jiangtao Feng's Zhiyong Wu', Lingpeng Kong²
Sharrk-LP, Shanghai AI Laboratory 'The University of Hong Kong (gongshansan, limukai, fengj inngtao, wuzhiyong] org. on 1pk0cs. mexecs.hku... hku.hk.
ABSTRACT
Recently, diffusion models have emerged new paradigm for generative mod- els. Despite the success in domains using continuous signals such as vision and audio, adapting diffusion models natural language is under-explored due the discrete nature of texts, especially for conditional generation. We tackle this challenge by proposing DIFFUSEQ: a diffusion model designed for sequence- to-sequence (SEQ2SEe) text generation tasks. Upon extensive evaluation over a wide range of SEQ2SEQ tasks, we find DIFFUSEQ achieving comparable or even better performance than six established baselines, including of-th- art model that is based on pre-trained language models. Apart from quality, an intriguing property of DIFFUSEQ is its high diversity during generation, which is desired in many SEQ2SEQ tasks. We further include a theoretical analysis re- vealing the connection between DIFFUSEQ and autorggressive/onn-uttrrggg models. Bringing together theoretical analysis and empirical evidence, we demon- strate the great potential of diffusion models in complex conditional language gen- eration tasks.
INTRODUCTION
Among existing generative models, GAN (Goodfellow et al., 2014) suffers from the instability issue (Salimans et al., 2016), subjecting to mode collapse (Metz et al., 2017); VAE (Kingma & Welling, 2014) has to rely on surrogate objectives to approximate maximum likelihood training and Flow-based models (Dinh et al., 2017) has to use specialized architectures to construct reversible transform. Diffusion models (Ho et al., 2020; Nichol & Dhariwal, 2021) have circumvented sev- eral of these limitations and emerged as derpinned by non-equilibrium thermodynamics network (Song & Ermon, 2019). To date, the major breakthroughs are in domains using continuous signals, such as vision (Saharia et al., 2022a;b; Ramesh et al., 2022) and audio (Kong et al., 2020). However, extending continuous diffusion models to natural language remains open challenge due to the inherently discrete nature of texts.
paradigm for generative models, theoretically un- et al., 2015) and score-matching
On the basis of unconditional generation in continuous space which is illustrated in Figure (a), isting efforts (Hoogeboom et al., 2021; Austin et al., 2021) start customizing diffusion models to text in discrete space on unconditional language modeling (i.e., free text generation). Diffusion-LM (Li et al., 2022), as in Figure ((b), models texts in continuous space and proposes to use an extra-traines classifier as guidance (i.e., the condition signal x) impose subtle changes (usually complex, fine- grained constraints) on generated sentences. Nonetheless, these models do not naturally generalize to conditional language modeling (i.e., the model assigns probabilities p(w x) to sequences of words (SEQ2SEQ) setting where the condition is
w given x). In the more general requence-to-sequeee also a sequence of words, applying Diffusion-L.M can be difficult. The reason is that classifiers are attributes-oriente
and we can not train hundreds-of-then thooussnnd classifiers to model the semantic meaning between conditions and generated sentences.
SBQ2SEQ is an essential setting in NLP that covers wide range of important tasks such as open- ended sentence generation, dialogue, paraphrasing, and text style transfer. In this paper, we propose
Code available https://g/t http://ggggtuutteee www.c/ow///.aacceeeeee Shsssshaahia NLP /Di ffuseq

---

Published as a conference paper at ICLR 2023
t-
t=T/2
TelT
T/ t=0
(e)
(b)
(c)
forward process ((-)
reverse Pe ()
conditional signals
hidden space of texts guidance
(8) Unconditional Gaussian diffusion process in continuous space
(b) Classifier gluddeeiegguided diffusion process (Diffusion-LM)
(c) Classifier-free diffusion guided polinss space (DiffuSeq)
L=T/2
Cm0
Figure 1: The demonstration of unconditional, classifier-guided and classifier- free diffusion models.
DIFFUSEQ, depicted in Figure I(c), classifier-fre free diffusion model that supports SEQ2SEQ text generation tasks. By modeling the conditional probability of the target sentence given context using one single model, one advantage of DIFFUSEQ is that this paradigm allows complete model to fit data distribution and utilize conditional guidance, rather than depending on a separate classifier.
Different from canonical generation approaches in autoregressive (AR) left-to-right manner (Rad- ford et al., 2019), DIFFUSEQ generates text tokens parallelly in the eoo-autoreeressvve (NAR) way. To corroborate the effectiveness of our DIFFUSEQ, conduct experiments on four SEQ2SEQ tasks. Compared to AR and NAR models, which suffer from the "degeneration" problem (Holtzman et al., 2019) and rely decoding strategies, DIFFUSEQ achieve considerable sentence-level diversity without sacrificing the quality (see $ 4.2).
To sum up, we make series of technical and conceptual contributions: (a) we are the first deploy the diffusion model on SEQ2SEQ text generation, and our proposed DIFFUSEQ as a condi- tional language model is trained end-to-end in classifier-free manner, (b) establish theoretical connection among AR, NAR and DIFFUSEQ models, and justify DIFFUSEQ as an extension of iteR models; (c) with strong empirical evidence, we demonstrate the great potential of diffusion models in complex conditional language generation tasks.
2
PRELIMINARY AND PROBLEM STATEMENT
Preliminary. A diffusion model typically contains forward and reverse processes. Given a data point sampled from a real-world data distribution Zn q(z). the forward process gradually cor- rupts Zg into standard Gaussian noise Zr ~ No0, D). For each forward step E [1, ...,, T", the with 3, E (0,1) as differ-
perturbation is controlled by q(z)/z(-1) ent variance scales. Once the forward process is completed, the reverse denoising process tries to gradually reconstruct the original data Zg via sampling from 27 by learning a diffusion model fo-
N(zz: vI ,,
Problem Statement. Many recent efforts have been devoted to adapting diffusion models dis- crete texts (See & 5). However, they all focus on unconditional sequence modeling. this paper, target the
text generation tasks. In particular, given m-length source sequence w [wT. ..., w }. we aim to learn a diffusion model that can produce a n-length target sequence w {w - wx) conditioning the source sequence.
DIFFUSEQ
We propose DIFFUSEQ to extend vanilla diffusion models to learn conditional text generation (as shown in Figure 2), concerning the model architecture and the training objective.
Forward Process with Partial Noising, In the beginning of forward process, follow Diffusion- LM (Li et al., 2022) to design an embedding function EMB(w) to map the discrete text w into a continuous space. In particular, given pair of sequence w and w DIFFUSEQ learns unified
2

---

Published as a conference paper at ICLR 2023
Reverse process
Forward process
Caussian Noise
Roundling
How long was the trip?
Po Pe
Pel (w((z))
W
w
It was year.
9 (z [w)
7/T
7---1
7.
Egg Open-Domail Dialogue
Embedding map
Partial Gaussian Noise
Word Embeddings
Text
Sequence to Sequence
Figure 2: The diffusion process of our conditional diffusion language model DIFFUSEQ. Given the source w and the target w we pair-wisely transform them into continuous space Zo. The partial Gaussian noise is iteratively added the target space of ze.
feature space of w and w by embedding transformation and concatenation as EMB( (MBB) (EMB(WF),... EMB ( ( ), EMB(W)), [MB(ww)) (m++x)xdd The transformation allows us to adapt discrete textual input into the standard forward process, by extending the original forward chain a new Markov transition {o(2o]w=O0) = -N(EMB(w
We denote z, x, + y, to simplify the wordings, where x, and y, represent parts of 2, that belong to w and w respectively. For each forward step ((zg|z(-1), we gradually inject noise into last step's hidden state z obtain z,, Unlike conventional diffusion models that corrupt the whole 24 (both x, and y+) without distinction, we only impose noising on Yt- This modification (termed partial noising) allows adapt diffusion models for conditional language modeling.
Reverse Process with Conditional Denoising. The ultimate goal of the reverse process is to recover the original 2o by denoising Pe (Zo:T) p(zT) II po(Z:-1/ze). We model the learning process po pg(z=i//z)) = (Z jel (Z, t), (2,,t) using the proposed diffusion model DIFFUSEQ: (z, t), where the (a(+)) and (-(-) is the parameterization of the predicted mean and standard deviation of (Z(--1/2 (2)) in forward process, derived using Bayes' rule. The detailed deriva- tions are in Appendix A. With the partial nosing strategy adopted in the forward process, impose the input as the condition when denoising as shown in Figure 1. The proposed conditional denoising is classifier-free by nature: we do not require extra-trained classifiers to control the de- noising process.
transformer architecture to model fe. which spontaneously models the se- mantic relation between X2 and ye. We compute the variational lower bound (CvLB) following the original diffusion process. Fround corresponds to rounding operation Figure
Specifically,
po (2T) log g(Zr (Zo)
q( g(zz-
Zo, Zz)
LVL. Egizr (20)
+
log
Po
(2:)
fx
C.
(I)
log 2o go(20) (2o) w po (2 22))
log po (www= myy (20)
Co We further simplify the training objective as follows (details in Appendix A):
min LVLB
min
Σ I|zo
fo(ze, 1)1
|EMB(w fo(z1, 1)11 log pe ( e 2o)
min Σ Ilyo [[ED Fo ()1,
1)1 [|EMB(W"))
here use fo t) to denote the fractions of recovered zo corresponding to yo- Note that al- though in the first term, we only compute the loss w.r. Yn- due to the attention mechanism in the transformer, the reconstruction of yo also takes Xy into account, thus the gradients from the first term
3

---

Published as a conference paper at ICLR 2023
will also affect the learning of Xu- The mathematically equivalent regularization term R(1 Zu kull3)) regularize the embedding learning. We further share the embedding function between source and target sequences, enabling the training of two different feature spaces jointly. This sets DIFFUSEQ away from existing solutions in vision such as GLIDE (Nichol et al., 2022).
Training and Inference Methods. In our preliminary experiments, we find that the high diversity in NLP datasets and long diffusion steps often result in insufficient training. We hypothesize the reason is that sampling step uniformly causes unnecessary noise in the LVLB objective. We hence employ importance sampling (Nichol & Dhariwal, 2021) to address this problem.
Li
LVLB = Etrpp
Pr x V EICA), Σ =
(3)
Pr
Intuitively, the importance-weightee sampling algorithm will spend more steps on diffusion steps with larger Li, and vice versa.
To conduct SEQ2SEQ generation given the condition EMB(w"), ) we randomly sample Yr (00, I) and concatenate with EMB(w") to obtain Zr. We can now repeat the reverse process until we arrive Zg. each sampling step, an anchoring function is executed towards reparame- terized 2, Specifically, the anchoring function: (a) operates rounding on to map it back to word embedding space following et (2022); (b) replaces the part of recovered 2,-1 that belongs to w with the original Xo, considering that this part is recovered from corrupted z, via fo and not strictly equals to Xo. Note that (b) is designed for DIFFUSEQ.
To improve the quality of generation, we apply the widely used Minimum Bayes Risk (MBR) de- coding strategy (Koehn, 2004). We first generate a set of candidate samples S from different random seeds of DIFFUSEQ and select the best output sequence that achieves the minimum expected risk under meaningful loss function (e.g. BLEU or other cheaper metrics like precision). In practice, use the negative BLEU score in implementation.
Connections to AR, Iter-NAR, and Fully-NAR Models. To better understand the behavior of DIFFUSEQ, we give the theoretical connection to autoregressive (AR), iterative non-autoregressive (iter-NAR), and fully non- von-autoregressive (fully-NAR) models. We argue that DIFFUSEQ can be seen as an extension of iter-NAR model. Detailed graphical learning discrepancies of these four cases are discussed in Appendix B for reference.
AR models learn p( w. (w") by autoregressive decomposition based on left-context:
PAR(WW (w") (w")
II p( /w w")
(4)
initial prediction
progressive left-context prediction
while fully-NAR models (Gu al., 2018; Qian et al., 2021) learn the conditional probability given independent assumption for fast inference:
Piully-ARR (w /w") /
II l(w"/w"". (=I....m
(5)
To make a better analogy to AR and NAR models, we use a lossless way to formulate iterative NAR models (Gu al., 2019; Ghazvininejad et al., 2019) by introducing series of intermediate sequences Wi:k-1- w w with K editable iterations:
NAR w 1:n (w))
Σ
II E....
p(w
II
w p( 144, Relmm w
T.K. 1=1...
(6)
initial prodiction
progressive full-context prodiction
Previous study (Huang et al., 2022) shows that there is a gap called conditional total correlation between AR Eq. (4) and fulRy-NAR Eq. (5) learning paradigms, because of lossy decomposition of NAR models. However, when comparing iter-NAR Eq. (6) with AR Eq. (4) models, they both can be factorized into an initial prediction term and a progressive prediction process based on different context (i.e. left-context in AR and full-context in iter-NAR), and the diserepancy pointed out by
4

---

Published as a conference paper at ICLR 2023
Huang al. (2022) is therefore closed in iter-NAR assuming sufficient steps. By showing DIF- FUSEQ an extension of the iter-NAR model, we offer a justification that it will not suffer from the conditional total correlation for the same reason.
A straight-forwat way to formulate pure continuous diffusion models is to introduce a series of Gaussian noise-corrupted features along with diffusion steps: y1:7-1, Yo y.yr (O,
Paiffusion (w" wn)
O(w"lyo w")
II yly w"),
yr "y...
tinal prediction
progressive full-context difflssion
where p(y:-1ly: w*) describes the diffusion step on continuous representations y. The rounding operation in DIFFUSEQ maps the continuous vectors to discrete w for each time step t, we in addition introduce this into Eq. (7):
PDIFUSSO (w.lww")
Σ
Dyw"lyo, w")
II p(wlly,, w (l()
(8)
wi Yo
plwilyr, w
plyel Dpwwwww w").
(9)
By rearranging Eq. (8) into Eq. (9), we can see DIFFUSEQ can be seen as more generalized form of iter-NAR Eq. (6) before marginalizing out (yr,..., despite the different initialization of yr A more detailed derivation is shown in Appendix C.
4 EXPERIMENTS
We conduct experiments to validate the effectiveness DIFFUSEQ on four different tasks, against six strong AR/NAR baselines.
4.1 EXPERIMENTAL SETUP
Tasks and Datasets. SEQ2SEQ generation covers a wide range of tasks, among which we choose four typical and popular tasks. Open domain dialogue requires models to generate informative responses given dialogue context. We use Commonsense Conversation Dataset (Zhou et al., 2018), which extracted from Reddit single-round dialogs, with over 3 million conversational pairs. Question generation((G) aims to generate questions given context as input. To obtain sufficient training samples, we use the dataset Quasar-T (Dhingra et al., 2017) preprocessed by Lin et al. (2018), and then generate document-question pairs to obtain 119K training samples (details in Appendix D.I). Text simplification aims to revise the complex text into sequences with simplified grammar and word choice. Jiang et al. (2020) constructs corpus consisting of 677K complex- simple sentences with revision alignment. Paraphrase task generates an alternative surface form in the same language expressing the same semantic content. We adopt widely used QQP sourced from the community question answering forum Quora, with 147K positive pairs.
Baselines. We consider three groups of models as baselines, covering both AR and NAR architec- tures. The first group of methods adopts encoder-decoder architecture (Cho et al., 2014) which is well-studied for SEq2SEQ tasks, and we conduct experiments on two popular models: GRU with attention and Transformer ( (Vaswani et al., 2017). The second group is the finetuned large pre-trained language model (PLM), among which GPT2 (Radford et al., 2019) has demonstrated great success in almost all SEO2SEQ tasks. We further compare to GPVAE (Du et al., 2022), which augments pre-trained T5 (Raffel et al., 2020) with VAE improve the generation diversity. For the last group of baselines, consider LevT (Gu al., 2019), widely used, strong iterative NAR model. All baselines are trained following instructions in their papers, and details can be found in Appendix D.2.
For NAR models, yr uniform copied from source sentence or unk's token embedding (Gu et al., 2018); for diffusion models, yr is sampled from normal distribution (0, I). Phttps: /www. kaggl com/ hamm//qqoree quest on-pairs.
5

---

Published as a conference paper at ICLR 2023
Table 1: The overall results of different methods on different SEQ2SEQ tasks. The first group o of methods adopt autoregressive encoder-decoder architecture and the second group is the fine- tuned large pre-trained language model (also autoregressive manner) while the last group & is non-autoorereessive The best results are bold, and the best results without PLMs are underlined.
Tasks
Methods
BLEUT
R-Lt
Scoret
dist-11
selfB. / div-4f
Len
GRU-attention
0.0068
0.1054
0.4128
0.8998
0.8008/0.1
4.46
Transformer-base
0.0189
0.1039
0.4781
0.7493
0,3698/0.6477
19.5
Open
GPT2-base base FT
0.0108
0.1508
0.5279
0,9194
0.0182/0.999
16.8
Domain
GPT2-large FT
0.0125
0.1002
0.5293
0.9244
0.0213/0.9933 9938
Dialogue
GPVAE-T5*
0.0110
0,1009
0.4317
0.5625
0.356000.5551
20.1
NAR-LevT
0,0158
0.0550
0.4760
0.9726
0.7103/0. 1416
4.11
DIFFUSEQ (Ours)
0.0139
0.1056
0,5131
0.9467
0.0144/0.9971
13.6
GRU-attention
0.0651
0.2617
0.5222
0.7930
0.9999/0.3178
10.1
Transformer-base
0.1663
0.3441
0.6307
0.9309
0.3265/0.7720
10.3
Question Generation
GPT2-base FT
0.0741
0.2714
0.6052
0,9602
0.1403/0.9216
10.0
GPT2-large FT
0.1110
0.3215
0.6346
0.9670
0,2910/0.800 8062
9.96
GPVAE-T5*
0.1251
0.3390
0.6308
0,9381
0.3567/0.7255
11.4
NAR-LevT DIFFUSEQ (Ours)?
0.0930
0.2893
0.5491
0.8914
0.9830/0.4776
6.93
0.1731
0.3665
0.6123
0.9056
0.2789/0.8100
11.5
GRU-attention o
0.3256
0,5602
0.7871
0.8883
0.9998/0.38 3311
18.9
Transformer-
0.2693
0.4907
0.7381
0.8886
0.6924/0.5099
18.5
Text Simpli-
GPT2-base FT
0.3083
0.5461
0.8021
0.9439
0.5444/0.660 6047
16.1
GPr2-large FT* GPVAE-T5
0.2693
0,5111
0.7882
0,9464
0.6042/0.5876
15.4
fication
0.3392
0.5828
0.8166
0.9308
0.8147/0.4355
18.5
NAR-LevT
0.2052
0.4402
0.7254
0.9715
0.9907/0.3271
8.31
DIFFUSEQ (Ours) :
0.3622
0.5849
0.8126
0.9264
0.4642/0.6604
17.7
GRU-attention 9
0.1894
0.5129
0.7763
0.9423
0.9958/0.322
8.30
Transformer base
0.2722
0.5748
0.8381
0.9748
0.4483/0.7344
7345
11.2
GPT2-base FT
0.1980
0.5212
0.8246
0.9798
0.5480/0.6245
9.67
Paraphrase
GPT2-large FT GPVAE-T5
0.2059
0.5415
0.8363
0.9819
0.7325/0.5022 5500
9.53
0.2409
0.5886
0.8466
0.9688
0.5604/0.61
9.60
NAR-LevT
0.2268
0,5795
0.8344
0,9790
O. 0,9995/0.33 3339
8.85
DIFFUSEQ (Ours) F
0.2413
0.5880
0.8365
0.9807
0.2732/0.8641
11.2
Evaluation. We evaluate the generated sequences from two aspects: quality and diversity. To evaluate the quality, we use the standard metric BLEU (Papineni et al., 2002) and ROUGE (Lin, 2004) score. Since string- siiggssmmillaittbbaiee based metrics can be unsatisfactory for open-ended generation, we also report BERTScore (Zhang et al., 2019) that assesses the semantic similarity between gen- erated sentences and references. Details are Appendix D.4. Higher scores of BLEU, ROUGE and BERTScore reflect better performance. As for diversity, we use distinct unigram (dist-1) to measure intra-diversity within each generated sentence, where the lower dist-1 indicates that the generated sentence contains more repeated words. For sentence-level diversity evaluation, we con- sider sentence-level self-BLEU (Zhu et al., 2018) to measure the n-gram overlap between the set of outputs w.r.t source sentence, and additionally use diverse 4-gram (div-4) (Deshpande et al., 2019) to measure the ratio of distinct 4-grams in the set of outputs per source sentence. The lower SeUf-BLEU and higher div-4 suggest higher diversity of generation. For each method including DIFFUSEQ, we generate 3 samples for each source sentence to compute the diversity metrics.
Implementation Details. Our DIFFUSEQ is based on the 12 layers of Transformer with 12 at- tention heads, where the time step embedding is plugged akin to the position embedding. The maximum sequence length is 128, with embedding dimension d 128, diffusion steps T = 2, 000 and a square-root noise schedule. To reduce the out-of- vocabulary generation, we apply Byte Pair Encoding (Sennrich al., 2016) to construct the vocabulary. After conducting the diversity beam

---

Published as a conference paper at ICLR 2023
Table Sample outputs in QQP test set, conditioned on the same X.
Original sentence: How do make friends.
Paraphrase reference: How to make friends ?
GPT2-large finetune
GPVAE-TS
DIFFUSEQ
How can make friends?
How can I make friends?
How can I make friends better?
How can make friends?
How do I make friends?
How can I make friends?
How can make friends?
How can make friends? How can make friends? What's the best way to make
How you make friends?
How can make friends?
What the hest way make friends?
How do make friends and
How can I make friends and more something?
keep them?
friends and make make friends?
search (DBS) (Vijayakumar et al., 2016) for the Transformer- base model and GPT model, we find that DBS does not always promote diversity over temperature sampling and therefore list the best diversity results. We compute the accuracy metrics of DIFFUSEQ using MBR with the size of candidate samples [8] 10. The experiment is deployed on NVIDIA A100 Tensor Core GPUs, and we use 4 GPUs on training and single GPU on sampling.
4.2
MAIN RESULTS
As shown in Table 1, we conclude that DIFFUSEQ achieves comparable or even higher generation quality compared with strong baselines. At the same time, DIFFUSEQ consistently demonstrates its superiority in generating diverse outputs given the same input sequence.
from Table 1, DIFFUSEQ wins competitions over least quality metric against 6 baselines x 4 tasks. Although NAR models such as LevT can also outperform AR baselines some- times, they still lag well behind DIFFUSEQ by large margins (i.e., relative improvements over 50% for BLEU in QG task and R-L in Dialogue task). Even compared with pre-trained then finetuned GPT2 models, DIFFUSEQ still delivers superior performance than the base variant, and is compara- ble with the large variant, which has 8.2 times more parameters than DIFFUSEQ. These empirical results amply support our findings in $ 3, where we theoretically analyze the potential of diffusion models in modeling text sequences compared with AR models given sufficient diffusion steps.
As
DIFFUSEQ, as a member of the deep generative model family, also exhibit the capacity to generate highly diverse sequences. As suggested by self-BLEU (lower is better) and div-4 (higher is better), in almost all cases, DIFFUSEQ significantly outperforms AR baselines in terms of sentence-level diversity (i.e., producing diverse outputs given the same input). For diversity in word choice within one sentence, we consider dist-1: a higher dist-1 indicates less repetition within a sentence. As see from Table 1, DIFFUSEQ has less repetition compared with encoder-decoder methods, but still fall behind the pre-trained GPT2 models (the same situation with BERTScore). These results suggest there is still room for improvement (e.g., use pre-training techniques) in diffusion models' token-level choice. Different from NAR-LevT, DIFFUSEQ does not rely on an extra length prediction module but automatically decides by the padding token instead and is able to generate longer output sentences, indicated by the last column for average generation length.
In Table 2, we provide examples to showcase DIFFUSEQ's ability to generate diverse samples. More examples can be found in Appendix D.5.
Text Simplification
Paraphrase
Question Generation
0.16
0.36
0.25
0.74
0.34
0.32
0.22
0.31
100
0.31
Diflusey
0.20
6PT)
OPWEETS
GPT2-base
GPT2-targe
697)
20
10
a
candidate sizes [S]
candidate sizes [S]
diy-4
The increase of BLEU score with different candidate sizes (S).
Figure 3:
Figure 4: Trade-off between quality and di- versity (details Appendix D.3).
7

---

Published as a conference paper at ICLR 2023
Text Simplification
Question Generation
030
08
RU
sumpa
06
0,900
636
02
0.25
shged
000
Generation process
10
1/c
00
60000
Sampling Steps
Sen
2:00
Figure 5: The curve of BLEU/div-4 score along with generation process (percentage of steps).
Figure 6: The BLEU and inference speed of DIFFUSEQ and GPT2-large.
4.3 ANALYSIS
We conduct a series of analysis to investigate the effectiveness of different aspects in DIFFUSEQ.
Diversity Ensures Quality. Generating high-quality texts with high diversity is an important re- quirement for many text generation applications and the trade-off between quality and diversity is always critical concern in open-ended NLG tasks (Zhang et al., 2021). Different from AR models relying on the decoding strategy like temperature and nucleus sampling (Holtzman et al., 2019) and VAE models sampling latent variable from Gaussian Prior, the natural advantage of DIFFUSEQ is to generate different sentences along with a series of random Gaussian noise. In Figure 4, we elu- cidate that DIFFUSEQ have better trade-off between generation quality (BLEU) and sentence-level diversity (div-4). Here we further demonstrate that the high diversity provided by DIFFU SEQ can be turned into better quality.
MBR is a common strategy to improve generation quality by aggregating and ranking candidate sequences, and we find that the upper bound of MBR is decided by a diversified candidate To valid this, we simultaneously apply MBR on both DIFFUSEQ and GPT2 with various candidate sizes [S). The results are shown Figure 3. As we can see, DIFFUSEQ lags behind GPT2 without using MBR ([S) 1) with small candidate set (|S) 3). However, as [S] increases, DIFFUSEQ starts to outperform GPT2 by an increasing margin. The reason that autoregressive models like GPT2 tend to generate highly similar candidates (as discussed in $ 4.2), which impedes the effectiveness of MBR. As |S] increases to 20, DIFFUSEQ still shows better rising trends than GPT2, Our findings also stress the importance of better ranking methods in diffusion research.
Step-wise Analysis against Iterative NAR. Given the underlying theoretical connection between iterative NAR and DIFFUSEQ discussed 3. we empirically investigate the behavior of LevT and DIFFUSEQ by analyzing their step-wise quality (i.c. BLEU) and diversity (i.c. div-4) curves. As is suggested in Figure 5, LevT grows fiercely in quality at the very beginning of generation, and quickly slows down in the successive refinement process. But DIFFUSEQ behaves differently, with BLEU score growing slowly first, increasing rapidly as the diffusion process progresses and finally surpassing LevT. It is also observed that the diversity of both LevT and DIFFUSEQ is determined at the very early stage regardless of future refinement or diffusion, where DIFFUSEQ consistently outperforms LevT diversity any stage of generation. We conjecture that DIFFUSEQ explores more possible results at the first half of generation process, and soon converges to several potential candidates when it is closed to the end of steps. In this case, DIFFUSEQ shows its capacity to take both generation quality and diversity into consideration, and this is the capacity that iterative-NAR and even AR models can not obtain, due the different learning paradigms.
Inference Speed. The slow sampling speed is one of the major concerns about diffusion models. Here we fix the number diffusion steps during training for DIFFUSEQ while shrinking the infer- ence steps following DDIM (Song et al., 2020). As we can see from Figure 6, when reducing the inference 1,000 diffusion steps on single GPU, DIFFUSEQ achieves higher BLEU score than GPre-large yet registers closer inference speed to GPT2-large.
Effectiveness of Joint Training, In DIFFUSEQ, the representations of w and w are jointly trained using the same embedding function EMB(.) (stated in 3). To validate the effectiveness
8

---

Published as a conference paper at ICLR 2023
Table 3: Results with or without joint training for Question Generation task.
Setting
BLEU
R-Lt
Scoret
selfB./ div-41
DIFFUSEQ (w/o reranking) Fix EMB(W") as pre-trained
0.1567
0.3484
0.5947
0.2789/0.8100
0.0110
0.0687
0.3769
0.0174/0.9376
of this joint training strategy, we compared it with the training strategy commonly used in text-to- image diffusion models (Nichol et al., 2022; Ramesh et al., 2022). In particular, we decouple the training of (M") and EMB(w") by replacing EMB(w") with representations extracted from a pre-trained BERT-tiny model (Ture et al., 2019). From Table 3, we find that the decoupled training strategy results in poor performance.
RELATED WORK
Diffusion Models for Text Modeling. Text-to-Image generation using diffusion models has de- veloped many potential applications. Models such as Imagen (Saharia et al., 2022b) and DALL- E (Ramesh et al., 2022) are usually two-staged relying on the pre-trained models, requiring the alignment between the embedding vectors from two sources. GLIDE (Nichol al., 2022) explores diffusion model with classifier-free (Ho & Salimans, 2022) guidance by setting guidance scale dur- ing training. The target space these models is not discrete text space but stable vectors of pixel values. There other works of diffusion on text generation, but they stick to the original encoder- decoder architecture and the diffusion process is interspersed on the decoder (Savinov et al., 2021), or the latent space (Yu et al., 2022).
For text generation using the diffusion models, Hoogeboom et al. (2021) introduce the multinomial diffusion for character-level text generation, the forward categorical noise is applied through the Markov transition matrix. Austin et al. (2021) generalize discrete text diffusion models by introduc- ing the absorbing state ([MASK]). ). However, discrete diffusion models may suffer from the scaling of the one-hot row vectors, and they only generate text samples unconditionally in discrete space. Diffusion-LM (Li al., 2022) and Analog Bits (Chen et al., 2022) propose a new language model diffused on the continuous latent representations, with different mapping functions that connect the discrete and continuous space of texts. Compared with our work, we focus on the SEQ2SEQ dif- fusion models for text generation in the continuous space and our work is the first to explore this setting to the best of our knowledge.
Diffusion Models for Conditional Generation. Related to Sanditional-VAB (Zhao et al., 2017) 7), we can consider the latent encoded input x condition. Diffusion-LM (Li al., 2022) adopts the plug-and-play approaches (Dathathri et al., 2020) to compose fine-grained constraints on the generated sentences, but it fails condition on the whole source sentence in SEQ2SBQ tasks. Noted that this controllable generation method is orthogonal to our DIFFSEQ, in other words, we can fur- ther add classifier-guide constraints on the SEQ2SEQ output to further control the text generation. There are other conditional diffusion models on the time series prediction like CSDI (Tashiro et al., 2021) 1) or audio generation like WaveGrad (Chen et al., 2021), but their class conditions are usually attributes that are easy model, while the contextual texts as conditions are much more complex.
6 CONCLUSIONS
We propose DIFFUSEQ to tackle SEQ2SEQ tasks in a diffusion way, which contains the strong po- tential to achieve better generation quality and diversity trade-off. The capability enables favorable characteristics of DIFFUSEQ to further enhance the quality of final results, by leveraging minimum Bayes risk decoding algorithm. Besides, we theoretically connect the AR and NAR models to DIF- FUSEQ, and show that DIFFUSEQ is powerful extension of iterative-NAR model. The empirical results demonstrate that DIFFU SEQ is also a powerful model for text generation. matching or even surpassing competitive AR, iterative NAR, and large-scale pre-trained models on quality and diver- sity. Given the limited progress of current diffusion models on text generation, our study addresses promising achievements by such new
learning paradigm.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
We would like to thank the anonymous reviewers and other peers for their valuable advice, and we also acknowledge Chenxin An's efforts update the generation results for the Transformer-bn model on QG and Paraphrasing tasks. This work is partially supported by the Shanghai Committee of Science and Technology (Grant No. 21DZ1 100100) and the joint research scheme of the National Natural Science Foundation of China (NSFC) and the Research Grants Council (RGC) under grant number N_HKU714/21.
REFERENCES
Jacob Austin, Daniel D Johnson. Jonathan Ho, Daniel Tarlow, and Rianne van den Berg. Structured denoising diffusion models in discrete state-spaces. Advances in Neural Information Processing Systems, 2021.
Nanxin Chen, Yu Zhang, Heiga Zen, Ron Weiss, Mohammad Norouzi, and William Chan. Wave- grad: Estimating gradients for waveform generation. In International Conference on Learning Representations, ICLR, 2021.
Ting Chen, Ruixiang Zhang, and Geoffrey Hinton. Analog bits: Generating discrete data using diffusion models with self-conditioning, arXiv preprint arXiv:2208.042000 2022.
Kyunghyun Cho, Bart van Merrienboer, Çaglar Gülçehre, Dzmitry Bahdanau, Fethi Bougares, Hol- ger Schwenk, and Yoshua Bengio. Learning phrase representations using rmn encoder-decoder for statistical machine translation. In Conference on Empirical Methods in Natural Language Processing, EMNLP. 2014.
Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosin- ski, and Rosanne Liu. Plug and play language models: A simple approach to controlled text generation. In 8th International Conference on Learning Representations, ICLR, 2020.
Aditya Deshpande, Jyoti Aneja, Liwei Wang, Alexander G Schwing, and David Forsyth. Fast diverse and accurate image captioning guided by part-of-speech. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10695-10704, 2019.
Bhuwan Dhingra, Kathryn Mazaitis, and William W Cohen. Quasar: Datasets for question answer- ing by search and reading, arXiv preprint arXiv: 039904 2017.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. In International Conference Learning Representations, ICLR, 2017.
Wanyu Du, Jianqiao Zhao, Liwei Wang, and Yangfeng Ji. Diverse text generation via variational encoder-decoder models with gaussian process priors. In Proceedings of 6th Workshop on Struc- tured Prediction for NLP of the Association for Computational Linguistics, ACL, 2022.
Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer. Mask pask-predict: Parallel decoding of conditional masked language models. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 61 622621. Association for Computa- tional Linguistics, 2019.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Infor- mation Processing Systems, 2014.
Jiatao Gu, James Bradbury, Caiming Xiong, Victor OK Li, and Richard Socher. Non-autoreeeeessvv neural machine translation. International Conference on Learning Representations, ICLR, 2018.
Jiatao Gu, Changhan Wang, and Junbo Zhao. Levenshtein transformer. Advances in Neural Infor- mation Processing Systems, 2019.
10

---

Published as a conference paper at ICLR 2023
Jonathan Ho and Tim Salimans. 122.: 2022.
Classifier-free diffusion
guidance.
arXiv preprint
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 2020.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious of neural text degeneration. International Conference on Learning Representations, ICLR, 2019.
Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, and Max Welling. Argmax flows and multinomial diffusion: Learning categorical distributions. Advances in Neural Information Processing Systems, 2021.
Fei Huang, Tianhua Tao, Hao Zhou, Lei Li, and Minlie Huang. On the learning an transformers. In International Conference Machine Learning, ICML, 2022.
Chao Jiang, Mounica Maddela, Wuwei Lan, Yang Zhong, and Wei Xu. Neural crf model for sentence alignment in text simplification. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL, 2020.
Diederik P. Kingma and Max Welling. Auto-Encoding Variational Bayes. In International Confer- ence on Learning Representations, ICLR, 2014.
Philipp Koehn. Statistical significance tests for machine translation evaluation. Proceedings of the 2004 conference on empirical methods in natural language processing, pp. 388-395, 2004.
Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. In International Conference on Learming Representations, ICLR, 2020.
Xiang Lisa Li. John Thickstun, Ishaan Guirajani, Percy Liang, and Tatsunori B Hashimoto. Diffusion-Im improves controllable text generation. arXiv preprint arXiv:2205.14217, 2022.
Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, 2004.
Yankai Lin, Haozhe Ji, Zhiyuan Liu, and Maosong Sun. Denoising distantly supervised open- domain question answering. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL, 2018.
Luke Metz, Ben Poole, David Plau, and Jascha Sohl-Dickstein. Unrolled generative adversarial networks. In International Conference on Learning Representations, ICLR, 2017.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew,
Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. In International Conference on Machine Learning, ICML, 2022.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffuuson probabilistic models.
In International Conference on Machine Learning, ICML, 2021.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, ACL, 2002.
Lihua Qian, Hao Zhou, Yu Bao, Mingxuan Wang. Lin Qiu, Weinan Zhang, Yong Yu, and Lei Li.
Glancing transformer for non-autoregressive neural machine translation. Proceedings of the 59th Annual Meeting the Association for Computational Linguistics and the I 11th International Joint Conference on Natural Language Processing VVolue Long Papers), 2021. 2021.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models unsupervised multitask learners. 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140): 2020.
11

---

Published as a conference paper at ICLR 2023
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022.
Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. Palette: Image-to-image diffusion models. In ACM SIGGRAPH 2022 Conference Proceedings, 2022a.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kam- yar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorealistic text-to-image diffusion models with deep language understanding. arXiv preprint arXiv:2205. 14487, 2022b.
Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, Xi Chen, and Xi Chen. Improved techniques for training gans. In Advances in Neural Information Processing Systems, 2016.
Nikolay Savinov, Junyoung Chung, Mikolaj Binkowski, Erich Elsen, and Aaron van den Oord. Step- unrolled denoising autoencoders for text generation. In International Conference on Learning Representations, ICLR, 2021.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Lingnistics (Volume 1: Long Papers), pp. 1715-1725. Association for Computational Linguistics, 2016.
Jascha Sohl-Dickster Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learn- ing, ICML, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In Interna- tional Conference on Learning Representations, ICLR, 2020.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in Neural Information Processing Systems, 32, 2019.
Yusuke Tashiro, Jiaming Song, Yang Song, and Stefano Ermon. Csdi: Conditional score-based diffusion models for probabilistic time series imputation. Advances in Neural Information Pro- cessing Systems, 2021.
Tulia Ture, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. Well-read students learn better: The impact student initialization on knowledge distillation. CoRR, abs/1908 00999088.89... 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, łukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- fion processing systems, 2017.
Ashwin K Vijayakumar, Michael Cogswell, Ramprasath R Selvaraju, Qing Sun, Stefan Lee, David Crandall, and Dhruv Batra. Diverse beam search: Decoding diverse solutions from neural se- quence models. arXiv preprint 1600.226 2016.
Yu, S Xie, X Ma, B Jia, B Pang, R Gao, Zhu, S-C Zhu, and YN Wu. Latent diffusion energy- based model for interpretable text modeling. In International Conference on Machine Learning, ICML, 2022.
Hugh Zhang, Daniel Duckworth, Daphne Ippolito, and Arvind Neelakantan. Trading off diversity and quality in natural language generation. In Proceedings the Workshop on Human Evaluation of NLP Systems (HumEval), pp. 25-33. Association for Computational Linguistics, 2021.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi Bertscore: Evalu- ating text generation with bert. In International Conference on Learning Representations, ICLR, 2019.
12

---

Published as a conference paper at ICLR 2023
Tiancheng Zhao, Ran Zhao, and Maxine Eskenazi. Learning discourse-level diversity for neural dialog models using conditional variational autoencoders. In Proceedings of the 55th Anmual Meeting of the Association for Computational Linguistics, 2017.
Hao Zhou, Tom Young, Minlie Huang, Haizhou Zhao, Jingfang Xu, and Xiaoyan Zhu. Common- sense knowledge aware conversation generation with graph attention. In International Joint Con- ference on Artificial Intelligence. IJCAI, 2018.
Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. Texygen: A benchmarking platform for text generation models. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, pp. 1097-1100, 2018.
13

---

Published as a conference paper at ICLR 2023
A OBJECTIVE DERIVATIONS OF DIFFUSEQ
The diffusion model is well-known as its ability to achieve the trade-off between flexibility and tractability of the models' probability distributions, compared with GAN, VAE and Flow-based models. Following Ho et al. (2020); Nichol & Dhariwal (2021); ); Song al. (2020), we systematically define the forward noising process and reverse denoising process on latent continuous space z.
The forward noising is to perturb the structure of data Zo. is finally changed into the partial Gaussian noise with yr (O, I) through Ttep forward random disturbance
q(z: zı-1) NN(z v1 11
(10)
with 1 = 1, T and (Be E (0, 1)]l=1 are the variance schedule. Let Cr. 1 3r and az I Cg. we have:
Zt -1 vI 014-1 1400 V 0204-191-2
(II)
Vaiza VI-ārt, āre,
where e stands for Gaussian noises. the end, ( = (1
We use
sqrt noise schedule in Diffusion-LM (Li et al., 2022), that is, ār = = vt/T+ with s as small constant the start of noise level. The reverse process then denoises 2,, aiming to recover original and is defined as:
T
(Z = p(2r) II
(2- (0 (21), (o(24-1/2,) =
(00(()))
(12)
The learning of pe is based diffusion model DIFFUSEQ: fe(z,, t), where the (-) and on(-) is the predicted parameterization of the mean and standard variation of g(z (2 in forward process. Using Bayes' rule:
9(Z)-1 1Z0) Zo)
(12 = g(2, 12 20)
(13)
(Zz (2))
Substitute Eq. (11) to it and we can get the parameterized mean of q(zz Ze, Zo):
(z zo) yar(1 -ā-1)
Ot. 2, 18 20.
(14)
and for brevity, we short the coefficient of z; and Zo as li and & respectively.
We can use the variational lower bound to optimize the negative log-likelihood E|- (x)) < LVLB- The objective can be further rewritten to be a combination of several KL-divergence and entropy terms following Sohl-Dickstein al. (2015).
log g(ZT (20)
g(z |20, Ze) log
LVLB Lr LT-1 + Lo E By(zn. (2o))
po (Z(T))
po po 1
(15)
log go(zo Zo wr
log po "oowwww"" @w 2o)
(21)
For 1 T 1, we compute the parameterization of L, by substituting Eq. (14) minimize the difference from and pig following Ho et al. (2020):
qZz 2o, 2t+1)
= Exg [allen(es 2n) - -400 Laa .)12
L. Ezo
log
Po 11+1))
(16)
Ezo
Z U2, + Ezo - (Uz + E.(((2,,)) ())p
E Rzo 01120 - (o(2,t)113)
14

---

Published as a conference paper at ICLR 2023
where C = 2|/0o||* 12 is loss independent constant. Then the optimization of training loss ming LVLB can be further simplified as:
min 0
[1(2)))11 + Σ
I20 fo t)11 ||EMB(w"My) "(y) - -f((zz 1))12 log (w by (
1=2
+ minn
1100 So(z,, 1)112 ||EMB Sa(z1, 1)112 - log ( (20)
min
Ilyo fo(2,,))11 ||EMB(w")
RR(12011 12)
0
(17)
B GRAPHICAL MODELS OF AR, FULLY NAR, ITERATIVE NAR AND DIFFUSEQ MODELS
We start from the conditional sequence generation problem, which aims to learn conditional proba- bility p( (ww.. 1:m w") with w and w AR models learn p( p(w (w=) by autoregressive decomposition
based on left-context:
PAR(Wi) /w") p(w"/
II
p( /W / w")
(18)
initial prediction
progressive left-context prediction
consisting of an initial prediction and an autoregressive left-context prediction process, while fully- NAR models (Gu et al., 2018; Qian et al., 2021) learn the conditional probability given independent assumption for fast inference:
PARR (w IT)
II 1....
p(w p(w"l("))
(19)
To make a better analogy to AR and NAR models, we use a lossless way to formulate iterative NAR models (Gu al., 2019; Ghazvininejad et al., 2019) by introducing series of intermediate sequences w Wi:k-1 w w as:
Piter-NAR ( (w
imm )
Σ
"(w"lw")
II k=1...
(ww++llw. w")
w. W
Σ
p(w lx))
II I...--
p( (wwk+1/we Iwg. w
(20)
p(w", (w2)
II
p(w
Wi w
w wh
K
initial prediction
progressive full-context prediction
Previous study (Huang et al., 2022) shows that there is a gap called conditional total correlation between AR and fully-NAR learning paradigms, because the lossy decomposition of NAR mod- els. This gap is mainly responsible for the performance drop from AR to NAR models. However, when comparing iter-NAR, Eq. (20), with AR models, they both can be factorized into an initial prediction term and progressive prediction process based on different context (i.e. left-context in AR and full-context in iter-NAR). The discrepancy as pointed out by Huang et al. (2022) is there- fore closed in iter-NAR assuming sufficient steps. By showing DIFFUSEQ is extension of the iter-NAR model, we offer justification that will not suffer from the conditional total correlation for the same reason.
15

---

Published as a conference paper at ICLR 2023
A straight- forward way to formulate naive diffusion models is to introduce series of Gaussian noise-corrupedd features Y1.r-1.Yo = y.Yr ~ (O,,)) on continuous space as:
Pdiffusion ( (w wx)
p(w Yo: w²)
II plyc-1lye, w*)
(21)
yr No
final step prediction progressive fulll context diffusion
Il p(w"lvo...")) Yr 1
II II p(yr-1,ly,,w")
(22)
t= 1.,,,,ii
where p(yr-1ly,, w*) describes the diffusion process contiguous representations The total number of diffusion steps is denoted as T. Thereafter we omit the independent decomposition on w and yr. To apply diffusion models on discrete space, the rounding operation in DIFFUSEQ maps the continuous vectors y to discrete w for each time step 4, we hence addition introduce both contiguous feature y and the discrete text w to represent the discrete text into Eq. (21):
pl w/ wh) = Σ
p(w w
II TT--.....
p(w.lye. w Oppr.///
(23)
w wi..... we yo
Σ
o(w"lyo w²)
w²)
(24)
w w.
yo....
T=.....
p(w0) yo, w²)
p ply::l/ w")
(25)
yo Yo
wh T......
p( yo, w*)
II ww
(26)
Yo
By rearranging Eq. (23) and Eq. (24), we can see that DIFFUSEQ can be seen as more generalized form of ite before marginalizing out .yr,....Yul where Eq. (23) and Eq. (24) are equivalent with different computation order, despite the different initialization yr. For NAR models, yr uniform copied from the source sentence or unk's 's token embedding (Gu et al., 2018), for diffusion models, yr is sampled from normal distribution (0,
It is notable that unlike AR and fully NAR models generating text all at once, iterative NAR and diffusion models feature a self-corrected text generation process. The graphical comparison shown in Figure 7.
C FROM DIFFUSEQ TO ITERATIVE NAR AND DIFFUSION MODELS
From DIFFUSEQ to Iterative NAR We show how to derive DIFFUSEQ to iterative non- autoregressive model on discrete space.
16

---

Published as a conference paper at ICLR 2023
AR
DiffuSeq
yx
yrS
yIA
FuRR
W
Yr-13
YI-12
YT-14
Ito-NAR
1.3
FL
y12
y1m
:
K
11
12
Fol
Vta
W
003
Figure 7: Graphical Models of AR, Fully NAR, iterative NAR and DIFFUSEQ models. For sim- plicity, we omit source node w". Gray nodes indicate dependency on the source node while white nodes are independent to the source node.
PDnrUSDo w///w"
pyw w
II wly--//?)powWll>> w*) T=.....1
Wi No
p(wflyr.
II p(w/ly,, w "p((../ww
reorder computation
I= I.....0
Σ
p(WWlyYT w
p(w/lye w mixel
Σ
pyw w
pww
marginalize over y
I.....
p(w
II K-1
w*)
align and reversely.
w.y
NAR w"))
From DIFFUSEQ to diffusion model diffusion model on continuous space.
We show how to derive DIFFUSEQ to the straight-forward
17

---

Published as a conference paper at ICLR 2023
PDIFFUSEO (w
Σ
pyw w
II pww=-//e)ppw//xxw. (-
W wy Vo...No
p(w" No. w²)
II
Σ
plyy-1/w")p(w/lle w*)
Yo
p(w" w²)
II w")
marginalize over w
No
t= T.
Paiffusion w ww.)
D DETAILS OF EXPERIMENTS
D.1 PROCESSING OF QUESTION GENERATION DATASET
To construct high-quality document-question pairs from the Quasar-T dataset, which consists of (document; question, answer) triplets, we extract (documents, question) pairs if answer ex- actly matches document,. After pre-processing, we obtain I 119K document-question training pairs.
D.2 SETTINGS BASELINES
We compare the settings of different models, including the number of parameters and how to sample the different output sentences, as shown in Table 4. For plain GRU-based encoder-decoder methods, we do not implement diversity search algorithms on it, thus its sentence-level diversity could be very poor. For NAR-LevT, we set max iteration 9 and follow the termination condition mentioned in the original paper. For GPVAE-T5, we tune the scalar to find the best trade-off between quality and diversity on the dev The scalars of all four tasks are 2. We implement GPT2 baselines using HuggingFace Trans formers and for the baseline Transformer-base we use Fairsee rseq.
Table 4: The comparison for different models
Models
# Parameters
Learning Paradigm
Diversity Source
GRU-attention
65M
encoder-decoder encoder-decoder
Transformer-base
80M
TBmperature/DBS
GPT2-base FT
117M
pretrain-finetune
Hybrid strategy Hybrid strategy Gaussian sampling
GPT2-large FT GPVAE-T5
774M
pretrain-finetune
220M
pretrain+VAE
NeR- DIFFUSEQ
80M
non-autoreerressive non-autoregressive
91M
Gaussian sampling
D.3 DIVERSITY AND QUALITY TRADE-OFF SETTINGS
We list the details to obtain Figure 4. For GPVAE-T5, set different scalars as 1, 3, 4. For DIFFUSEQ, we choose trained models at different training steps to achieve different trade-off points. For other baselines, there is explicit factor control the diversity generation, leave them as single points in the figure.
'Including top-p sampling, temperature, diversity beam search (DBS) and etc. Implement using Hugging- Face transfoorer https: / /gi thub. com hoog/ www.ce/traasaooooooeoeo
18

---

Published as a conference paper at ICLR 2023
D.4 METRICS
The used BLEU score is sentence-level smoothed from BLEU-1 to 4, and used ROUGE-L score longest common subsequence based statistics. The implementation based NLTK* and torchmetr rimetrrccs cs. The n-gram based metrics may fail to capture the semantic meaning of sentences, so we consider using SeorS Specifically, we use microsoft/ besberta-xlarg--ai to help BERTScore correlate better with human scores.
D.5 GENERATION RESULTS
For different tasks we list some generation examples. As we can see Table 5, Table 6 and Table 7, DIFFUSEQ tends to generate diverse outputs, but sometimes the sentence is not as fluent finetuned GPT2.
Table 5:
Sample outputs with different random seed in Dialogue test set.
Utterance: How long does the dye last?
Response: Just did this days ago, not sure how it'll fade yet!
GPVAE-T5
NAR-LeyT
* I'm not sure, I'm not sure. I've tested a few times, but don't 't know for sure. I've
* half
I'm not sure. I'm not sure how long lasts, I'm sure it 'll get better. It's been a while since
half.
I've been using it for about a year and a half. I've been using it for about a year and a half.
half.
GPT2-large finetune
DIFFUSEQ
* Two weeks in my case.
* About an hour, days or so.
I've had it for about a year.
4 days.
The dye can sit around for a month then you can wash
I'm not sure about this, about the same kind of time.
Table Sample outputs with different random seed in Question Generation test set.
Statement: The Japanese yen is the official and only currency recognized in Japan.
Question: What is the Japanese currency?
GPVAE-T5
NAR-LevT
What the japanese currency
What the basic unit of currency for Japan
What is the japanese currency
What is the basic unit of currency for Japan ?
What the japanese currency
* What the basic unit of currency for Japan
GPT2-large finetune
DIFFUSEQ
What is the basic unit of currency for Japan?
What the Japanese currency
What the Japanese currency
Which country uses the "yen yen" in currency
What the basic unit of currency for Japan?
What the basic unit of currency?
html
19

---

Published as a conference paper at ICLR 2023
Table 7: Sample outputs with different random seed in Text Simplification test set.
Complex sentence: People experience loneliness for many reasons, and many life events may cause it, such as a lack of friendship relations during childhood and adolescence, or the physical absence meaningful people around a person. Simplified: One cause of loneliness is a lack of friends during childhood and teenage years.
GPVAE-T5
NAR-LevT
People experience loneliness for many reasons, and many life events may cause it, such as lack of friendship relations during childhood and adolescence, or the physical absence of meaningful people around person
People may experience reashapphap- phapphaphhaphappahout life reasit.
People can experience loneliness for many reasons, and many life events may cause it, such as lack of friendship relations during childhood and adolescence, or the physical absence of meaningful people around person
People may experience reashapphap- phapphaphapphappahot life reasit.
People can experience loneliness for many reasons, and many life events may cause it, such as a lack of friendship relations during childhood and adolescence, the physical absence of meaningful people around person
People may experience reashapphap- phapphapphaphappabou life reasit.
GPT2-large finetune
DIFFUSEQ
Loneliness can be caused by many things.
Many life events may cause of loneliness
People can also be very experience loneliness for many reasons.
Loneliness can affect people in many ways.
People can experience loneliness for many reasons, and many life events may, cause it.
Loneliness can be caused by many things.
20


Reviews:
[{"review_id": "ls__RgerkSn", "reviewer": "Reviewer_NoKK", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper constructs a conditional (sequence-to-sequence) generative diffusion model for text: DIFFUSEQ. Methodologically, the work is quite similar to Diffusion-LM (Li et al., 2022) with the notable adoption of importance sampling (following Nichol and Dhariwal, 2021) to improve training efficiency. Experimentally, this work evaluates DIFFUSEQ using automatic quantitative metrics against a strong collection of baseline methods in four conditional generation tasks: dialog, question generation, text simplification, and paraphrasing.", "summary_of_the_review": "This paper is an interesting empirical study of seq2seq text diffusion models, with some (fixable) issues in the writing.", "strength_and_weaknesses": "Methodologically, this work is a minor extension of Diffusion-LM (Li et al., 2022). Nevertheless, this style of text generation model is quite new, so the empirical work evaluating these models for seq2seq modeling tasks is both novel and interesting as it helps to build the community's emerging understanding of these moels. The chosen collection of seq2seq modeling tasks and datasets are interesting, and the choice of baseline methods for comparison are both strong and thorough.\n\nThe empirical evaluation compares DIFFUSEQ's performance in each of 4 tasks to (1) encoder-decoder autoregressive models (both GRU and Transformer) (2) fine-tuned LLM's (GPT2 and T5) and (3) the non-autoregressive Levenshtein transformer. DIFFUSEQ's quantitative performance is strong compared to all baselines, and is even competitive with finetuned pre-trained LLM's: this is impressive given that DIFFUSEQ is a smaller model, trained on much smaller datasets.\n\nI also appreciate the architectural ablation of the use of joint versus separate embeddings for sequence pairs (Table 3). One additional ablation that could be interesting to see is a comparison of DIFFUSEQ trained with and without importance sampling. I believe this is the first paper to apply importance sampling to text diffusion models, and so it would be quite interesting to see the effect of this adjustment.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "ZXXV-qUtOyP", "reviewer": "Reviewer_jTcw", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces conditional diffusion models in the NLP setting, demonstrates its empirical advantages, and draws theoretical connections with autoregressive and iterative-nonautoregressive models.", "summary_of_the_review": "Overall a good paper with few loose ends, but writing, clarity, and story-telling could be improved.", "strength_and_weaknesses": "The paper leaves good overall impression due to high-quality writing, propose positioning against state of the art, and good empirical study. Story-telling could be improved with proper background/notation introduction, clearing any abuse of notation, offloading any derivations to supplementary, and focusing on key findings and theoretical conclusions being drawn. Particularly, \"connections\" paragraph seems to be overly detailed, yet Eq. 1 and 2 are neither too detailed nor distilled to the barebones. I appreciate the author's usage of equation annotations, yet I found them hard to follow anyway.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "OxWKJhJB-GW", "reviewer": "Reviewer_juT2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies the application of denoising diffusion probabilistic models to seq2seq text generation. Authors propose DiffuSeq - a diffusion-based model for seq2seq task using a technique they call \"partially noising\". Authors also show the theoretical connection between DiffuSeq and standard AR / NAR  language models. They evaluate DiffuSeq on a number of seq2seq tasks such as dialogue generation and paraphrasing, against several autoregressive and non-autoregressive baselines, achieving comparable generation quality. On top of that, authors note that DiffuSeq generates more diverse outputs. Authors analyze the behavior of the proposed model, report inference throughput and ablate several design decisions.", "summary_of_the_review": "I left a generally negative review, mainly because I do not see the need for the extra complexity of DiffuSeq, compared to simply treating the source text x as an extra input. My concern is based on how input text is treated in existing diffusion-based text-to-image models[1,2]. \nOutside of that, the paper appears to be generally decent, with both strong and weak points as described above. I am open to discussion and, _should authors prove that my main concern is wrong_, increasing my score.\n\n", "strength_and_weaknesses": "\n### Strengths\n\n- the proposed method is evaluated in multiple seq2seq tasks, supporting the claim for general seq2seq modeling\n- authors attempt to show the connection between DiffuSeq and existing seq2seq strategies\n- the paper is reasonably well-written and easy to follow\n\n### Weaknesses\n\nI have a single main concern, justifying my current (negative) recommendation -- the theoretical need for a new type of denoising diffusion specific to Seq2Seq problems. The other weaknesses i found are fairly minor and are listed at the bottom.\n\nAs authors note in the paper, prior works have already adapted denoising diffusion to unconditional language modeling. To extend their results to seq2seq problems, one must find a way to incorporate additional inputs to the diffusion model. In continuous \"text-to-image\" diffusion, this can be done by simply feeding the extra inputs to the model.\nModern diffusion models for conditional image generation, such as Imagen[1] or DALLE-2 [2] simply attend to the input text inside the diffusion model __without any changes to the underlying diffusion or noise distribution__.\n\nHence, my main question: __does DiffuSeq really need the extra complexity of \"partially noising\", instead of just treating x as inputs?__ To the best of my understanding, the x part is not generated by the model, but frozen throughout the diffusion process. As a result, the model does not learn to denoise it, and can effectively treat it as an extra input to the existing (unconditional) text diffusion methods. Please correct me if my understanding is flawed in any way.\n\nIf DiffuSeq does *not* need the extra math, then this paper effectively becomes an empirical study of existing diffusion models for seq2seq applications, which leads to my __second concern: non-standard evaluation tasks__. As authors state in the paper, seq2seq is a very popular and competitive area of research with many popular datasets. However, the datasets/tasks chosen in the paper are not among the popular ones (see [3]), which makes it difficult to assess whether the baseline models are state-of-the-art for those tasks.\n\n\n\n[1] https://arxiv.org/abs/2205.11487\n\n[2] https://cdn.openai.com/papers/dall-e-2.pdf\n\n[3] https://paperswithcode.com/area/natural-language-processing", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "15oAfZj8eDy", "reviewer": "Reviewer_LMuu", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes diffuseq – a diffusion model for sequence-to-sequence text generation tasks. The x and y pairs are concatenated together and sent to the forward diffusion process. Different with diffusion-lm’s classifier-guided diffusion, diffuseq here uses classifier-free diffusion guided by points in space.  Similar with diffusion-lm, an embedding on words is earned jointly with the diffusion/reverse processes.\nExperiments on rich text generation tasks show that the proposed diffuseq could achieve better results than strong pretrained language model baselines or vae variant baselines.\n", "summary_of_the_review": "The concatenate of x and y together for diffusion is the major novel part with a well-designed objective function. Most parts of the paper are well written. The score will for sure be improved if equation 2 can be enriched and more comparison with strong baselines such as diffusion-lm can be given here.\n\nThis paper is above the average level and higher scores are deserved with the updating of the current unclear parts.\n\n-- \n\nafter reading the authors' feedbacks, I decide to rank higher this paper. Basically, the general idea of seq2seq with diffusion is an interesting direction and many related seq2seq tasks should benefit from it. It is good to see that the authors enriched their core ideas and replace some less important parts of page 4 to the appendix, hopefully, this makes this paper easier to be followed.\n\n[still, a bit doubt about that diffusion-LM can not deal with seq2seq problems since in Diffusion-LM's paper, a number of controllable text-to-text generation tasks were reported... anyway, this only influence the comparison part of this paper, not the central idea of this seq2seq-diffusion models.]\n", "strength_and_weaknesses": "Strong:\n\n1. Diffuseq framework, with new objective for training and concatenates x (given) and y (target) for diffusion model training and consequent inferencing;\n\n2. Better results compared with a list of strong baselines.\n\nWeak:\n\n1. Different with the paper’s arguments, it is a bit hard to find the significant difference in terms of technical novelty with existing diffusion-LM – concatenating x and y actually almost doubles the diffusion space and possibly makes the training/inferencing speed to be much slower than diffusion-lm – a detailed experimental comparison with diffusion-lm is required. \n\n2. Some essential parts need to be enriched – such as equation 2. And most equations in page 4 are less important to be included – give the spaces to equation 2 – which is your novel objective function. (appendix A’s extension is far from enough, such as how EMB’s item square was removed).\n\n\nDetailed questions and comments\n\n1. Figure 1, the t=T/2 figures are real examples? Frequently in my experiments, the figures are clear only at the last several steps and did not achieve the clear results at t=T/2 steps.\n2. Also in figure 1, can you make it clearer the difference of “guidance” in diffuseq vs. “condition” in diffusion-LM – align these differences in figures with the specific equations’ differences.\n3. Equation 2, more details are preferred and even appendix A is not clear enough of from the two equations: such as how the EMB included item changed from square to without square. I could not find the details in appendix A.\n4. Page 4’s equations are less important (most are not novel points of this paper) and I think they can be reduced or part of them can be moved to appendix – leaving more spaces for explaining equation 2 – which is your own objectives.\n5. The “1” involved in equation 2 is a bit ambiguous. Basing on current description, I think it is t=1, not a simple 1 and prefer to see how exactly you implemented this in your code – did you use t=[0, 1] or t=[0, T]?\n6. Table 1, can you also give the results of diffusion-LM? More than half baselines selected here are less comparable than existing diffusion-based generation models.\n7. In “inference speed”, “1000 diffusion steps” a similar inference speed. I am not able to understand this part. Any detailed reasons on this? How about the speed comparison on other tasks besides “question generation”? does the same speech comparison results hold?\n8. Table 3, right-bottom item, “0.9376” is larger than “0.8103” and why this happen? Any detailed error analysis on this? If you say diffusion-seq is strong at diversity, then this result actually did not align with that.\n\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster