Diffusion-LM Improves Controllable Text Generation
Authors: Xiang Lisa Li
John Thickstun
Ishaan Gulrajani
Percy Liang
Tatsunori Hashimoto
Venue: NeurIPS 2022 Accept (2022)
Keywords: controllable text generation
controlled generation
infilling
language model
diffusion model

Abstract: Controlling the behavior of language models (LMs) without re-training is a major open problem in natural language generation. While recent works have demonstrated successes on controlling simple sentence attributes (e.g., sentiment), there has been little progress on complex, fine-grained controls (e.g., syntactic structure). To address this challenge, we develop a new non-autoregressive language model based on continuous diffusions that we call Diffusion-LM. Building upon the recent successes of diffusion models in continuous domains, Diffusion-LM iteratively denoises a sequence of Gaussian vectors into word vectors, yielding a sequence of intermediate latent variables. The continuous, hierarchical nature of these intermediate variables enables a simple gradient-based algorithm to perform complex, controllable generation tasks. We demonstrate successful control of Diffusion-LM for six challenging fine-grained control tasks, significantly outperforming prior work.

Diffusion-LM Improves Controllable Text Generation
Xiang Lisa Li Stanford University
John Thickstun Stanford University
Ishaan Gulrajani
Stanford Univeristy
jthickst@stanford. edu
igul mrul@stanfordedd
Percy Liang
Tatsunori B. Hashimoto
Stanford Univeristy
Stanford Univeristy
pliang@cs wtt.:csssssaaao.deeee.o.oo
Stashim@staa.oord.eee
Abstract
Controlling the behavior of language models (LMs) without re-training is a major open problem in natural language generation. While recent works have demon- strated successes on controlling simple sentence attributes (c.g., sentiment), there has been little progress complex, fine-grate fine-grained controls (e.g., syntactic structure). To address this challenge, we develop a new non-autoregressive language model based continuous diffusions that call Diffusion-LM. Building upon the recent successes of diffusion models in continuous domains, Diffusion-LMM iteratively denoises a sequence of Gaussian vectors into word vectors, yielding sequence of intermediate latent variables. The continuous, hierarchical nature of these inter- mediate variables enables simple gradient-baat based algorithm to perform complex, controllable generation tasks. We demonstrate successful control of Diffusion-LMM for six challenging fine-graine enn-igrined control tasks, significantly outperforming prior work
Introduction
Large autoregressive language models (LMs) are capable of generating high quality text 39 but in order to reliably deploy these LMs in real world applications, the text generation process needs be controllable: we need to generate text that satisfies desired requirements (e.g. topic, syntactic structure). A natural approach for controlling a LM would be to fine-tune the LM using supervised data of the form (control, text) [T]. However, updating the LM parameters for each control task can be expensive and does not allow for compositions of multiple controls (c.g. generate text that both positive sentiment and non-toxic). This motivates light- and modular glug-and-play approaches [6] that keep the LM frozen and steer the generation process using external classifier that measures how well the generated text satisfies the control. But steering frozen autoregressive LM has been shown to be difficult, and existing successes have been limited to simple, attribute-level controls (e.g., sentiment or topic)
56
(55)
In order to tackle more complex controls, we propose Diffusion-LM, new language model based on continuous diffusions. Diffusion-LM starts with sequence of Gaussian noise vectors and incrementally denoises them into vectors corresponding to words, as shown in Figure [ These gradual denoising steps produce hierarchy continuous latent representations. We find that this hierarchical and continuous latent variable enables simple, gradient-basse based methods to perform complex control tasks such constraining the parse tree of generated sequence.
Continuous diffusion models have been extremely successful vision and audio domains 13
41
4 but they have been applied text because of the inherently discrete nature of text
"Code is available at ott.hhaii.tttgee//...t.ooo..///a
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Gaussian
Gradually Danoising
Word
Xyy
1
X y
Xg
Diffusion-LM
Starbucks is coffee shop.
Query Classifler
Gradient Update
Classifier
Parse Tree
Vero Phonge
Noun PITDDO
Noun Phraso
Figure I:
Diffusion-LM iteratively denoises sequence of Gaussian vectors into word vectors, yield- ing a intermediate latent variables of decreasing noise level XT. ... Xp- iteratively perform gradient updates on these continuous latents optimize for fluency (parametrized by Diffusion-LM) and satisfy control requirements (parametrized by a classifier).
For controllable generation, we
(3). Adapting this class of models text requires several modifications standard diffusions:
we add an embedding step and a rounding step to the standard diffusion process, design training objective learn the embedding. and propose techniques to improve rounding (S4). We control
Diffusion- LM using gradient- based method, as shown in Figure || This method enables us to steer text generation process towards outputs that satisfy target structural and semantic controls. iteratively performs gradient updates on the continuous latent variables of Diffusion-LMM to balance fluency and control satisfaction $5.T
To demonstrate control of Diffusion-LM,, we consider six control targets ranging from fine-grained attributes (e.g., semantic content) to complex structures (e.g.. parse trees). Our method almost doubles the success rate of previous alug-and-play methods and matches or outperforms the fine-tuning oracle on all these classifier-uuide control tasks (7.11. In addition these individual control tasks, we show that we can successfully compose multiple classifier classifier-guided guided controls to generate sentences with both desired semantic content and syntactic structure (7.2). Finally, we consider span-anchored controls, such length control and infilling. Diffusion-L.M allows us to perform these control tasks without classifier, and our Diffusion-L.M significantly outperforms prior plug-and-play methods and is on-par with an autoregressive LM trained from scratch for the infilling task (
2 Related Work
Diffusion Models for Text. Diffusion models 47 have demonstrated great success in continuous
data domains 1 33 24, 31), producing images and audio that have state-of-the-art sample quality. To handle discrete data, past works have studied text diffusion models on discrete state spaces, which defines corruption process on discrete data (e.g., each token has some probability to be corrupted to an absorbing or random token) I [6]. In this paper, we focus on continuous diffusion models for text and to the best of our knowledge,
work is the first to explore this setting. In contrast to discrete diffusion LMs, our continuous diffusion LMs induce continuous latent representations, which enables efficient gradient- based methods for controllable generation.
Autoregressive and Non-autoregressive LMs. Most large pre-trained LMs are left-to-right au- toregressive (e.g., GPT-3 B PaLM 55). The fixed generation order limits the models' flexibility in many controllable generation settings, especially those that impose controls globally on both left and right contexts. One example is infilling, which imposes lexical control on the right contexts; another example is syntactic structure control, which controls global properties involving both left and right contexts. Since autoregressive LMS cannot directly condition on right contexts, prior works have developed specialized training and decoding techniques for these tasks 46 36 For example, Qin et al. 37 proposed a decoding method that relaxes the discrete LM outputs to continu- variables and backpropagates gradient information from the right context. Diffusson LM can condition on arbitrary classifiers that look at complex, global properties of the sentence. There are other non-au autorerrsssve LMs that have been developed for machine translation and speech-to-text tasks 12 43 However these methods are specialized for speech and translation settings, where the entropy over valid outputs is low, and whether they work for language modeling remains open problem. We leave detailed discussions to Appendix
Plug-and-Play Controllable Generation. ald--and-play controllable generation aims to keep the LM frozen and steer its output using potential functions (c.g., classifiers). Given probabilistic

---

potential function that measures how well the generated text satisfies the desired control, the generated text should be optimized for both control satisfaction (measured by the potential function) and fluency (measured by LM probabilities). There are several plug-and-plas approaches based on autoregressive L.Ms: FUDGE 55 reweights the LM prediction each token with an estimate of control satisfaction for the partial sequence; GeDi 25 and DExperts [28] reweight the LM prediction at each token with
a smaller LM finetuned/trained for the control task.
The closest work to ours is PPLM
K which runs gradient ascent on an autoregressive LM's hidden activations to steer the next token satisfy the control and maintain fluency. Because PPLM based on autoregressive LMs, it can only generate left-to-right. This prevents PPLM from repairing and recovering errors made in previous generation steps. Despite their success on attribute (e.g., topic) controls, we will show these alug-and-play methods for autoregressive LMs fail on more complex control tasks such as controlling syntactic structure and semantic content $7.1 We demonstrate that Diffusion-L.M is capable of plug-and-play controllable generation by applying classifier-guided gradient updates to the continuous sequence of latent variables induced by the Diffusion-L.M.
3
Problem Statement and Background
We first define controllable generation (3.1 and then review continuous diffusion models $3.3
3.1 Generative Models and Controllable Generation for Text
Text generation is the task of sampling w from a trained language model Pim (w),, where w [w]... w.] is sequence of discrete words and pio (w) probability distribution over sequences of words. Controllable text generation is the task of sampling w from a conditional distribution p(w c), where denotes control variable. For syntactic control, с can be a target syntax tree (Figure I while for sentiment control, could be desired sentiment label. The goal of controllable generation is to generate w that satisfies the control target c.
Consider the plug-and- play controllable generation setting: we are given a language model Pmm(w) (w) trained from large amount of unlabeled text data, and for each control task, we given classifier p(c w) trained from smaller amount of labeled text data (e.g., for syntactic control, the classifier probabilistic parser). The goal is to utilize these two models to approximately sample from the posterior p(w | c) via Bayes rule p(w c) x Pim Pmm(w)- p(c fluent, and the p(c w) encourages
w). Here, Pim (w))) encourages w to be fulfill the control.
3.2 Autoregressive Language Models
The canonical approach language modeling factors Pmm in autoregressive left-to-right mannar, Pim (w)) = Pmm (wn) II: [mn
Fes). In this case, text generation is reduced to the task of repeatedly predicting the next token conditioned on the partial sequence generated far. The next token prediction Pim (z; Tes) is often parametrized by Transformer architecture [52]
3.3 Diffusion Models for Continuous Domains
A diffusion model
I 33] is a latent variable model that models the data Xo E Rd as Markov chain x.... Xo with each variable in R&d, and XT is Gaussian. The diffusion model incrementally denoises the sequence of latent variables XT:1 to approximate samples from the target data distri- bution (Figure 2. The initial state pe (xT) s (0, I), and each denoising transition X; + x,-1 is parametrized by the model pe (xt-1 xx) = (x; us (x,, t), Ee (x(xx,)) t)). For example, μp and Me may be computed by a U-Net or Tranformer.
To train the diffusion model, we define forward process that constructs the intermediate latent variables X1:7- The forward process incrementally adds Gaussian noise to data Xy until, at diffusion step T, samples Xy are approximately Gaussian. Each transition X₁-1 - X; is parametrized by q(x, x,- = NXx;; VT- B,x,-1 3,I), where the hyperparameter 3r is the amount of noise added at diffusion step I. This parametrization of the forward process q contains no trainable parameters and allows us to define a training objective that involves generating noisy data according to pre-defined forward process g and training model to reverse the process and reconstruct the data.
3

---

Gaussian Noise
Rounding
Text
Pe (X Xz)
Pe (w(w
Xo)
XT
>...
X
X4
XO
W
(xx
x
Xg w)
Noising
Embedding
Figure 2: A graphical model representing the forward and reverse diffusion processes. addition to the original diffusion models [13]. add Markov transition between Xo and w, and propose the embedding $4.1 and rounding $4.2
techniques.
The diffusion model is trained to maximize the marginal likelihood of the data Exo Dlg (xo)],
and the canonical objective is the variational lower bound of log pe (xo)
((x||x))
log g(xx-1 Xu. Xt)
Lvib (Xo)
log
log po (xo
(( Xxo)
(x()
po (xt-1 xx)
t=2
However, this objective can be unstable and require many optimization tricks to stabilize 33 To
circumvent this issue, Ho et al. [13] devised a simple surrogate objective that expands and reweights each KL-divergence term in Lubb obtain mean-squared error loss (derivation in Appendix which we will refer to as
sample (Xo)
Σ
E || t) fi(x,, (x)x xo) ((xe(x0)
1=1
where (x,,,, Xo) the mean of the posterior q(x;-1 No. xx) which is a closed from Gaussian, and ue (x(x f) is the predicted mean of (x(-1 xx) computed by neural network. While Csimple is no longer valid lower bound, prior work has found that empirically made training more stable and improved sample quality We will make use of similar simplifications in Diffusion-LM to stabilize
training and improve sample quality (S..I.
Diffusion-LM: Continuous Diffusion Language Modeling
Constructing Diffusion-LM requires several modifications to the standard diffusion model. First, must define an embedding function that maps discrete text into a continuous space. To address this, we propose an end-to-end training objective for learning embeddings (84.1). Second, we require rounding method to map vectors in embedding space back to words. To address this, we propose training and decoding time methods to facilitate rounding (4.2p.
4.1 End-to-end Training
To apply a continuous diffusion model to discrete text, we define an embedding function EMB (w:) that maps each word to a vector Rd. We define the embedding of a sequence w of length n to be: EMB(w) (EMB(W1),
EMB( (w.))] E IRnd
We propose a modification of the diffusion model training objective (Equation D that jointly learns the diffusion model's ''s parameters and word embeddings. In preliminary experiments, we explored random Gaussian embeddings, as well as pre-trained word embeddings (35 39). We found that these fixed embeddings are suboptimal for Diffusion-L.M compared to end-to-end training
As shown Figure 2 our approach adds Markov transition from discrete words w X in the forward process, parametrized by 4s Xx) w) (EMB(WW (w), Gol). In the reverse process, we add trainable rounding step, parametrized by pa Xo) = Pe po(w: xx), where po x;)) is softmax distribution. The training objectives introduced in $3 now becomes
Leze (w)
E (xowx/
(Xv) log (Xxo [x) log pe (w/xx))]]
(2)
simple (w)
E To Sxxor///w
[Csimplo (Xoo) + ||MB (w) - (4e -ogpollo
"Our definition of Lumple here uses different parametrization from Ho et al. loss in terms of ps (x, while they express in terms of (x,,,,).
13 We define our squared
"While trainable embeddings perform best on control and generation tasks, we found that fixed embeddings onto the vocabulary simplex were helpful when optimizing for held-out perplexity. We leave discussion of this approach and perplexity results to Appendix K as the focus of this work is generation quality and not perplexity.

---

Learned Embeddings
NOUN
W from [ following We derive simple
PROPN
AUX
the simplification in and our derivation de- tails are in Appendix Since we are training the embedding function, now contains trainable parameters and we use the reparametrization trick 42 20 to backpropagate through this sam- pling step. Empirically, we find the learned em- beddings cluster meaningfully: words with the same part-of-speech tags (syntactic role) tend to be clustered, as shown in Figure
VERB
DET
AD)
ADV
SCONJ
Figure 3: A t-SNE [51] plot of the learned word
embeddings. Each word colored by its POS.
4.2 Reducing Rounding Errors
The learned embeddings define a mapping from discrete text to the continuous Xo. We now describe the inverse process of rounding a predicted X back to discrete text. Rounding is achieved by choosing
the most probable word for each position, according to argmax Po(w (w Xx) = II==1 Po Po(w; xz)- Ideally, this argmax- roundiin would be sufficient to map back to discrete text, as the denoising steps should ensure that Xy lies exactly on the embedding of some word. However, empirically, the model fails to generate X, that commits to single word.
One explanation for this phenomenon is that the Csimple(Xo) term in objective 2 puts in- sufficient emphasis on modeling the structure of Xo- Recall that we defined Ssimple (x))
E Ex, (x, (xx,, xo)ll where our model po(x,, () directly predicts the mean of Px(x,-1 X+) for each denoising step 1. In this objective, the constraint that Xo has to commit single word embedding will only appear in the terms with near 0, and we found that this parametrization required careful tuning to force the objective to emphasize those terms (see Ap- pendix Mp.
Our approach re-parametrizes Csimple to force Diffusion-LM to explicitly model Xo in every term of the objective. Specifically, derive analogue to Simple which is parametrized via Xo. Xxo-simple (Xo) = Σ Ex. [xf t) - xol1 where our model fe(x,, t) predicts xo directly This forces the neural network to predict xy in every term and we found that models trained with this objective quickly learn that Xo should precisely centered at a word embedding.
LeZe
We described how ce-parametrization can be helpful for model training, but also found that the same intuition could be used at decoding time in a technique that we call the clamping trick. In the standard generation approach for Xo xo-parametrized model, the model denoises X; to x;-1 by first computing an estimate of Xo via fo (xx, t) and then sampling x conditioned on this estimate: X Vā /ax,, +1+ āe, where ae = -I'' (1- 3.) and ~ NN n In the clamping trick, the model additionally maps the predicted vector fo(x,, 1) to its nearest word embedding sequence. Now, the sampling step becomes X;-1 = vā. Clamp( fx(x,, + v1 ar. The clamping trick forces the predicted vector to commit to a word for intermediate diffusion steps, making the vector predictions more precise and reducing rounding errors
Decoding and Controllable Generation with Diffusion-IM
Having described the Diffusion-L.M. we now consider the problem of controllable text generation (5.D) and decoding (5.2)
'Predicting Xo and x;-1 is equivalent up to scaling constants the distribution of x,-1 can be obtained in closed form via the forward pracess x VTxp + /1 āe, see Appendix J for further details.
"This follows from the marginal distribution q(x, | xp), which is a closed form Gaussian since all the Markov transitions are Gaussian.
'Intuitively, applying clamping trick early diffusion steps with t near T may be sub-optimal, because the model hasn't figured out what words to commit to. Empirically, applying clamping trick for all diffusion steps doesn't it hurt the performance much. But to follow this intuition, one could also set the starting step of the clamping trick as hyperparameter.
5

---

5.1 Controllable Text Generation
We now describe a procedure that enables plug-and-plas control on Diffusion-LM. Our approach to control is inspired by the Bayesian formulation in [3.1 but instead of performing control directly on the discrete text, we perform control on the sequence of continuous latent variables X0:T defined by Diffusion-LM, and apply the rounding step to convert these latents into text.
Controlling Xo.T is equivalent to decoding from the posterior p( p(xo.r(c) II p px,,11 Xr, c), and we decompose this joint inference problem to sequence of control problems at each diffusion step: p(x;-1 Xg, c) p(x--1 x²) p(c x x,). We further simplify p(c x(-1.X)) p(c
X;-1) ) via conditional independence assumptions from prior work on controlling diffusions [49 Consequently, for the t-th step, we run gradient update on X
Vx, logp(xe-1 Xr, c) Vx,
pxx xx) Vx, logp(c xx-1);
where both log p( (x+-1 xx) and logp(e p(c x(-1) are differentiable: the first term parametrized by Diffusion-LMM and the second term is parametrized by neural network classifier.
train the classifier on the diffusion latent variables and run gradient updates on the latent space x to steer it towards fulfilling the control. These image diffusion works take one gradient step towards Vx, log p(c x,-1) per diffusion steps. To improve performance on text and speed up decoding, we introduce two key modifications: fluency regularization and multiple gradient steps.
Similar work in the image setting 8
40
To generate fluent text, we run gradient updates on a control objective with fluency regularization: A log p(x;-1 xx) + log p(c X;-1). where À is hyperparameter that trades off fluency (the first term) and control (the second term). While existing controllable generation methods for diffusions do not include the À log X2) term in the objective, we found this term to be instrumental for generating fluent text. The resulting controllable generation process can be viewed as a stochastic decoding method that balances maximizing and sampling p(x;-1 Xx, c), much like popular text generation techniques such as nucleus sampling [14] or sampling with low temperature. In order to improve the control quality, we take multiple gradient steps for each diffusion step: we run 3 steps of the Adagrad [00] update for each diffusion steps. To mitigate for the increased computation cost, we downsample the diffusion steps from 2000 200, which speeds up our controllable generation algorithm without hurting sample quality much.
5.2 Minimum Bayes Risk Decoding
Many conditional text generation tasks require a single high-quality output sequence, such as machine translation or sentence infilling. In these settings, we apply Minimum Bayes Risk (MBR) decoding to aggregate a set of samples S drawn from the Diffusion-LM. and select the sample that achieves the minimum expected risk under a loss function (e.g., negative BLEU score): w argmin Ew'es [ST ((w, w'). We found that MBR decoding often returned high quality outputs, since low quality sample would be dissimilar from the remaining samples and penalized by the loss function.
26
6 Experimental Setup
With the above improvements on training S84) and decoding {5) we train Diffusion-Lar for two language modeling tasks. We then apply the controllable generation method to 5 glssidfier-guided control tasks, and apply MBR decoding to classifier-free control task (i.e. infilling).
6.1 Datasets and Hyperparameters
We train Diffusion-L.M on two datasets: E2E RE and ROCStories 12
The E2E dataset consists of 50K restaurant reviews labeled by fields including food type, price, and customer rating. The ROCStories dataset consists of 98K five-sentence stories, capturing rich set of causal and temporal
We tried ablations that replaced Adagrad with SGD, but we found Adagrad to be substantially less sensitive hyperparameter tuning.

---

input (Semantic Content)
food Japanese
output
Browns Cumbridge good for Japanese food and also children friendly near The Sorrento
input (Parts-of-speech)
PROPN AUX DET ADJ NOUN NOUN VERB ADP DET NOUN ADP DET NOUN PUNCT Zizzi is a local coffee shop Incated on the outskirts of the city
output text
input (Syntax Tree)
(TOP (S (NP (*) (*5) (*9) (VP (*) (NP (NP (*) (*)))))
output text
The Twenty Two has great food
input (Syntax Spans)
(7, 10, VP)
ourput text
Wildwood pub multicultural dishes and is ranked 3 stars
input (Length)
14
output text
Browns Cambridge offers Japanese food located near The Sorrento in the city centre..
input (left context)
My dog loved tennis balls.
input (right context)
My dog stolen every one and put it under there.
output text
One day. I found all of my lost tennis balls underneath the bed.
Table 1: Example input control and output text for each control tasks.
commonsense relations between daily events. This dataset is more challenging to model than E2E, because the stories contain larger vocabulary of 11K words and more diverse semantic content.
Our Diffusion-LM is based on Transformer [52] architecture with 80M parameters, with a sequence length = 64, diffusion steps 7 - 2000 and a square-root noise schedule (see Appendix A for details). We treat the embedding dimension as a hyperparameter, setting d = 16 for E2E and for ROCStories. See Appendix
28
B for hyperparameter details. At decoding time, we downsample
to 200 diffusion steps for E2E and maintain 2000 steps for ROCStories. Decoding Diffusion-LM for 200 steps is still 7x slower than decoding autoregressive LMs. For controllable generation, method based on Diffusion-LM is 1.5x slower than FUDGE but 60x faster than PPLM.
6.2 Control tasks
We consider 6 control tasks shown in Table I the first 4 tasks rely on a classifier, and the last 2 tasks are classifier free For each control task (e.g. semantic content), we sample 200 control targets c (e.g., rating=5 star) from the validation splits, and we generate 50 samples for each control target. To evaluate the fluency of the generated text, we follow the prior works [55 6 and feed the generated text to a teacher LM (i.c.. carefully fine-tuned GPT-2 model) and report the perplexity of generated text under the teacher LM. We call this metric Im-score (denoted as Im): a lower Im-score indicates better sample quality. We define success metrics for each control task follows:
Semantic Content. Given field (e.g.. rating) and value (e.g.. star), generate sentence that covers field=value, and report the success rate by exact match of 'value'.
Parts-of-speech. Given a sequence of parts-of-speech (POS) tags (e.g., Pronoun Verb Determiner Noun), generate a sequence of words of the same length whose POS tags (under an oracle POS tagger) match the target (e.g.. I ate an apple). We quantify success via word-level exact match.
Syntax Tree. Given a target syntactic parse tree (see Figure []. generate text whose syntactic parse matches the given parse. To evaluate the success, we parse the generated text by off-the- shellff parser 222, and report F1 scores.
Syntax Spans. Given a target (span, syntactic category) pair, generate text whose parse tree over span [ matches the target syntactic category (e.g. prepositional phrase). We quantify success via the fraction of spans that match exactly.
Length. Given target length 10,..., 40, generate a sequence with length within 4:2 of the target. In the case of Diffusion-LMN LM, we treat this as a classifier- free control task.
Infilling. Given left context (O₂) and a right context (O2) from the aNLG dataset 2 and the goal to generate a sentence that logically connects O₁ and O2 (algorithm details in Appendix G For evaluation, we report both automatic and human evaluation from the Genie leaderboard 19
'Length classifier-frre for Diffusion-LMM based methods, but other methods still require classifier. use GPT (38) as the teacher LM whereas we use a fine-tuned GPT-2 model because
"Prior works 55 d our base autoregressive LM and Diffusion-LA, -M both generate UNK tokens, which does not exist in pretrained vocabularies of GPT.
7

---

Semantic
Content
Parts-of-speech
Syntax Tree
Syntax Spans
Length
ctrl t
Im
ctrl f Im 4
ctrl
Im
ctll t
Im |
ctrl
Im 4
PPLM
9.9
5.32
FUDGE 69.9
2.83
27.0
7.96
17.9
3.39
54.2
4.03
46.9
3.11
Diffusion-L.M
81.2
2.55
90.0
5.16
86.0
3.71
93.8
2.53
99.9
2.16
FT-sample
72.5
2.87
89.5
4.72
64.8
5.72
26.3
2.88
98.1
3.84
FT-search
89.9
1.78
93.0
3.31
76.4
3.24
54.4
2.19
100.0
1.83
Table 2: Diffusion-LM achieves high success rate (ctrl 1) and good fluency (Im L) across all 5 control tasks, outperforming the PPLM and FUDGE baselines. Our method even outperforms the fine-tuning oracle (FT) controlling syntactic parse trees and spans.
6.3 Clasiffer-Guided Control Baselines
For the first 5 control tasks, we compare our method with PPLM, FUDGE, and a fine-tuning oracle. Both PPLM and FUDGE are plug-and-play controllable generation approaches based on an autoregressive LM, which train from scratch using the GPT-2 small architecture [39]
PPLM
This method runs gradient ascent on the LM activations increase the classifier proba- bilities and language model probabilities, and has been successful on simple attribute control. We apply PPLM control semantic content, but not the remaining 4 tasks which require positional information, as PPLM's classifier lacks positional information.
For each control task, FUDGE requires future discriminator that takes in prefix sequence and predicts whether the complete sequence would satisfy the constraint. At decoding time, FUDGE reweights the LM prediction by the discriminator scores.
FUDGE 55
FT. For each control task, we fine-tune GPT-2 on (control, text) pair, yielding an oracle conditional language model that's not plug-and -ladp We report both the sampling (with temperature 1.0) and beam search (with beam size 4) outputs of the fine-tuned models, denoted as FT-sample and FT-search.
6.4 Infilling Baselines
We compare to 3 specialized baseline methods developed in past work for the infilling task.
DELOREAN 36 This method continuously relaxes the output space of left-to-right autoregressive
LM, and iteratively performs gradient updates on the continuous space to enforce fluent connection to the right contexts. This yields continuous vector which is rounded back to text.
COLD COLD specifies an energy- based model that includes fluency (from left-to-right and right-to-left LM) and coherence constraints (from lexical overlap). It samples continuous vectors from this energy-basee basee model and round them to text.
AR-infilling, We train an autoregressive LM from scratch to do sentence infilling task 9 Similar to training Diffusion-LM, we train on the ROCStories dataset, but pre-process it by reordering sentences from (O,, middde O2) to (O,,O2, Omidale). At evaluation time, we feed in Ö₁, O2, and the model generates the middle sentence.
Main Results
We train Diffusion-LMMr on the E2E and ROCStories datasets. In terms of negative log- li-liikllhood (NLL,, lower is better), find that the variational upper bound of Diffusion-IMM NLL 10 underper- forms the equivalent autoregressive Transformer model (2.28 vs. 1.77 for E2F, 3.88 vs 3.05 for ROCStories) although scaling up model and dataset size partially bridges the gap (3.88 -> 3.10
ROCStories). Our best log-likelihoods required several modifications from S we explain these and give detailed log-likelihood results in Appendix K Despite worse likelihoods, controllable generation based on our Diffusion-LM results in significantly better outputs than systems based on autoregressive
LMs, we will show in $7.1 $7.2 and $7.3
7.1 Classifier- Guided Controllable Text Generation Results
As shown in Table 2 Diffusion-LM achieves high success and fluency across all classifier-guides control tasks. It significantly outperforms the PPLM and FUDGE baselines across all tasks.
10 Exact log- likelihoods are intractable for Diffusion-LMN LM, report the lower bound C

---

Syntactic Purse
(NP ) (VP* (NP ( (VP* (NP (ADJP +))))))) (VP
ADJP ( ADJP }))))
FUDGE
Zizzi cheap restaurant [incomplete]
Diffusion-LM
Zizzi pub providing family friendly Indian food Its customer rating is low
FT
Cocum is Pub serving moderately priced meals and the customer rating is high
Syntactic Parse
(S(S V (Pp* (PP*(NP *)))** (NP ****) (VP* * (NP ( NP + y (SBAR WHNP (
VP (NP**)))) }))))))*
FUDGE
coffee and fast food place named The Cricketers which family friendly with customer rating of 5 out of
the city near The Portland
Diffusion-LM
Located on the riverside, The Rice Boat is restaurant that serves Indian food.
FT
Located near The Sorrento, The Mill pub that serves Indian cuisine.
Table 3: Qualitative examples from the Syntax Tree control. The syntactic parse tree is linearized by nested brackets representing the constituents, and we use the standard PTB syntactic categories. Tokens within each span are represented as We color failing spans red and bold the spans of interest that we discuss in $7.1
Semantic Content Syntax Tree
Semantic Content Parts-of-speech
semantic ctri t syntax ctrl f
semantic etrl f
POS ctrl t Im
FUDGE
61.7
15.4
3.52
64.5
24.1
3.52
Diffusion-LM
69.8
74.8
5.92
63.7
69.1
3.46
FT-PoE
61.7
29.2
2.77
29.4
10.5
2.97
Table 4: In this experiment, compose semantic control and syntactic control: Diffusion-LM achieves higher success rate (ctrl f) at some cost of fluency (Im 4). Our method outperforms both FUDGE and FT-PoE (product of experts of two fine-tuned models) on control success rate, especially for the structured syntactic controls (i.e. syntactic parse tree and POS).
Surprisingly, our method outperforms the fine-tuning oracle on controlling syntactic parse trees and spans, while achieving similar performance on the remaining tasks.
Controlling syntactic parse trees and spans are challenging tasks for fine-tuning, because conditioning on the parse tree requires reasoning about the nested structure of the parse tree, and conditioning on spans requires lookahead planning to ensure the right constituent appears at the target position.
We observe that PPLM fails in semantic content controls and conjecture that this is because PPLM is designed to control coarse-grand poarse-grained attributes, and may not useful for more targeted tasks such as enforcing that a restaurant review contains reference to Starbucks.
FUDGE performs well on semantic content control but does not perform well on the remaining four tasks. Controlling a structured output -Parts-of-speech and Syntax Tree) is hard for FUDGE because making one mistake anywhere in the prefix makes the discriminator assign low probabilities to all continuations. In other control tasks that require planning (Length and Syntax Spans), the future discriminator difficult train, as must implicitly perform lookahead planning.
The non-autoregressive nature of our Diffusion-LMM allows it to easily solve all the tasks that require precise future planning (Syntax Spans and Length). We believe that it works well for complex controls that involve global structures (Parts-of-speect) Syntax Tree) because the coarse-to- representations allow the classifier exert control on the entire sequence (near t TT) as well as on individual tokens (near = 0).
Qualitative Results. Table shows samples of Syntax Tree control. Our method and fine-tuning both provide fluent sentences that mostly satisfy controls, whereas FUDGE deviates from the constraints after the first few words. One key difference between our method and fine-tuning is that Diffusion-LMM is able to correct for a failed span and have suffix spans match the target. the first example, the generated span ("Family friendly Indian food") is wrong because it contains 1 more word than the target. Fortunately, this error doesn't propagate to later spans, since Diffusion-LM adjusts by dropping the conjunction.
7.2 Composition of Controls
One unique capability of plug-and-play controllable generation is its modularity. Given classifiers for multiple independent tasks, gradient guided control makes it simple to generate from the intersection of multiple controls by taking gradients on the sum of the classifier log- prgbbabbbiliiies.

---

Automatic Eval
Human Eval
BLEU-4
ROUGE-L1
CIDEr t BERTScore t
Left-only DELOREAN
0.9
16.3
3.5
38.5
n/a
1.6
7.9
41.7
n/a
COLD
1.8
19.5
10.7
42.7
n/a
Diffusion
7.1
28.3
30.7
89.0
0.37
10.03
-0.02
AR
6.7
27.0
26.9
89.0
0.39
+0.02 -0.03
Table 5: For sentence infilling, Diffusion-LM significantly outperforms prior work COLD ti Delorean [36] (numbeer taken from paper), and matches the performance of an autoregressive LM (AR) trained from scratch to do infilling.
and
We evaluate this setting the combination of Semantic Content Syntax Tree control and Semantic Content + Parts-of-speech control. As shown in Table our Diffusion-L.M achieves a high success rate for both of the two components, whereas FUDGE gives up the more global syntactic control. This is expected because FUDGE fails to control syntax on its own.
Fine-tuned models are good at POS and semantic content control individually but do not compose these two controls well by product of experts (PoE), leading to a large drop in success rates for both constraints.
7.3 Infilling Results
As shown in 5 diffusion LM significantly outperforms continuous relaxation based methods for infilling (COLD and DELOREAN). Moreover, our method achieves comparable performance to fine-tuning specialized model for this task. Our method has slightly better automatic evaluation scores and the human evaluation found no statistically significant improvement for either method. These results suggest that Diffusion LM solve many types of controllable generation tasks that depend on generation order or lexical constraints (such as infilling) without specialized training.
7.4 Ablation Studies
We verify the importance of our proposed de- sign choices in [4 through two ablation studies. We measure the sample quality of Diffusion-
Imnescore (ower is better)
Leamed.ys.. Bandom Embaddings
Detammttiioaioon
Ea
LM using the Im-score on 500 samples $6.2
Learned v.s. Random Embeddings (4.1)
Learned embeddings outperform random beddings on the ROCStories, which is a harder language modeling task. The same trend holds for the E2E dataset but with a smaller margin.
2
random cmb
128
16
64
128
Figure 4: We measure the impact of our proposed design choices through Im-score. We find both learned embeddings and reparametrization substan- tially improves sample quality.
Objective Parametrization ($4.2). We pro- pose to let the diffusion model predict Xy di- rectly. Here, compare this with standard parametrization in image generation which parametrizes by the noise term e. Figure (right) shows that parametrizing by Xa consistently attains good performance across dimensions, whereas parametrizing by works fine for small dimensions, but quickly collapses for larger dimensions.
8 Conclusion and Limitations
We proposed Diffusion-L.M. novel and controllable language model based continuous diffusions, which enables new forms of complex fine-grained control tasks. We demonstrate Ciffusion-LMPi success in 6 fine- fine-grained control tasks: our method almost doubles the control success rate of prior methods and is competitive with baseline fine-tuning methods that require additional training.
We find the complex controls enabled by Diffusion-LM to be compelling, and we are excited by how Diffusion-L.M a substantial departure from the current paradigm discrete autoregressive generation. As with any new technologies, there are drawbacks to the Diffusion-LMs that we constructed: (1) it has higher perplexity; (2) decoding is substantially slower; and (3) training converges more slowly. We believe that with more follow-up work and optimization, many of these issues can be addressed, and this approach will turn out to be a compelling way do controllable generation at scale.
10

---

Acknowledgments and Disclosure of Funding
We thank Yang Song, Jason Eisner, Tianyi Zhang, Rohan Taori, Xuechen Li, Niladri Chatterji. and the members of p-lambda group for early discussions and feedbacks. We gratefully acknowledge the support of a PECASE award. Xiang Lisa Li is supported by a Stanford Graduate Fellowship.

---

References
[1] Jacob Austin, Daniel D. Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg. Structured denoising diffusion models in discrete spaces. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021. URL T https: http::////ppeeevieee.. http://ww.ee///ooooiidiii----
[2] Chandra Bhagavatula, Ronan Le Bras, Chaitanya Malaviya, Keisuke Sakaguchi, Ari Holtzman, Hannah Rashkin, Doug Downey, Wen tau Yih, and Yejin Choi Abductive commonsense reasoning. In International Conference on Learning Representations, 2020. URL httpa //peerrvi ew.
HKDD
[3]
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Ad- vances in Neural Information Processing Systems, volume 33, pages 1877-1901, Curran Associates, Inc., 2020. URL htpp///pp//// nww. wttpppppppp/////////e.../ pd
4457c0d6bfcb45ee 111444886bbbe 8b www.0foacc422f44440
[4] Nanxin Chen, Yu Zhang, Heiga Zen, Ron J Weiss, Mohammad Norouzi, and William Chan. Wavegrad: Estimating gradients for waveform generation. In International Conference on Learn- ing Representations, 2021. URL httpttttss/:/peeeeee
net. wwt./ee//oorua forum?i d-N.aM.jcFaa8..
[5] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hrrrhng Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam M. Shazeer. Vinodkumar Prabhakaran, Emily Reif, Nan Du, Benton C. Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcfa, Vedant Mísra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Oliveira Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang. Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathleen S. Mellstmellstemm, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.0233 2022.
[6] Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu. Plug and play language models: simple approach to controlled text generation. In International Conference on Learning Representations, 2020. URL httpa:
//openreviex.co wtt.:e/wweeeeeteoommeaa-aa-aeeee
[7] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. ArXiv, abs/1810.04805, 2019.
[8] Prafulla Dhariwal and Alexander Quinn Nichol. Diffusion models beat GANs image synthesis. In A. Beygelzimer, Y. Dauphin, P. Liang, J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021. URL httttst:///ppeetet wtt.:/wwwwee//oomtttt/.
5 1 AAAAAUCVZaV
[9] Chris Donahue, Mina Lee, and Percy Liang. Enabling language models to fill in the blanks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2492-2501. Online, July 2020. Association for Computational Linguistics. doi: 18.53/5//v1/ 2020.acl-main.22. m2... URL https: : htt.aacaannnoooooo ogy
www..oor///aa... ww.oor//2000 .ac21.cccccccc -ma n..
[10] John C. Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. In J. Mach. Learn. Res., 2010.
12

---

[11] Marjan Ghazvininejad, Omer Levy. Yinhan Liu, and Luke Zettlemoyer. Mask-pun mask--rediict Parallel decoding of conditional masked language models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP--ICNIPP)) pages 6112-6121, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10. 10.18653/v1/D19-1 1633. URL
[12] Jiatao Gu, James Bradbury, Caiming Xiong. Victor O.K. Li, and Richard Socher. Non- autoregressive neural machine translation. In International Conference on Learning Rep- resentations, 2018. URL httpps:///ppeeeeeiieeee wwt.://oowwnininilinn1
[13] Jonathan Ho, Ajay Jain, and Picter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, pages 1840-6851, 2020.
[14] Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In International Conference on Learning Representations, 2020. URL
https://openres eew. net www.eeeew.eetteeieeetttoooo Forummii-
[15] Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, and Max Welling, Argmax flows and multinomial diffusion: Towards non-autoregressive language models. arXiv preprint arXiv::102..5339, 2021.
[16] Emiel Hoogeboom, Alexey A. Gritsenko, Jasmijn Bastings, Ben Poole, Rianne van den Berg, and Tim Salimans. Autoregressive diffuusion models. In International Conference on Learning Representations, 2022. URL https::/ net/
[17] Lukasz Kaiser, Samy Bengio, Aurko Roy, Ashish Vaswani, Niki Parmar, Jakob Uszkoreit, and Noam Shazeer. Fast decoding in sequence models using discrete latent variables. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 2390-2399, PMLR, 10-15 Jul 2018. URL
mlr /ttpcceeiiigg..rrrrrreee///e//ea pptps//v00kaassrrll.a... html
[18] N. Keskar, B. McCann, L. R. Varshney, Caiming Xiong, and R. Socher. Ctrl: A conditional transformer language model for controllable generation. ArXiv, abs/1909.05858. 2019.
[19] Daniel Khashabi, Gabriel Stanovsky, Jonathan Bragg, Nicholas Lourie, Jungo Kasai, Yejin Choi, Noah A. Smith, and Daniel S. Weld. Genie: A leaderboard for human-in-the-loop evaluation of text generation. ArXiv, abs/2101.06561, 2021.
[20] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. International Confer- ence on Learning Representations (ICLR), 2014.
[21] Diederik P Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. arXiv preprint arXiv:2107.0066 2021.
[22] Nikita Kitaev and Dan Klein. Constituency parsing with self-attentive encoder. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2676-2686 Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10. 00.16633/11/P188 8- 1249. URL www.ltttss:.. / ac anthol hoh. wtg.p/...o....... 1249
[23] Zhifeng Kong and Wei Ping. On fast sampling of diffusion probabilistic models. In ICML Workshop Invertible Neural Networks, Normalizing Flows, and Explicit Likelihood Models, 2021. URL htttt:://ppseeeseeeeee
[24] Zhifeng Kong, Wei Ping. Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. arXiv preprint ar arXiv:2009.09761 2020.
[25] Ben Krause, Akhilesh Deepak Gotmare, Bryan McCann, Nitish Shirish Keskar, Shafiq Joty, Richard Socher, and Nazneen Fatema Rajani. GeDi: Generative Discriminator Guided Sequence Generation. arXiv preprint 20Xx:22009.06367, 2020.
13

---

[26] Shankar Kumar and William Byrne Minimum Bayes-risk decoding for statistical machine translation. In Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics: HL.T-NAACI. 2004, pages 169-176, Boston, Massachusetts, USA, May May 2004. Association for Computational Linguistics. URL
0rrg:///hoo00oo0.0/000000000 org/N0// 1022
[27] Jason Lee. Elman Mansimov, and Kyunghyun Cho. Deterministic non-autoregressive neural sequence modeling by iterative refinement. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1 1173-1182, Brussels, Belgium, October- November 2018. Association for Computational Linguistics. doi: 10. (0.18653/11/D18-111 1149. URL
[28] Alisa Liu, Maarten Sap, Ximing Lu, Swabha Swayamdipta, Chandra Bhagavatula, Noah A. Smith, and Yejin Choi. DExperts: Decoding-time controlled text generation with experts and anti-experts. In Proceedings of the 59th Annual Meeting of the Association for Com- putational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 6691 Online, August 2021. Associa- tion for Computational Linguistics. doi: 11t1665/ww//22221.coooogggg org ac1-1cnq.
URL https
[29] Xuezhe Ma, Chunting Zhou, Xian Li, Graham Neubig, and Eduard Hovy. FlowSeq: Non- autoregressive conditional sequence generation with generative flow. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-JJCNLP), pages 4282-4292 Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/
019-14337
[30] Chenlin Meng. Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun Zhu, and Stefano Ermon. SDEdit: Guided image synthesis and editing with stochastic differential equations. In International Conference on Leurning Representations, 2022. URL htops://openrevieex
EE
[31] Gautam Mittal, Jesse Engel, Curtis Hawthorne, and Ian Simon. Symbolic music generation with diffusion models. arXiv preprini arXiv:2103.1609) March 2021.
[32] Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. A corpus and cloze evaluation for deeper understanding of commonsense stories. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pages 339-849, San Diego, California, June 2016. Association for Computational Linguistics. doi: 10. 1/N16- 1098. URL httttt://ccaatttaaatt /a//caaann http://caantthoooogoooo org/N16- 1098
[33] Alex Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. arXiv preprint (tt.::2222.067722 2021.
[34] Jekaterina Novikova, Ondfej Dušek, and Verena Rieser. The E2E dataset: New challenges for end-to-end generation. In Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue, pages 201-206 Saarbrücken, Germany, August 2017. Association for Computational Linguistics. doi: 06553/111117-5555 URL
[35] Jeffrey Pennington, Richard Socher, and Christopher Manning. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Doha, Qatar, October 2014. Association
Language Processing (EMNLP), pages for Computational Linguistics. doi: 10.3115\v1/D14-111 1162. URL https: / hntho/olccltoooooyyyoooooo
brg/b14-1162
[36] Lianhui Qin, Vered Shwartz, Peter West, Chandra Bhagavatula, Jena D. Hwang. Ronan Le Bras, Antoine Bosselut, and Yejin Choi. Back to the future: Unsupervised backprop- decoding for counterfactual and abductive commonsense reasoning. In Proceedings of the 2020 Con- ference on Empirical Methods in Natural Language Processing (EMNLP)), pages 794-805, Online, November 2020. Association for Computational Linguistics. doi: 10. 10.1886 16553/12220000 emnlp-main.5 URL htttss:/accaattho..et org/
org/2020. 2020. emnlp-main.58
14

---

[37] Lianhui Qin, Sean Welleck, Daniel Khashabi, and Yejin Choi. Cold decoding: Energy-based constrained text generation with langevin dynamics, 2022. URL httpss://azziv..rr////a abs/
2202. 7705
[38] Alec Radford and Karthik Narasimhan. Improving language understanding by generative pre-training. huttps://ppena: wtt.:/o/////./.. http://log///aaaa http:///wggg.aaguaaaeauuueeeeeee
2018.
[39] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever, Language models are unsupervised multitask learners. http:://owoaiiioomooool/lltttaep 2019.
[40] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents, 2022. URL
abs/
2204. 06125
[41] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, April 2022.
[42] Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. arXiv preprint arXiv:1401 1.002, 2014.
[43] Chitwan Saharia, William Chan, Saurabh Saxena, and Mohammad Norouzi. Non-autoreereesive machine translation with latent alignments. In Proceedings the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pages 1098-1 1108, 2020.
[44] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Bureu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho, David J Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion models with deep language understanding, 2022. URL httphttts///rxxi.orrr abs
2205. 11487
[45] Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In International Conference on Learning Representations, 2022. URL htttp:::/pppeeeeeeee
net httt//orumiddss IIIaaxxIpha-
[46] Lei Sha. Gradient-guided unsupervised lexically constrained text generation. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pages 8692-8703, 8703, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/ v1/2020.emnlp main.
arg/2020. smnip-main. 701
[47] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan and Surya Ganguli. Deep unsu- pervised learning using nonequilibrium thermodynamics. In Francis Bach and David Blei, editors, Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 2256 2265, Lille, France, 07-09 Jul 2015. html
PMLR. [eLLittts:/::rrceeeeet /ttp:eeeeeaaaanioopppeeee/////// htt.a//ppp..eee//////aaaaacaaaaa
[48] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations, 2021. URL https://opperrvvicc
net/ httr//oummmi-SStliiraaeaeee
[49] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021. URL httpss://pperrvvattt
net/ Not//oruu?1d-PXTTG..
[50] Mitchell Stern, William Chan, Jamie Kiros, and Jakob Uszkoreit. Insertion transformer: Flexible sequence generation via insertion operations. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 5985. PMLR, 09-15 Jun 2019. URL https://prroceeding mlr. prcss/ wwwppppww/sss..etp V97/stern19a.htm
[51] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. Journal of Machine Learning Research, 2008.
15

---

[52] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Guyon. U. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Ad- vances in Neural Information Processing Systems, volume 30, pages 5998-6008. Curran Associates, Inc., 2017. URL http::///pppcccccci.gg./oooo// new.ueiiinnninnnnnno.oee new.rnnurripp. ctp/p/////////////11l 316o2355544c0110000553332aa2aaa2
[53] Ben Wang and Aran Komatsuzaki, GPT-J-6B: 6 Billion Parameter Autoregressive Language Model. http:////g...aaa.ccc/ooooott/t// tranesoorr-rrsrrr-erreo. jax May 2021.
[54] Chunqi Wang. Ji Zhang, and Haiqing Chen. Semi-autoorerresivve neural machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 479-488, Brussels, Belgium, Oetober-November 2018. Association for Computational Linguistics. doi: 10.18653/v 1/D18-1044
URL https: http:::://acaaatthooooooog ottpooooooooooo111////111
[55] Kevin Yang and Dan Klein. FUDGE: Controlled text generation with future discriminators. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 351 1-3535, Online, June 2021, Association for Computational Linguistics. doi: 10. 18653/v1/202) wtw.///naaalnmmaaa.../a naacl-main.271 URL
https: /ach
antha Togy. 202 naacc -ma a 276
[56] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemooer. Opt: Open pre-trained transformer language models, 2022. URL https //axivv www.:org/aaao. wtt.v/oorrr/aa/220.000000 010668
16


Reviews:
[{"review_id": "_AawjCGFUu2", "reviewer": "Reviewer_Pvpw", "summary": "This paper proposed Diffusion-LM, a non-autoregressive language model based on continuous diffusions, for controllable text generation. The non-autoregressive model would iteratively denoise a sequence of Gaussian vectors into word vectors, and the control information is injected by running gradient updates on intermediate latent variables. The main contribution of this paper is that it demonstrated how to combine non-autoregressive model and continuous diffusion model and use it for controllable text generation. The experiment result shows that Diffusion-LM significantly outperforms prior work.", "questions": "1. I'm not sure about the training process, is this understanding right: the output vectors of each decoding pass of the non-autoregressive model are called latent variables, and these variables are used to predict the control information. After the diffusion process, the final output  would compute a cross entropy loss with the reference.\n\n2. I don't see the reason why evaluating on syntax-related tasks. Are these tasks useful in real scenario? Or they are just deigned tasks to highlight the strengths of Diffusion-LM?", "limitations": "The authors had mentioned their limitations in the paper: higher PPL and slower training, and the training process itself might need large GPU memory. The fewer diffusion steps might speed up decoding process but needs further exploration to avoid performance loss.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The writing is good and the paper is well presented.\n2. Detailed derivations are included, and the experiments are adequate and demonstrate good performance.\n3. Combining the diffusion model from CV and non-autoregressive model is not a trivial task, this paper explores the changes and improvements needed to apply Diffusion-LM on controllable text generation.\n\nWeakness:\n1. The training speed is much slower and the generated text suffers from higher PPL.\n2. Some downstream tasks still need three classifiers and the cost for these classifiers is high.\n3. Some downstream tasks may lack practical application.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Sn1JT7ucCHN", "reviewer": "Reviewer_sK4s", "summary": "This paper introduces a new non-autoregressive language model based on continuous diffusion models, which have been quite successful in the vision area. Specifically, this paper proposes a novel training objective to train an embedding function that can convert the discrete text tokens into a continuous space. Correspondingly, this paper also introduces methods to improve the rounding from a continuous latent variable to discrete text. The abovementioned techniques facilitate training a Diffusion-LM. Besides, this paper shows that the control over text generation can be done on continuous latent space and the proposed approach yields successful control of Diffusion-LM for six fine-grained control tasks compared to prior work.", "questions": "- Line 77-78, I don't think the cited paper has shown that current non-autoregressive approaches fail for language modeling. Would you provide some more details here?\n- Line 174, I do not quite understand the following sentence:\n\" a mixture of Dirac delta distribution will only appear in the terms with $t$ near zero\". Could you elaborate more?\n- In addition to the fine-grained text control tasks, I wonder how the proposed model performs on coarse-grained tasks, e.g., controlling simple sentence attributes like sentiment?\n", "limitations": "Yes", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- Very well-written, I enjoyed reading the paper\n- There have been few works exploring diffusion models on NLP and this is the first work to explore continuous diffusion models on NLP. It looks definitely interesting and could potentially inspire others to keep working along this line.\n- The proposed end-to-end training looks effective in learning embeddings.\n- Empirical results demonstrate the success of text control for *fine-grained* control tasks.\n\n\n**Weakness**\n- My primary concern is the scalability of Diffusion-LM. Though under the current experiment setup, i.e., 80M parameters for Diffusion-LM and GPT-2 Small, the proposed approach outperforms existing methods based on autoregressive LM on text control. One of the advantages of existing works is that they can use arbitrary large pre-trained LMs that are available. It is not clear to me how the scaling low works for the current proposed Diffusion-LM and text control approach.\n- Though Diffusion-LM achieves better performance on text control, I wonder what could be the most appealing reason to choose a Diffusion-LM over an autoregressive LM generally? As I understand, autoregressive LMs can benefit many downstream NLP tasks in addition to the text control. I am curious about what justifies the application of Diffusion-LM here.\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Ndth2hOoKO", "reviewer": "Reviewer_C1jB", "summary": "Recent studies on controllable text generation often fall short of complex controls like syntactic structure. This paper proposes Diffusion-LM that adopts the idea of denoising diffusion probabilistic models, which have been proven effective, in the vision domain, to naturally incorporate the gradient information from an external classifier for the controls of interest. The main challenge of applying diffusion models to text generation is the discrete nature of language data. The authors propose to introduce an embedding step and its reverse rounding step as the solution. Based on such a remedy, the rest of the model is constructed and trained very similarly to the original DDPM model. Plus, the authors also propose an alternative objective to reduce rounding errors by reparameterizing the model to explicitly predict $x_0$ at every timestep. \n\nExperiments on two datasets and six different control tasks show that the proposed diffusion-LM yields acceptable perplexity and superior performance for controllable generation.", "questions": "- Inaccurate terminology. For controls like syntactic structures, the control signal comes from an external parser. I hence suggest using another term rather than “classifier” for sources of control signals.", "limitations": "Most limitations of this paper have been discussed in the paper. ", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. Very novel methodology contribution that introduces continuous diffusion probabilistic models into text domain with domain-specific adaption. To the best of my knowledge, this is one of the first (or early) successful attempts to study diffusion models for language generation.\n2. Impressive performance on controllable text generation, which may motivate further studies on this research problem.\n3. The paper is well-organized and mostly clearly described.\n\nWeaknesses:\n\n1. Yet inefficiency in decoding. \n2. The model runs upon sequences with short and fixed lengths (64 tokens), which may prevent the approach from generalizing to more practical cases. Are there any results about longer or more flexible/dynamic length choices?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "23EDFY0qvCD", "reviewer": "Reviewer_GmR4", "summary": "This paper seems to be the first application of diffusion models for (discrete) language data. To deal with the discrete nature, they propose, to my understanding, a softmax layer that maps dense vectors into a distribution over the vocabulary. Another proposed technique is to force the neural network to predict x_0 in every term. The authors test the proposed diffusion models on several datasets and show better results than a set of baseline methods.", "questions": "1. Lots of important details are given in Appendix, including the modeling choices for realizing all functions and I really do not know how the classifiers for these control tasks are implemented. I also don't know how infilling is completed.\n2. Although this work makes a successful use of diffusion methods. A question is: why would one study diffusion model in the first place? The authors mainly compare diffusion models with autoregressive models to illustrate their advantages. But how about non-autoregressive  models such as https://aclanthology.org/D19-1633.pdf ? I feel non-autoregressive models also (1) enjoy continuous latent representations,  (2) generate all tokens in parallel. and (3) induce a coarse-to-fine hierarchy. In fact, the first and last points are rather vague since any multi-layer neural network can be interpreted in such ways. I feel the discussion with iterative non-autoregressive generation is insufficient.", "limitations": "n/a", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The proposed methods seems effective compared to previous plug-and-play controllable generation approaches.\n2. People may be interested in such a successful application of diffusion methods in NLP.\n3. The proposed methods are simple and well-examined by ablation studies.\n\nWeaknesses:\n1. The report of computational cost is missing (though the authors mentioned that the decoding is substantially slower).\n2. I feel it is better to show some human evaluation results regarding the results in Table 1. \n3. Might not be a (major) weakness. But I have a hard time reading this paper. The writing is very abstract. I feel important deduction and concrete design are always missing.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept