Diffusion-LM Improves Controllable Text
Generation
Xiang Lisa Li, John Thickstun, Ishaan Gulrajani, Percy Liang,Tatsunori B. Hashimoto
NeurIPS 2022

Max Mehta

1Outline
Motivation
Related Works
Approach
Experiments
Results
Conclusion2

Motivation

Diffusion-LM

3

Motivation➔Controlling the behavior of language models (LMs)◆ autoregressive (text generation) ⟶ controllable (real-world deployment)◆ simple sentence attributes (sentiment) ⟶ complex, ﬁne-grained control (syntactic structure)➔Fine-tune with supervised data (control, text)◆ expensive◆ ❌ multiple controls➔

Plug-and-play is the answer◆ LM frozen - external classiﬁer (guides generation + satisfy control)◆ Guiding frozen autoregressive LM is hard - limited to sentiment or topic

4

Proposal➔

Diﬀusion-LM◆ non-autoregressive◆ continuous diﬀusion➔

Continuous diﬀusion for discrete text (novel) - need modiﬁcations◆ embedding step & rounding step➔

Six control tasks (4 classiﬁer-guided + 2 classiﬁer-free) - semantic & structure◆ individual control tasks + multiple classiﬁer-guided controls◆ outperforms/on-par with prior plug-and-play & autoregressive LM

5Related Works

Diffusion-LM6Latent Diffusion
Denoising diﬀusion probabilistic models (DDPM)
Markov Chain
Ho, J., Jain, A., & Abbeel, P. (2020). Denoising Diffusion Probabilistic Models. ArXiv. /abs/2006.11239

Denoising diﬀusion implicit models (DDIM)
Non-Markovian
FasterDeterministic
Random noise iszeroed
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2021). High-Resolution Image Synthesis with Latent
Diﬀusion Models. ArXiv. /abs/2112.10752

ForwardSong, J., Meng, C., & Ermon, S. (2020). Denoising Diﬀusion Implicit Models. ArXiv. /abs/2010.02502

Reverse (Generative)noiseapproximation
model (Trained)7

Plug and Play Language Models➔

Controllable text generation➔

Pretrained LM + attribute classiﬁers➔

Modular (no retraining needed)➔

Good performance◆ high ﬂuency (perplexity)◆ control (classiﬁer accuracy)◆ human eval➔

Simple coarse-grained control◆ sentiment◆ switching topic

Dathathri, S., Madotto, A., Lan, J., Hung, J., Frank, E., Molino, P., Yosinski, J., & Liu, R. (2019).
Plug and Play Language Models: A Simple Approach to Controlled Text Generation. ArXiv.
/abs/1912.02164

8

Approach

Diffusion-LM

9

Diffusion-LM➔

Controllable text generation - sample w from a conditional distribution p(w | c)
◆ w = [w1 ···wn] - sequence of discrete words◆ c - control variable◆ Goal: generate w that satisﬁes the control target c➔

Diﬀusion-LM (continuous + non-autoregressive)➔Gaussian noise vectors ⟶ (denoise) ⟶ continuous latent representation ⟶ (denoise) ⟶ word vectors◆ gradient-based methods - complex control tasks

10Diffusion-LM Modifications➔

Continuous diﬀusion for discrete text (novel) - need modiﬁcations➔

Embedding function◆ discrete text ⟶ continuous space◆ end-to-end training objective (learn embeddings)➔Rounding method◆ vectors in embedding space ⟶ discrete text
◆ re-parametrization + clamping

11

End-to-end Training➔

Embedding function
sequence w of length n➔
➔
➔

➔

➔

Embedding Markov transition (forward)➔◆

EMB(wi): word ⟶ vector in Rd

Training objective (DDPM) - stableMSEµθ(xt,t): predicted mean of pθ(xt−1 | xt)

reverse process

Rounding Markov transition (reverse)

New training objective12

Reducing Rounding Errors➔

Rounding function➔

x0 does not commit to a single word embedding◆ Lsimple(x0)❌ model the structure of x0 well (constraint: t ⟶ 0)
◆ Solution: reparametrize Lsimple(x0)⟶➔➔Clamping during decoding◆ Reparametrize reverse generation process

forced to predict x0 in every term - x0 centered at asingle word embedding⟶Map fθ(xt,t) (estimate of x0) to nearest word embedding - fθ(xt,t) centered at a single word embedding for every step➔fθ(xt,t) predictions more accurate + rounding errors reduced

13

Controlling text generation➔

Control x0:T (continuous latent variables)➔Gradient update on xt−1 (time step t)
parametrized by NNclassiﬁer
parametrized by Diﬀusion-LM➔

Classiﬁer trained on latent variables➔

Fluency regularization◆ generate ﬂuent text
◆ stochastic➔

λ - hyperparameter
ﬂuencycontrolMultiple gradient steps◆ 3 steps of Adagrad update per diﬀusion step◆ ↑ computation - downsample diﬀusion steps (2000 ⟶ 200) - ↑ controlled generation speed
14

Controlling text generation➔Minimum Bayes Risk (MBR) decoding◆ single high-quality output◆ machine translation, sentence inﬁlling➔aggregate samples S from Diﬀusion-LM

negative BLEU score
Sample with minimum
expected risk under L

15

Experiments

Diffusion-LM

16

Experiments➔➔

Trained on 2 datasets
DatasetSize

Composition

E2E

50K restaurant reviews

8 ﬁelds including food type, price, and customer rating

ROCStories

98K ﬁve-sentence stories

causal and temporal commonsense relations between daily eventsDiﬀusion-LM model◆ Transformer (80M parameters)◆ sequence length n = 64, diﬀusion steps T = 2000, square-root noise schedule◆ embedding dimension: d = 16 for E2E & d = 128 for ROCStories◆ decoding time diﬀusion steps T = 200 for E2E & T = 2000 for ROCStories

17

Experiments - Control Tasks➔

6 control tasks (4 classiﬁer-guided + 2 classiﬁer-free)◆ sample 200 control targets from val - 50 samples for each control target➔Fluency: generated text ⟶ teacher LM (GPT-2)◆ perplexity of generated text under the teacher LM - metric: lm-score (lower = better sample quality)18

Experiments - Control Tasks
Control Task

Evaluation Method

Success Metric

Semantic Content

Given: ﬁeld & value
Task: generate a sentence where ﬁeld=value

exact match of value

Parts-of-speech

Given: sequence of POS tags
Task: generate sentence of same length with matching POS tags

word-level exact match

Syntax Tree

Given: syntactic parse tree
Task: generate text with same syntactic parse

F1 scores from oﬀ-the-shelf parser

Syntax Spans

Given: (span, syntactic category) pair
Task: generate text with same syntactic parse tree over span

fraction of spans that match exactly

LengthGiven: length
Task: generate a sequence of ±2 length

fraction of correct lengths

Inﬁlling

Given: left context (O1) & right context (O2) from aNLG dataset
Task: sentence that logically connects O1 and O2

automatic and human evaluation
19

Experiments - Baselines
Baseline

Description

Control Tasks

PPLM

Plug-and-play autoregressive LM trained from scratch on GPT-2
No positional information

Semantic Content

FUDGE

Plug-and-play autoregressive LM trained from scratch on GPT-2

Semantic Content, Parts-of-speech,
Syntax Tree, Syntax Spans, Length

FT

Oracle conditional LM (ﬁne-tuned GPT-2)
FT-sample (sampling) & FT-search (beam search)

Semantic Content, Parts-of-speech,
Syntax Tree, Syntax Spans, Length

DELOREAN

autoregressive LM
left-to-right ﬂuency

Inﬁlling

COLD

energy-based model
left-to-right & right-to-left ﬂuency + coherenceInﬁlling

AR-inﬁllingtrain autoregressive LM from scratch with ROCStories
preprocess: (O1,Omiddle,O2) to (O1,O2,Omiddle)

Inﬁlling20

Results

Diffusion-LM

21

Main Results➔

Diﬀusion-LM on E2E and ROCStories◆ negative log-likelihood (NLL) - lower is better◆ underperforms autoregressive models◆ ↑ model & dataset size - better➔

controllable generation is better for Diﬀusion-LM

22

Controllable Text Generation Results➔

Diﬀusion-LM outperforms baselines◆ Non-autoregressive nature = good for future planning (spans, length) & global structures (tree, POS)
◆ coarse-to-ﬁne representations = control on entire sequence (t ⟶ T) & individual tokens (t ⟶ 0)23

Qualitative Results➔

Syntax Tree
◆ Diﬀusion-LM & FT do well (1m + ctrl), FUDGE deviates◆ Diﬀusion-LM > FT: correct for a failed span - no errors in suﬃx spans24Composition of Controls➔

Plug-and-play controllable generation = modular◆ generate from the intersection of multiple controls◆ Composition results: Diﬀusion-LM > FUDGE & FT

25

Infilling Results➔

Diﬀusion-LM outperforms baselines (COLD & DELOREAN)◆ comparable to AR-inﬁlling◆ ↑ automatic evaluation, ~ human evaluation

26

Ablation Studies

Learned v.s. Random Embeddings

Objective Parametrization (x0 vs noise term 𝜖)Noise scheduleArchitecture27

Conclusion

Diffusion-LM28

Conclusion➔

Diﬀusion-LM◆ novel & controllable LM◆ continuous diﬀusion + non-autoregressive◆ complex ﬁne-grained control tasks➔

Success in 6 control tasks◆ doubles success rate of baselines◆ comparable to dedicated ﬁne-tuning methods➔

Limitations◆ higher perplexity◆ decoding is substantially slower (7x slower than autoregressive LMs)
◆ training converges more slowly
29

Questions

Diffusion-LM30