Editing + Diffusion for Text
Generation
Machel Reid

Who am I?
- Currently RS at Google Brain in London- Working on multilinguality research + edit-models research

How do we create content?

Writing papers

Writing code

Contrast this with current language models…
Autoregressive generation

They have good performance and work for many things….

….but they’re very counter intuitive

With current autoregressive LMs- We cannot revise text in an intuitive way.

- Iterative refinement is hard.

- This is an important task for that is simple for humans to perform butextremely difficult for models

Given this, I will go over some my work on editing- Application-centric editing for text style transfer in “LEWIS: Levenshtein
Editing for Unsupervised Text Style Transfer” ACL Findings 2021

- Learning to Model Editing Processes, Findings of EMNLP 2022

- New preprint! DiffusER: Discrete Diffusion via Edit-based Reconstruction

Applying editing for style transfer

LEWIS: Levenshtein Editing for
Unsupervised Text Style Transfer
Joint work with Victor Zhong (UW)Text Style Transfer: Motivation & Problem Deﬁnition
Negative to Positive:I had a terrible time… → I had a great time…

Positive to Negative:
The worst ribs I’ve ever had! → Probably the best ribs ever!

Many current style transfer approaches require fully
regenerating large portions of the original sentence

These sentences have a large text overlap, so editing
could be a good idea

The text style transfer task
POSITIVE: I had a really great time at the theater, they attended to all of myneeds.
↓NEGATIVE: I had a really terrible time at the theater, they ignored all my requests.

Benefits of editing- Efficient- Allows for content preservation + fluency preservation- More precise control over the sequence transduction process

Levenshtein Editing
Predict Levenshtein operations {<ins>, <keep>, <repl>, <del>} andgenerate for <ins> and <repl> operations

Levenshtein Editing (cont.d)
Negative to Positive:
I had a terrible time… → I had a great time…

Positive to Negative:
The worst ribs I’ve ever had! → Probably the best ribs ever!

LEWIS
2 Steps:1.

Synthesizing pseudo-paralleldata

2.

Learn a Levenshtein editing
model with a coarse-grained
editor and ﬁne-grained
generator to modify style oftext

Synthetic data generation-Use classiﬁer attention to replace style-speciﬁc text with the SLOT token
Fill that text with both style-speciﬁc LMs to create pseudo-parallel dataResults- LEWIS improves over previous work on human and automatic evaluation!
- With our synthetic parallel data, editing based methods work much better...

However, what if we can make editing notapplication-based, but more general?

Learning to Model Editing
Processes
Joint with with Graham Neubig (CMU)

Wikipedia Edits

Why do we want to model it?
- Humans generate content iteratively (not in one pass -> GPT-style)Why do we want to model it?
- Humans generate content iteratively (not in one pass -> GPT-style)Why do we want to model it?
- Humans generate content iteratively (not in one pass -> GPT-style)- Are there patterns when editing processes?

Problem Definition- We want to model the likelihood of the current document by way of an entire
sequence of document edits

Problem Definition: n-order editing- We want to model the likelihood of the current document by way of an entire
sequence of document edits, but we can simplify this to a Markov process(which is single step editing; Reid and Zhong., 2021)- However, when modeling edit processes (the aim of this work), we look toinclude the context of previous revisions (controlled by n).

Problem Definition: Edit Operations- Practically, we use edit operations (ei) (INSERT, DELETE, KEEP, REPLACE) toedit this and make this process more efficient:Problem Definition: Edit Log Likelihood- We can then use this formulation to define edit log-likelihood (which we useto train our model)

Problem Definition: Decomposed Log Likelihood- We can also decompose edit log likelihood into theoperation prediction:- And operation-conditioned generation

Model: EditProModel: EditPro

Components- Edit Encoder- Edit Operation Prediction- Generating Replacements and Insertions- Encoding Edit History

Generating Replacements and Insertions- E.g. we take a mean pool of replaced tokens and sum them with a REPLACE
embedding and use that to initialize a decoder for that span

Edit-compressed history- We use previous edit operations to to compress previous edit history into
their separate spans of edits- (Hard to explain here, so please refer to the paper!)Data

WikiRevisions & CodeRevisions- We propose the datasets with full document-level edit history for bothnatural language (WikiRevisions from Wikipedia) and code (CodeRevisionsfrom Github)Evaluation Metrics

Evaluation Metrics- Edit Perplexity (ePPL), exponent of the NLL for both edits and generated
outputs, normalized by length of both outputsEvaluation Metrics- Edit Perplexity (ePPL), exponent of the NLL for both edits and generated
outputs, normalized by length of both outputs- Operation Perplexity (oPPL), exponentiated NLL of operation prediction

Evaluation Metrics- Edit Perplexity (ePPL), exponent of the NLL for both edits and generated
outputs, normalized by length of both outputs- Operation Perplexity (oPPL), exponentiated NLL of operation prediction- Generation Perplexity (gPPL), exponentiated NLL of generating replaced
or inserted spans (when compared with ground truth edit seq)

Experiments & ResultsTasks- Edit Modeling- Edit Classification- Conditional Editing- Edit-conditioned Generation

Edit Modeling
Extra order edit modeling helps!

Edit Modeling
Extra order edit modeling helps! Knowing where text came from helps predict
future iterations.

Downstream Tasks
Same findings hold, even for discriminative edit-based tasks

Example from sampling from a edit model
Not super fluent, but is likely to be
an artefact of:- Scale (small, undertrainedmodel)- Data: Wikipedia is acomparative cacophony toother forms of creation

But can we make this notion more general?

Introducing text + edit-based diffusion
models!

go/diffuser

DiffusER: Diffusion via Edit-based
Reconstruction
Joint work with Vincent Hellendoorn, Graham Neubig @CMU

go/diffuser

Setupgo/diffuser

Most text generation models
Left-to-right
Pros:-Simple and effective setup

Cons:-Hard to refine
Not much flexibility whengeneratinggo/diffuser

Non-autoregressive models
E.g. CMLM/MLMs
Pros:- Simple- Effective- FastCons:
- Arguably even less flexibility
than AR models

go/diffuser

Text diffusion models

go/diffuser

Diffusion models have two components1)

Corruption (or forward process)E.g. in images
Full image → noise2)

Denoising (or backward process)
Generative Modeling
Noise → Full image[2006.11239] Denoising Diffusion Probabilistic Models

go/diffuser

Issues with diffusion models for text
- Unlike images, score-based generative modeling is not straightforward as
there is no clear method on how to formulate diffusion for categorical
distributions- The corruption process for text-based models and hence the denoisingprocess is also not straightforwardgo/diffuser

Previous work
Structured Denoising Diffusion Models in Discrete State-Spaces (i.e. using theBERT/CMLM objective in multiple steps)

go/diffuser

Previous work cont.d
Step-unrolled Denoising Autoencoders for Text Generation; SUNDAE (instead ofusing masks, replace iteratively using random text)

go/diffuser

SUNDAE cont.d- Pretty good MT performance- However, cannot make use offlexible edit-operators; paradigm
relatively inflexible compared tothe ideal for edits

go/diffuserOurs

go/diffuser

Issues with previous work
The main one there are too many restrictions placed on what is diffusion/whatconsists of diffusion etc…
1) E.g. for the MLM diffusion models, the model doesn’t actually learn tocorrect incorrect text -> just learns to fill masks

2) For the SUNDAE model, it overcomes the first limitation, however it is quite
restrictive in terms of the types of edits it can perform (essentially onlyreplacement).

go/diffuser

We aim to fix this by:- We use the SUNDAE style of using randomly sampled text rather than
<MASK>s (tackling problem 1)- We also include autoregressive generation in this process (though arguably
there could be a purely non autoregressive formulation of this) (this allows
us to have compatibility with AR models)- We use Levenshtein edit operations (i.e. KEEP, DELETE, REPLACE, INSERT) tomake the editing both more controllable and flexible.

go/diffuser

Our corruption process is flexible + DELETE/INSERT are new1) Extremely flexible, uses edit operations
Instead of simply replacing tokens we can perform the following 4 operations:- KEEP (i.e. do nothing)- REPLACE (i.e. replace a span of words with another random span of words —not length constraint)- DELETE (i.e. delete a set of tokens, this means that they would have to beinserted in the next timestep)- INSERT (i.e. randomly insert a set of tokens, this means that they would haveto be deleted)go/diffuser

Edit-based Generation || Our corruption process

go/diffuser

Our denoising process
Two step process:1) Tagger -> we tag a corrupted sentence with the appropriate tags (i.e.
replace, insert, delete, etc)

go/diffuserOur denoising process
Two step process:1) Tagger -> we tag a corrupted sentence with the appropriate tags (i.e.
replace, insert, delete, etc) -> similar to LEWIS

2) Generator: after tagging and summing tag embeddings and wordembeddings, we generate using an autoregressive generator similar to CM3

go/diffuser

Our denoising process
Two step process:1) Tagger -> we tag a corrupted sentence with the appropriate tags (i.e.
replace, insert, delete, etc) -> similar to LEWIS

2) Generator: after tagging and summing tag embeddings and wordembeddings, we generate using an autoregressive generator similar to CM3
I have a <repl:0> dog </repl:0> his name is <insert:0> 
 <repl:0> great 
<insert:0> Jonathan 


go/diffuser

Our denoising process
Two step process:1) Tagger -> we tag a corrupted sentence with the appropriate tags (i.e.
replace, insert, delete, etc) -> similar to LEWIS

2) Generator: after tagging and summing tag embeddings and wordembeddings, we generate using an autoregressive generator similar to CM3
I have a <repl:0> dog </repl:0> his name is <insert:0> 
 <repl:0> great dog 
<insert:0> Jonathan 

I have a great dog his name is Jonathan

go/diffuser

Generation process- We perform 2d beam search, searching over 2 dimensions- Sequence level dimension (as standard)- Revision level dimension- We can keep refining indefinitely, however we find 8-12 refinements
work well.

go/diffuser

Generation process- We perform 2d beam search, searching over 2 dimensions- Sequence level dimension (as standard)- Revision level dimension- We can keep refining indefinitely, however we find 8-12 refinements
work well.
Sequence-level dimension

go/diffuser

Generation process- We perform 2d beam search, searching over 2 dimensions- Sequence level dimension (as standard)- Revision level dimension- We can keep refining indefinitely, however we find 8-12 refinements
work well.
Sequence-level dimension

Revisiondim

Decoder Initialization Techniques
Instead of initializing with continuous representations, we can actually initializeour decoder with discrete sequences

go/diffuser

Quantitative Results
With no distilled data,
performs almost as
well as standard ARfor the first time!

Without task-specifictechniques, works
well with styletransfer

Example Generation

Ablations + Insights

Takeaways

Takeaways- Text editing has a lot of promise and has been shown to be performant incertain situations but there is still a ways to go (LEWIS)- A large drawback has been lack of data for style edits but withdiffusion-inspired models we may be able to get there…
- But we have shown that we can incorporate this editing ability withoutcompromising performance significantly and sometimes improving it!

Future Ideas

Improving Data Quality of Edits- Issues with Wikipedia include:-Conflicting views
SpamBots- Could we get golden Overleaf/Google Docs data?

Classifier-guided DiffusER- One large issue: the corruptions in DiffusER are random and are largely
conditioned on the task objective (i.e. machine translation, summarization)- But can we use classifiers to induce different types of edits/paths?

Using DiffusER style models for data augmentation- Given a seed sequence you can sample iteratively to form different
perturbations of the same sequence.

Future ideas- Have a large scale pre-trained self-editing model- Ideally everyone should be using edit models!
- We need better data (e.g. Google Docs), where contributors are working
towards a somewhat agreed goal to train better models- Are there task-specific diffusion formulations that we could learn tocombine?
- Ensembling large LMs/humans in the discrete space via iterative refinement(nice for API users; PEER paper does a great job in this direction!)

Thank you!

Q&A

Twitter: @machelreid
Email machelreid@google.com