DiffusER: Diffusion via Edit-based Reconstruction
Authors: Machel Reid
Vincent Josua Hellendoorn
Graham Neubig
Venue: ICLR 2023 poster (2023)
Keywords: text generation
editing
denoising autoencoders

Abstract: In text generation, models that generate text from scratch one token at a time are currently the dominant paradigm. Despite being performant, these models lack the ability to revise existing text, which limits their usability in many practical scenarios. We look to address this, with DiffusER (Diffusion via Edit-based Reconstruction), a new edit-based generative model for text based on denoising diffusion models -- a class of models that use a Markov chain of denoising steps to incrementally generate data. DiffusER is not only a strong generative model in general, rivalling autoregressive models on several tasks spanning machine translation, summarization, and style transfer; it can also perform other varieties of generation that standard autoregressive models are not well-suited for. For instance, we demonstrate that DiffusER makes it possible for a user to condition generation on a prototype, or an incomplete sequence, and continue revising based on previous edit steps.

Published as a conference paper at ICLR 2023
DIFFUSER:
DIFFUSION VIA EDIT-BASED RECON-
STRUCTION
Machel Reid
Vincent J. Hellendoorn
Google Research" mache 1ee re
Software and Societal Systems Department Carnegie Mellon University
vhe lendoo cmu, edu
Graham Neubig
Language Technologies Institute,
Camegie Mellon University
Inspired Cogmition gneubi gêcs cmu edu
ABSTRACT
In text generation, models that generate text from scratch one token at time are currently the dominant paradigm. Despite being performant, these models lack the ability to revise existing text, which limits their usability in many practical scenarios. We look to address this, with DIFFUSER (Diffusion via Edit- based Reconstruction), new edit-based generative model for text based on denoising diffusion models class of models that use Markov chain of denoising steps incrementally generate data. DIFFUSER not only strong generative model general, rivalling autoregressive models several tasks spanning machine trans- lation, summarization, and style transfer; it can also perform other varieties generation that standard autoregressive models are not well-suited for. For in- stance, we demonstrate that DIFFUSER makes possible for a user to condition generation prototype, an incomplete sequence, and continue revising based on previous edit steps.
INTRODUCTION
Revision and editing central how humans produce content; write and revise emails and papers, gradually produce works of art, and iterate on plans for a project. Despite this, the most dominant paradigm in text generation is purely autoregressive, producing text left-to-right in sin- gle pass (Bengio et al., 2003). Although models employing this single-pass form of generation are highly performant, they are limited by the inability refine existing text. To address this, we pro- pose DIFFUSER: Diffusion via Edit-based Reconstruction, a flexible method to apply edit-based generative processes arbitrary text generation tasks. Specifically, we take inspiration from dif- fusion models (Sohl-Dickstein et al.) 2015) Ho et al.) (2020), generative models that generate by way of incremental denoising steps, and adapt this approach to text generation paradigm with a formulation similar to natural editing processes.
Prior work on text generation either focuses on improving the performance of standard autoregres- sive (AR) models through larger models and datasets (Vaswani et al. 2017 Sutskever et al.) 2014
Radford et al.) Brown et al. 2020) or on proposing new, non-autoregressive approaches (Gu et al, Ghaz7iinnnnnnggeie
al. 2019 Gu et al., 2019) improve general modes of text generation. A
thus far separate line of models has taken the perspective of modeling text edits for specific tasks:
e.g. style transfer (Reid & Zhong 2021 Malmi al. 2020), sentence fusion (Malmi et al. 2019),
and grammatical error correction DDae Kilgarriff) 2011). DIFFUSER unifies these two perspec- tives by enabling edit processes be applied to general purpose text generation without compro- mising performance or requiring external supervised data (Guu et al. 2018). This design enables it Work done partially while the University of Tokyo

---

Published as a conference paper at ICLR 2023
Sitr
Toranto
guary
trough
teel
po(oy (lay)
d<eela x)
Reconstruction Process
These
model
guaky
nomed
DER
Corruption Process
polay-alez
Stowes
model
named
DiFLaEER
wdkling
proceckines
of ity-s)
model
namedd
usid
edting
procosscs
(ly-))
d ayy-)
nexible
generation
Figure 1: DIFFUSER'S s text generation process. Orange represents replacements, blue represents insertions, red represents deletions, and white represents keep operations. This process largely imi- tates natural editing process (Reid & Neubig 2022).
to both generate and edit text, including externlly produced content, natural extension of the text generation paradigm.
DIFFUSER models text generation generation allows us to develop a synthetic formulation of natural editing processes (Reid & Neubig, 2022) using edit-based corruption and reconstruction. Our method starts from arbitrary sequence (either prototype generation, randomly sampled tokens, or null sequence) and progressively edits into the final sequence guided by the Levenshtein edit operations of INSERT, DELETE, KEEP, and REP LA as shown in Figure This enables flexible editing in range of contexts, including machine translation, summarization, style transfer, while also allowing for the possibility of taking outside input to guide and constrain generation.
series of diffusion steps the token level. This form of
Learning these edit-based diffusion processes required several innovations over standard autore- gressive and MLM-style iterative generation approaches (Ghazvininejad al. 2019 Austin et al.) 2021 Savinov et al. 2022), including forming edit-based corruption and reconstruction processes for training (Sec 3). as well as techniques to improve the quality of decoding sequences across both timesteps and token-leva generations (including 2D beam search; Sec 3.6 Sec 3.5).
To demonstrate the effectiveness of DIFFUSER, test method on three text generation tasks: machine translation, abstractive summarization, and text style transfer, and show on-par or improved performance compared purely autoregressive, single-pass and non-autoregressive methods. We also provide qualitative samples of the edit processes learned by the models in different settings and analyses on training and inference speeds, well the relationship between edit steps and performance.
Overall, we demonstrate the potential of edit-based generative models offer more performant generation, 2) greater interactivity between different models (as we can now perform edits in the discrete space on model generated output), and 3) more flexible/cottrollable generation.
BACKGROUND
DIFFUSER operates at the intersection of text generation, editing processes, and diffusion models. We first provide the background and intuition of these three techniques.
2.1 TEXT GENERATION
Most text generation models used in NLP today are autoregressive in nature. In this paradigm,
given sequence [su, Si,..., sy one model the likelihood of the entire sequence P(s) by modeling the probability of predicting each token in an autoregressive, often left-to-right, manner. This formulation, where the likelihood of token p(s:) is conditioned its predecessors Sca is
2

---

Published as a conference paper at ICLR 2023
shown below (Bengio al. 2003
N II E=U
P(s)
Models trained with this objective can then be sampled from, or searched over (e.g. using beam search), provide generations in downstream tasks such as machine translation summarization.
Non-autoregressivee models (Gu et al. 2017) are a different variety of generative models, in which sequence is generated in a single pass (removing the autoregressive conditioning on previously generated tokens) with multiple revision-level passes, often in the name of efficiency.
2.2 EDITING PROCESSES
Editing processes (Reid & Neubig 2022) are paradigm for modeling text by way of incre- mental revisions, taking inspiration from the the way humans generate text. Specifically, let X (xo, XI... x be a series of R versions of document, where Xo, x,, x represents the initial, intermediate (at timestep !), and final/current state of document, respectively. Using editing processes, we can model the probability of this series of documents versions occurring consecutively as follows:
x(X)=
[(xx1xx011) 1
2
With this formulation, editing processes can also be used to calculate the probability of only the final document while taking into account previous revisions, which is not possible in the traditional text generation setup as intermediate revisions are not explicitly known, using the equation below (Reid
& Neubig 2022.
p(xR)=
E
x(8).
(3)
(E &n kn=xn)
2.3 DIFFUSION MODELS
We now make the connection between editing processes and diffusion models (Sohl-Dickstein et al. 2015 Ho et al. 2020). Continuous diffusion processes are commonly applied in computer vision tasks to iteratively convert sample of noise into an image. This can be seen as an edit process in which the model iteratively edits noisy image to bring closer to a final, complete image. These continuous diffusion models are often trained by modeling Markov chain Xy .......... .x...XX where Xo represents the original image and XT represents Gaussian noise. This chain typically produced by incrementally adding Gaussian noise to x, to form Xx+1 (known as the forward or corruption process), wherein a model parameterised by Py is trained to reverse (or "denoise") this process to form the chain Σ Pa (x))
Analogized text, this allows us formulate natural edit processes as a discrete diffusion process in which a null string or prototype is iteratively edited into free form text. Our DIFFUSER method (Figure [T takes inspiration from this process, but parameterises the corruption process by way of sampled discrete edit operations applied over a discrete sequence of tokens. The success of our method supports the findings in the vision domain Bansal al. (2022), where it is found that diffusion models can learn to invert arbirtary transformations.
Previous work in diffusion models has largely focused on computer vision (Ho al.. 2020 Austin et al., 2021), in which the diffusion process is applied to raw image values. Within the context of natural language, both discrete diffusion models using only replacement operations (either applied to random tokens masked tokens) (Savinov et al. 2022 Austin et al., (021). and continuous diffusion over word embeddings al, 2022 have been proposed. Our model is a more flexible approach, using all four edit operations, towards diffusion models when compared with this work owing to its edit process formulation, and is also more compatible with current models (e.g. AR bootstrapping).
3

---

Published as a conference paper at ICLR 2023
3 DIFFUSER
DIFFUSER, being diffusion- method, has two main procedures: corruption and denoising. Unlike previous work (Ghazvininejad et al. 2019) Savinov et al., 2022 Gu et al. 2019) in which this procedure is relatively inflexible (e.g., due length restrictions and/or using continuous repre- sentations for the basis of the diffusion process). both our corruption process and denoising process are based on Levenshtein operations, allowing our model learn to take advantage of the flexibility of text editing when generating.
3.1 EDIT OPERATIONS
Given the central role of the Levenshtein edit operations our models, we provide brief overview of each operation and its role in the editing process. We Figure 1 as a guide when explaining each operation.
INSERT: The insertion operation is used to add new text sequence. For example in Figure "uses editing processes" added by DiffusER at timestep Tr-2-
DELETE: The deletion operation erases existing text. In Figure 1 this is shown when "These" gets deleted at timestep 2 + 2T-3.
REPLACE: The replacement operation works overwriting existing text with new text. This is shown Figure step XT - IT-1 where "filter Toronto guilty trough feel" replaced by "These model guilty named DiffusER""
KEEP: The keep operation ensures that a portion of the remains unchanged into the next itera- tion. This is illustrated in timestep 4T-2 + 2T-3 where "model named DiffusER" is kept.
3.2 EDIT-BASED CORRUPTION
The four Levenshtein edit operations described above allow us to transform any arbitrary sequence of tokens into another. This is in contrast iterative mask replacement, which can only introduce new tokens (Ghazvininejad et al. 2019 Austin et al. 202T] Savinov et al. (022). For every timestep corruption process q( x 1: & Elj is parameterized by two distributions: the distribution over edit types & (c.g. 60% keep, 20% replace, 10% delete, 10% insert), and the distribution over edit length &,. The latter can be parameterized by any distribution over non-negative integers, such as uniform distribution or Poisson distribution. For instance, to learn deletion operation in the reconstruction process, we insert randomly sampled distractor tokens, whereas, to learn an insertion operation we delete subset of tokens contained in the sequence.
3.3 EDIT-BASED RECONSTRUCTION
Our generative process is trained via the Edit-based Reconstruction (ER) process. ER can be thought of as the opposite of our corruption process, which we need find the appropriate edit operations to transform Xy to Xo, by way of x,-1,...,X!
That is, given a corrupted sequence Xy. we aim learn the process by which we can reverse the corruption in the following form.
T II t=
Px(xo)
(4)
Given that, model the likelihood of each timestep x,, this also be referred to edit process (Reid & Neubig 2022). As we include an edit process in our model and use Levenshtein can think of ER as two distinct steps: identify which edits should take place
tags for editing, (tagging process) and deciding which tokens should in these positions (generative process). This decomposition is shown here:
po (x--1 xx)
Po (ee (xe(xx))1g) ( (x, (xt-1 (x)
188
(5)
4

---

Published as a conference paper at ICLR 2023
thE parameterises the tagging model to estimate the likelihood of producing given set of where Ps Levenshtein edit operations (INSERT DF LETRR KERP REPLACE given x,, and y parame- tersies the generator model given sequence x; and edit operations e,. This decomposition via edit- operations allows the generation process to be more controllable and more flexible as it allows up to explicitly specify edit types associated with tokens to be edited, rather than leaving both processes to be implicit.
3.4 IMPLEMENTING DIFFUSER WITH TRANSFORMERS
When implemented with Transformers (Vaswani et al. 2017), DIFFUSER consists of two compo- nents: a tagger and generator. The tagger, a transformer network, is trained using cross-entropy loss over the ground-truth tag types to predict the edit operations that should be applied to the sequence, in preparation for the next generation step. Then, in the generation step, after removing tokens se- lected for deletion, we and replaced sequences autoregressively. Following this, we feed the output of this diffusion step into the tagger and perform another diffusion step. One step of this process be compared the reconstruction process used in Aghajanyan et al. (0022)
learned embedding to insert and replace types and generate the inserted
3.5 DECODING METHODS
DIFFUSER has an inherently different generation process from a standard autoregressive language generation model-in in addition operating sequence/token level (in which generation is com- posed of generating individual tokens in a single-revision; intra-revision), we also operate on a revision level (in which the text is expanded across diffusion steps, inter-revision). This allows us to experiment with different methods for decoding on both the intra-revision (single sequence level) and inter-revision levels (multiple version level), which explain below.
Beam Search One method for decoding is to perform beam search over b hypotheses at every step on the output of our autoregressive generator (intra-revision level), while performing greedy decoding the inter-revision level. Although being conceptually straightforward, this method has the limitation not searching over the inter-revision space (despite revisions being key component of our approach).
2D Beam Search We propose 2D beam search, in which we extend beam search as it is applied to token-level autoregressive generative models, and perform beam search using both an intra-revision width of b and an inter-revision beam width of This allows us to perform search on the inter- revision level, which we find results better downstream performance, but increases the beam count to b beams. Assuming fixed sequence length and maximum number of diffusion steps, we would decode as follows: We first use beam search with width at the token level and take the most likely candidates (measured with log-likelihood). These r candidates step of the diffusion model, wherein for each of r hypotheses the next diffusion step is performed with the token-level generator decoding with beam width of b. This leads to have X candidate hypotheses, of which we take the top r. This process repeats for each diffusion step thereafter.
then fed to the next
Nucleus Sampling To improve the diversity of generations, we also consider nucleus sampling based approach, where at every timestep X,, p = 0.6 to sample each token autoregressively at the intra-revision level, and greedily decode at the inter-revision level (i.e. no search sampling is performed over multiple diffusion steps).
nucleus sampling (Holtzman al. 2019) with
3.6 DECODER INITIALIZATION TECHNIQUES
Since our model is based edit processes, it offers flexibility in terms of the discrete sequence from which to initialize the text generation. Previous work on non-autoregressive translation often starts with [MASK] tokens (Ghazvininejad et al. 2019). null string (Gu et al. 2019) random tokens (Savinov et al. 2022). We include the latter two methods in our experiments, in addition to (1) experimenting with an AR Bootstrap, in which learn to bootstrap from text generated by purely autoregressive model, and (2) proposing to use the source-side text as an initial state for the DIFFUSER decoder.
5

---

Published as a conference paper at ICLR 2023
Ner ser)
Those modi
galby priuuc
Theae
model
gviry narned Diiiaii
Hill Sosuncca
Sasson Tokana
Reahitse
Figure 2: Figure illustrating bootstrapping methods for decoding.
Null Sequence In this setting, we simply initialize DIFFUSER with null string, in which the first edit is constrained to insertion.
Random Tokens In this setting, we initialize DIFFUSER with a series of random tokens, following
(Savinov et al. 2022). The model then learns edit this random sequence.
AR Bootstrap We bootstrap the reverse diffusion process by taking the output of DIFFUSER con- strained to generate autoregressively (essentially mimicking a standard autoregressive generator). We then use DIFFUSER to further edit the output of this operation.
Source Bootstrap In sruuence-to-sqquene setting, the source text, by setting Xy to be equivalent to s. As we show later sections, this is particularly useful in tasks such summarization in which the output be easily formulated as an editing version of the input.
also generate by bootstrapping using
4 EXPERIMENTS
4.1 MODELS
DIFFUSER We instantiate DIFFUSER with two separate Transformer models for the tagger and generator. We use the Transformer-base encoder-decoder (Vaswani al. 2017) architecture, with 6 layers, for the a hidden dimension of 512, feedforward dimension of 2048, 8 attention heads, and dropout p = 0.3.
Baselines (MT & Summ) We several Transformer baselines from previous literature for various tasks. We include a conventional 6-layer encoder-decoder Transformer model from Vaswani al. (0077) as well as models proposed in related work from the non-autoregressive generation literature: Levensthein Transformer (Gu et al. 2019), CMLM (Ghazvininejad et al. 2019), DisCo Kasai et al. 2020a Imputer (Saharia et al. 2020). and SUNDÃE (Savinov et al. 2022).
4.2 TASKS
Machine Translation We use the WMT' English-German dataset for our machine trans- lation experiments. We use the same preprocessing and post-processing steps as Ghazvinine- jadd et al. (2019). Unlike the standard in non-autoregressive translation work (Zhou al. 2019). we focus using the gold machine translation data instead of distilled data. We use a Poisson distribution E((} = 3) over edit operation lengths in our corruption process. Note that compute the edit operations the following ones, we use 12 diffusion steps, b = 5, and r = 3 for beam search, and (60% KEEP, 20% REPLACR, 10% INSERT, 10% DELETE) based on numbers from preliminary experiments.
words rather than tokens. For this task, as well as
Summarization We also benchmark on the CNN//ailyMaiil dataset for summarization NNllapati et al. 2016). Summarization is different in nature from machine translation in that it can be described as more conducive to edits as a good summary tends to preserve many parts of the input. We use the same post-procsssing steps as See et al. (2017). We use Poisson distribution ( 8) over edit operation lengths in our corruption process (to roughly model sentence boundaries).
Text Style Transfer We perform experiments using the Yelp (Shen et al. 2017) dataset for the un- -ng-end-Geeeraee
supervised text-style transfer task. We compare against methods such as Tag- Tag-and- Generate (Madaan ct al. 2020), Masker (Malmi et al. 2020),, and LEWIS (Reid & Zhong 2021). In contrast with ma- chine translation and summarization, text style transfer datasets often unaligned (i.e. without

---

Published as a conference paper at ICLR 2023
Model
En-De (MT)
CNN-DM (Summ)
AR Transformer (Vaswani al.
2017
27.3
36.8
SUNDAE (Savinov et al.
2022
26.3
37.0
CMLM Ghazvinneeja
2019
24.6
Levenshtein TransTormer"
Gu el al
2019
23.7
DisCo (?)
24.7
Imputer
25.2
DIFFUSER
27.2
37.8
DIFFUSER + AR bootstrap
28.8
38.4
DIFFUSER + source bootstrap
24.5
38.9
Table I: Machine Translation (MT) and Summarization (Summ) results on WMT' 14 En-De (gold) and CNN-Daie CNN-DailyMail. Experiments MT use BLEU while summarization uses ROUGE. DIF- FUSER is compatible with a standard autoregressive model, while outperforming previous methods.
Model
Accuracy
BLEU
Masker (Malmi et 2020
40.9
14.5
Tag and Generate (Madaan et al. 2020
86.2
19.8
LEWIS (Reid & Zhong 2021)
93.1
24.0
DIFFUSER
87.6
25.2
Table 2: Results Yelp dataset for text style transfer. Without task-specific training techniques, DIFFUSER performs comparably to previous task-specific methods.
source-target pairs) leading to the prominence of unsupervised text style transfer methods. We propose method of performing unsupervised text style transfer using DIFFUSER, following the synthetic generation method in Reid & Zhong (022). We train two separate, style-specific (e.g. positive and negative) DIFFUSER models on the style-specific data. We then perform transfer at test time, feeding text from each style into the model trained to edit in the opposite style (e.g. positive text + negative DIFFUSER model; negative text - positive DIFFUSER model). Following stan- dard practice, we measure performance with BLEU, Self-BLEU and Accuracy (based on a classifier trained disambiguate between different styles of text; (2011)
the classifier from Reid & ZZhog
4.3 RESULTS
Main Results We summarize our main results on both machine translation and summarization in Table [] As can be seen, for both machine translation and summarization tasks, DIFFUSER, using 12 diffusion steps, outperforms all non-autoregressive baseline and rivals or outperforms the fully autoregressive model. Particularly interesting how the various methods of initializing our model (i.e. AR Bootstrap and Source Bootstrap) can further improve performance well beyond the autoregressive baseline, depending on the task. We can see that for summarization, bootstrapping from the source input is more effective than bootstrapping from an abstractive autoregressive model. However, for both tasks, unlike many -uoeeuvoreeeessvve methods, we show that DIFFUSER complementary with token-level autoregressive methods and can be used naturally in conjunction with them.
Style Transfer Results We also perform unsupervised text style transfer using our DIFFUSER models using the Yelp (Shen al. (017)) dataset. The results can be seen Table 2 We show that even without task-specific techniques (such as synthetic data generation and classifier based style- specific token identification), we still have competitive performance with state of the art methods.
4.4 ANALYSIS
We were not able to reproduce the published results of the Levenshtein Transformer using their code hence reported BI.EU score of 23.7 slightly lower than that of 25.2 reported in Gu al. (0099)

---

Published as a conference paper at ICLR 2023
Source Docu-
(CNNYTboy's ro not gouna take it anymore. Really. Twisted Sister says that its 2016 tour will be its last, according to press release. Next year marks the band's 40th anniversary, and to velebrate, wou being titled "Korty and F**ck It." "It's official: Farewell," Twisted Sister singer Dee Snider posted on Facebook. Snider also noted that the hand will play new drummer, Mike Portnoy Adrenaline Mob. Portnoy replaces A.J. Pero, who died March 20. The will also perform two shows Pero's honer: one at Las Vegas' Hard Rack Hotel and Casino, the other the Starland Rallroom in Sayreville, New Jersey. The latter is in support of Pero's family. Twisted Sister's biggest hit, re Not Gonna Take It," hit the Top Forty 1984 and featured popolar video.
ment
Step
(CNNyThcy're not gonna take anymere. Really. Twisted Sister says that 2016 will be its last, according to press release. Next year marks the band's 40th anniversary, and to celebrate, the tour is being titeed "Forty and F*cc official: Farewell." Twisted Sister singer Suider posted on Facebonk, Snider also
the will play with new drummer, Mike Portnoy of Adrenaline Mob. Portney replaces A.J. Pero, who died March 20. The hand will also perform two shows in Pero's s honor: one at Las Vegas' Hard Rock Hotel and Casino, the other the Starland Ballroom in Sayreville, New Jersey. The latter in support of Pero's family. Twisted Sister's biggest hit, "We're 'e Not Gonna Take It," hit the Top Forry in 1984 and was featured in popolar video.
Step Twisted Sister says that its 2016 tour will be its last, according to press release. Next year marks the band's 40th anniversary, and to celebrate, the tour being titled Forty and P*ck lt." officiilllll Farewell," "wisted Sister singer Dee Smider posted--oeeeeeee on Peeeboek- Seider slse nesed thet the bond setdd-wiil-gee play with drammetr Make Postnoy- soary-----eee Addenatine Mob- Portnoy replaces A.J. Pero, who died March 20. The band will also perform two shows in Pero's is honor: one at Las Vegas' Hard Rock Hotel and Casino, the other at the Sturland Ballroom Sayreville, New Jersey. The latter support of Pero's family. Twisted Sister's biggest hit, "We're 'e Not Gonna Take It," hit the Top Forty in 1984 and was featured in a popular video.
Step Twisted Sister says that its 2016 tour will be its last, according to a press release. Next year marks the band's 40th anniversary, and to celebrate, the lour is being titled "Furty and 1*cck IL" Portnoy replaces A.J. Pero, who died March 20. The hand will nhee perform two shows in Pero's honor mone at tas Veger Hend-RRek--Hoeee Hntel and Ensine the-neher-st the Storlend Balireom--enn Sayroville- New Jessey. The intter is wwwpssppppppppeooeeee Pare's Twristed Sister's biggest hit, "Weise Not- Genns- Take it" hit- the Fort-- in 1984 and-wer feetured in popelar videein Las Vegas and New Jersey.
Step
Twisted Sister says its 2016 tour be its last, according to a press release. Next year marks the band's 40th anniversary, and to celebrate, the tour is being titled "Forty and F*ck It" Portnoy replaces A.J. Pero, who died March 20. The band will perform two shows in Pero's honor in Las Vegns and New Jersey.
Generated Summary
Twisted Sister says that its 2016 tour will be its last. Next year marks the hand's 40th anniversary, and to celehrate, the lour is being utled "Forty Hick AJJ. Pero, died March The band will perform two shows Pero's honor in Las Vegas and New Jersey.
Table 4: Example of our summarization DIFFUSER process on a test set example. Here we show that the majority of the summarization process is deletion coupled with minor edits. Despite this simplicity, we are able to improve over existing purely abstractive models.
We perform additional analyses on DIF- FUSER, specifically focusing on the de- coding method, the number of iterations versus the final BLEU score, and also a qualitative analysis of how text changes at every step.
Initialization
Decoding Method
BLEU
Random Tokens
Greedy
26.3
Random Tokens
Beam b =
26.7
Random Tokens
Beam 6 = 15
26.9
Random Tokens
Nucleus
26.8
Random Tokens
2D-Beam
27.2
Decoding Method Ablation We per- form an ablation of the decoding method, using DIFFUSER for 12 steps (as used in our main results) and showing results when comparing greedy decoding, (ID) beam search, nucleus decoding, and 2D beam search. We show that 2D-beam search tends to perform the best, likely because it searches over multiple diffu- sion steps, while other methods (greedy, beam, nucleus) are still competitive.
Table 3: Decoding method ablation on the MT test set.
Number of Edit Steps versus Performance We perform an analysis where we compare the num- ber of timesteps in our denoising diffusion process and the final BLEU score on WMT' 14 En-De when using 3D-Beam Search and random token initialization in Figure 4 Here it can be seen that most performance gains are in the initial diffusion timesteps (0-10), with diminishing gains (for ma- chine translation) or gradual losses (for summarization) between 10 and 30, after which performance marginally decreases towards 60 steps.
How does text change every step? We include a qualitative sample from our DIFFUSER sum- marization model (Table 4. We find that DIFFUSER learns edit processes intuitive the task at hand: namely largely deleting portions and making minor edits to the remaining text (similar to how human may perform summarization given news article).
8

---

Published as a conference paper at ICLR 2023
27.2
5 Beam
Diffusion steps versus BLBU /OOGE MT; Summaniization
27.0
Bcem D 15
36.8
Nucious
Deam
20.6
23
DimuttR BLEU)
25.4
Greedy
0
20
30
40
30
60
Reletive Spood, Groady
1x
@Trreteess
Figure Relative time (seconds) comparison between decoding methods, measured sin- gle V100 GPU. There is a trade-off between inference and performance. Faster well- performing decoding algorithms for diffusion models are an area for further work.
Number steps versus BLEU/ROUGE
Figure 4:
WMT' En-De and Summarization for both SUNDAE and DIF- FUSER. We observe fast initial progression with performance, leveling off as steps increase.
Time comparsion between decoding methods We also measure the impact of the various decod- ing algorithms used with results shown in Figure Beam search and 2DD-Beam Search performs significantly slower than greedy and nucleus sampling, demonstrating the potential for improved de- coding algorithms tailored for improving the trade-off between efficiency and accuracy in diffusion models.
5 RELATED WORK
Non-Autoregrresive Generation Work in machine translation has explored non/semi- autoregressive generation (Gu et al. 2017 et al., (2088), which often includes an iterative refine- ment step (Lee et al. 2018 Ghazvininejad et al. 2019: Kasai et al. [2020a] Gu et al. 2019). Previous methods in this space are often highly specialized underperform non-autoregressive methods due to the constraints imposed on generation for efficiency. This being said, Kasai et al. (2020b) demon- strated that non-autoregressive models are actually comparable in speed when using larger batch size instead of 1. Our method allows us to hone in on the notion of iterative refinement by way of editing processes, and is also relatively general, allowing to combine DIFFUSER with standard autoregressive models.
Learning Properties of Edits Previous work has also looked at studying exploiting the prop- erties of edits. This was initially worked on in the context of vector representation learning of edits (Yin et al. 2019 Marrese-Taylos Taylor al.) 202T). Concurrentlyy, a line of work has used edits for spe- cific tasks such as sentence fusion, style transfer and grammatical error correction (Malmi et al.) 2019 2020 Reid & Zhongg 2021 Omelianchuk et al.) (2000). Recent work has proposed editing processes (Reid & Neubig, 2022), in which document generation is looked at through the lens of its revision history, rather than just at a token level. We take inspiration from this work and devise a process by which arbitrary text generation tasks can be fitted into this framework.
6 CONCLUSIONS
We proposed DIFFUSER, an diffusion-based generative model for text using edits. DIFFUSER shows improvements across the tasks considered (machine translation, summarization, style trans- fer), with improved generative flexibility via incremental text improvement, and compatibility with standard autoregressive models. We hope that DIFFUSER with spur research edit-based gen- erative models, with further potentials including how we can leverage edits to ensemble models (regardless of parameter count) in the discrete space.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENTS
We thank Armen Aghajanyan, Daniel Fried, Edison Marrese- Taylor, Eric Wallace, and Luke Zettle- moyer for their helpful comments early discussions. We thank Ari Holtzman, Jungo Kasai, Aman Madaan, and Eric Wallace for feedback and proofreading the draft of this paper.
REFERENCES
Armen Aghajanyan, Bernie Huang, Candace Ross, Vladimir Karpukhin, Hu Xu, Naman Goyal, Dmytro Okhonko, Mandar Joshi, Gargi Ghosh, Mike Lewis, and Luke Zeuteeoyer. Cm3: A causal masked multimodal model of the internet, 2022. URL
www.xxii..rggg..... org/abs/
2221.07520
Jacob Austin, Daniel D. Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg. Structured denoising diffusion models in discrete state-spaces. In Marc' Aurelio Ran- zato, Alina Beygelzimer. Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Neural Information Processing Systems 34: Annual Conference on Neu- ral Information Processing Systems 2021, NewrIPS 2021, December 6-14, 2021, virtual, pp. 17981-17993, 2021. URL htttps://proceedd ieg.ueiippnngs..urppp.... www..nuuriic.mm wwwp:ppp.ccppppp/22.22222 aaaaeeeee2212tttsssa-aaaaaaaaaaa
Awwtraatttttaaa..........aa. html
Arpit Bansal, Fitan Borgnia, Hong-Min Chu, Jie S. Li, Hamid Kazemi, Furong Huang, Micah Gold- blum, Jonas Geiping, and Tom Goldstein. Cold diffusion: Inverting arbitrary image transforms without noise, 2022. URL htttps:// https////aaav..eee org/ /a////a///// /20/ 09392
Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Janvin. A neural probabilistic language model. J. Mach. Learn. Res., 3(nul)):1377-1155 mar 2003. ISSN 1532-4435.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam. Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Kerbert-Voss,, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, lla Sutskever, and Dario Amodei. Language models are few-shot learners. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020. URL httpps://proceeeeeeeee cb4 96741 8bfb8ac1 42 42f64a Abst ract. html
www.iis.urippp.... CC/ wwwpppape.... 2020/022000222 2020/hash.
Robert Dale and Adam Kilgarriff. Helping our own: The HOO 2011 pilot shared task. In Proceedings of the 13th European Workshop on Natural Language Generation, pp. 242-249, Nancy, France, September 2011. Association for Computational Linguistics. URI. hntps:
www.nthhloo...rg...oo... 1-
Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer. Mask-predict: Parallel decoding of conditional masked language models, 2019.
Jiatao Gu, James Bradbury, Caiming Xiong, Victor O. K. Li, and Richard Socher. Non- autoregressive neural machine translation. arXiv preprint arXiv:1711.022 2017.
Jiatao Gu, Changhan Wang, and Junbo Zhao. Levenshtein transformer. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' A'AchhéBBuc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 111199-11181 2019. URL httttt:://proceeeee http:///ooceeeenneeeeepep wwweeiinnsneuuiip.... nww.ips...p.... wt/pcc/paaa cc/ppppe// 200 9ww99//aahhhh 6 675 (0220559922222522 wfc.oeoe665ecoaooo 5 fb4 900489004446 66e Abst.
Kelvin Guu, Tatsunori B. Hashimoto, Yonatan Oren, and Percy Liang. Generating Sentences by Editing Prototypes. Transactions of the Association for Computational Linguistics, 6:437-450, 2018. doi: 10. 10.11630 162/tacl a 00030.
10

---

Published as a conference paper at ICLR 2023
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 53:6840-6857 2020.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case neural text degeneration, 2019. URL /tLLattss///x/x..or/////a//////
09751
Jungo Kasai, James Cross, Marjan Ghazvininejad, and Jiatao Gu. Non-autoregressive machine translation with disentangled context transformer. In Proceedings of the 37th International
Conference Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 5144-5155, PMLR, 2020a. URL http:
//proceedings. mlr www.ss.www.......e......eeee.ee. www.ee///w11//aaaa/22.
html
Jungo Kasai, Nikolaos Pappas, Hao Peng. James Cross, and Noah A. Smith. Deep encoder, shallow decoder: Reevaluating non-autoregressive machine translation, 2020b. htRlhhtss://axxit
bogg/bs/2000 2006 10369
Jason Lee, Elman Mansimov, and Kyunghyun Cho. Deterministic non-autoregressive neural se-
quence modeling by iterative refinement, 2018.
Xiang Lisa Li, John Thickstun, Ishaan Guirajani, Percy Liang, and Tatsunori B. Hashimoto. Diffusion-Im improves controllable text generation. arXiv preprint arXiv: Arxiv-2205. 14217, 2022.
Aman Madaan, Amrith Setlur, Tanmay Parekh, Barnabás Póczos, Graham Neubig, Yiming Yang, Ruslan Salakhutdinov, Alan W. Black, and Shrimai Prabhumoye. Politeness transfer: A tag and generate approach. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault (eds.). Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACI. 2020, Online, July 5-10, 2020, pp. 1869-1881. Association for Computational Linguis- tics, 2020. doi: 2020 acl main. 169
1/2020// 169. BtLp:tts: http:///./....1111.........0. 18/55v1333
Eric Malmi, Sebastian Krause, Rothe, Daniil Mirylenka, and Aliaksei Severyn. Encode, tag, realize: High-precision text editing, emalp, 2019. doi: 11855//
Eric Malmi, Aliaksei Severyn, and Sascha Rothe. Unsupervised Text Style Transfer with Padded Masked Language Models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 8671 8671-8680, Online, November 2020. Association for Computational Linguistics. doi: 10. 12.11665/1/2222.ccm http://20..mmmmmmm.mmm.mmmm
Edison Marrese- Taylor, Machel Reid, and Yutaka Matsuo. Variational inference for learning repre- sentations of natural language edits. In AAAI, 2021.
Ramesh Nallapati, Bowen Zhou, Cicero Nogueira dos Santos, Çaglar Gülçehre, and Bing Xiang. mns and beyond. In Yoav Goldberg Abstractive text summarization using sequence-to-seqee Conference on Computational Natural and Stefan Riezler (eds.), Proceedings the 20th SIGNLL Language Learning. CoNLL 2016, Berlin, Germany, August 11-12, 2016. pp. 280 29002990022 ACL, 2016. doi: 10. 10.18663/v11 1lk 1028. URL Ittpps://do1 oro /00. 18653vv 16- 028
Kostiantyn Omelianchuk, Vitaliy Atrasevych, Artem Chernodub, and Oleksandr Skurzhanskyi. Gec- tor grammatiaaia error correction: Tag, not rewrite, 2020.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners.
Machel Reid and Graham Neubig. Learning to model editing processes, 2022.
Machel Reid and Victor Zhong. LEWIS: Levenshtein editing for unsupervised text style transfer. In Findings the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 3932- 3944, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v1/021.
andings-acl.344 peattps:/acclaathote wwwphhhoolloyoo//2221111111 fi htwp////....iiiinnnnnnnnnnnn ngs-alll .34

---

Published as a conference paper at ICLR 2023
Chitwan Saharia, William Chan, Saurabh Saxena, and Mohammad Norouzi. Non- -on-autoregreessvve machine translation with latent aligmments. In Proceedings of the 2020 Conference on Empir- ical Methods in Natural Language Processing (EMNLP), pp. 1098-1108, Online, November 2020. Association for Computational Linguistics. doi: http:s::/accanthhone www.aatwwooo.....o....e..o...... 2020 emnlp-ma pp 83
1/2020.emmlp-mmann... URL
Nikolay Savinov, Junyoung Chung, Mikolaj Binkowski, Erich Elsen, and Aaron van den Oord. Step- unrolled denoising autoencoders for text generation. In International Conference on Learning Representations, 2022. URL https:://pprrreviieee
Abigail See, Peter J. Liu, and Christopher D. Manning. Get to the point: Summarization with ceinter-generatee networks. In Regina Barzilay and Min-Yen Kan (eds.), Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 August 4, Volume I: Long Papers, pp. 1073-1083 Association for Computa- tional Linguisties, 2017. doi: 10.18653/11/117-00999 URL htttps://doi.org/1...000
v1/P17-1099
Tianxiao Shen, Tao Lei, Regina Barzilay, and Tommi Jaakkola. Style transfer from non-parallel text by cross-alignment. arXiv preprint arXiv: 7705 09655, 2017.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics.. In International Conference on Machine Learn- ing. pp. 2256-2265 PMLR, 2015.
Iya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence sequence learning with ral networks. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Wein- berger (eds.), Advances Neural Information Processing Systems, volume 27. Curran Asso- ciates, Inc., 2014. URL https://ppooceeiiss new..ps....e.................... https/ccaapee//////leee. pdf
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. arXiv preprint arXiv: Arxiv- 1706.03762, 2017.
Pengcheng Yin, Graham Neubig, Miltiadis Allamanis, Mare Brockschmidt, and Alexander L. Gaunt. Learning represent edits, 2019.
Chunting Zhou, Graham Neubig, and Jiatao Gu. Understanding knowledge distillation in non- autoregressive machine translation. arXiv preprint arXiv: Arxip-1911.02227, 2019.
12


Reviews:
[{"review_id": "z0W_dI1zmdT", "reviewer": "Reviewer_MpCU", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies an edit-based generative text model that starts with a complete noise distribution (random gibberish) as input and then produces a series of edits to reach high-quality output. Inspired by diffusion models in CV, this Diffuser model rivals or outperforms standard autoregressive models on various generation tasks (MT, summarization), while also providing additional editing-based functionality.", "summary_of_the_review": "I think methods for non-traditional forms of generating text are widely underexplored, and this paper proposes a method that rivals typical autoregressive generation methods that are commonplace today. The overall writing and paper quality is solid.", "strength_and_weaknesses": "\nStrengths:\n* The objective and model design is quite novel and refreshing. I appreciate the careful design of the architecture---rather than focusing on a pure end-to-end system is decomposes the task into edit tagging and generation.\n* The training objective seems sensible, and the overall evaluation is pretty solid. I specifically appreciate using the model as a sort of \"general-purpose post-editing\" model (as shown in Diffuser + AR bootstrap). It would be nice to consider more post-editing-like baselines, e.g., just adding some off-the-shelf post editors or grammatical-error correction models to the MT output.\n\nWeaknesses:\n* While sensible, the training objective is a bit limited in that it denoises pure random tokens. Concretely, to get good performance at test time, the model needs to denoise generations that have various errors spanning semantic errors, syntactic inconsistencies, etc. These types of errors are quite far from swapping in random tokens in the input.\n* The editing-based evaluation is a bit limited. It would have been great to explore more some of the capabilities/failures of the model. For example, conditioning on various keywords on the target side, conditioning on a target syntactic style, or similar evaluations.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "g1mZTOdbCY7", "reviewer": "Reviewer_E15v", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes DIFFUSER, a denoising diffusion model for text generative tasks. It treats text generation as a Markov chain of Levenshtein edit steps to denoise from the initial text. An editing step is modeled as an editing process in existing work (Reid & Neubig). \n\nThe contribution of the paper is mainly 1) using the more flexible edit operations in diffusion models; 2) more thoughtful bootstrapping from autoregressive output or source text as initial text sequence; 3) employing 2D-beam search during decoding. \n", "summary_of_the_review": "The paper is well-motivated and provided valuable exploration in the recent popular direction of diffusion models in NLP area. The approach, although not strong in novelty, effectively employed various techniques to obtain strong experiment results on multiple tasks. The presentation is mostly clear and well organized, although there are multiple places that need to be clarified or further explained and discussed.\n", "strength_and_weaknesses": "Pros:\n- Well motivated, provided valuable exploration in the recent popular direction of diffusion models in NLP area.\n- Very effective diffusion approach in NLP by adopting flexible editing operations.\n- Strong results in experiments on multiple tasks and datasets, compared against strong baselines including recent literature on non-auto-regressive generation models. \n- Effective techniques improved on top of the diffusion approach, including bootstrapping and 2D-beam. Ablation and analysis are also provided.\n\nCons:\n- There is novelty in the approach, but it is not considered significant, since there is already existing work on NLP diffusion models as described in Sec 2.3.\n- The description of models and experiment settings is not clear enough. Table 4 could be moved to Appendix to make room for clearer and more detailed descriptions and explanations.\n  1. Since the tasks are sequence to sequence generation, the Diffuser model can be looked on as the decoder, should there also be an encoder to take in the input text, so that the decoder can condition the output on the input? This was not described anywhere in the paper.\n  1. How many diffusion steps, and what b and r in Summarization and Text Style Transfer tasks? \n  1. Equation (3) is not explained clearly, including some symbols not annotated.\n  1. There is no description of the Accuracy metric in Table 2.\n- It would be helpful if some experiments and analysis are included for the number of diffusion steps in *training*. \n- Numbers of training time and decoding time compared to baselines would be helpful, together with some discussion on the trade-off of efficiency and efficacy.\n\nMinor edit:\nPage 8: \"comparsion\" -> \"comparison\"\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "AlCBztuKsIv", "reviewer": "Reviewer_Gnic", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Motivated by how humans revise content and the success of diffusion models with continuous inputs, the authors propose a generative model of text based on multiple editing steps, with each step based on one or more text span editing operations (insert, delete, replace, keep). The generative model is trained to invert, step by step, the random edits of a \"text diffusion process\", which is specified by a prior over edits and edit length. Generation at each step is further decomposed into a model for generating the edit operations to apply at each position of the current input text, and a model for generating text given the input and the selected edit operations, similarly to many existing edit-based generation approaches.\n\nResults on machine translation (WMT 14') and summarization (CNN Daily Mail) show competitive performance (Table 1). An advantage of the proposed Diffuser model can refine text and so can be boostrapped by an autoregressive translation or the source to summarize, which boosts performance (Table 1). The authors also show that by simply training 2 models for positive and negative sentiment Yelp reviews and then using them to transform the input text to the target sentiment, performance is competitive with SOTA task specific models, which is impressive. Some ablations around decoding method vs. speed and performance and seed text type are also included.", "summary_of_the_review": "Overall, the presented text-edit based diffusion model is much anticipated, well motivated, and has been well executed. However, as detailed in the limitations section, more credit and context as it relates to previous work should be included when presenting the elements of the model, and both the strengths and limitations of the approach should be frankly discussed, in supplementary material if necessary, to give the paper more context and depth. If feasible, additional experiments that investigate the extent that generations can be flexibly controlled would further strengthen the paper.", "strength_and_weaknesses": "Strengths\n\n- The model is intuitively appealing, and the paper in general is well written.\n- The experiments demonstrate competitive performance, and the advantages of being able to bootstrap and refine results.\n\nLimitations\n\n- While the paper well written, and establishes the approach well, it could benefit tremendously from some additional supplementary material, to give the paper more depth, as discussed in the following points.\n- The work is well executed and the model intuitive, but in context, an obvious next step given current work on iterative text generation models, and the recent success of diffusion models. More credit should be given to previous related work that establishes similar reconstruction and corruption processes, similar two-stage editing, and similar decoding processes. These are not novel components of this paper as suggested by the current manuscript. Related, I'd like the authors, to the extent possible, to discuss the similarities/differences advantages/disadvantages over similar text editing models, such as the Levenshtien transformer. Why would it not perform as well as Diffuser?\n- More flexible control over generation is claimed as an advantage of the model, but this is really never demonstrated or exercised. While I can imagine variations of the model that could deliver this, the models investigated are trained and utilized end-to-end, with intermediate generations that have no notion of validity associated with them, and internal decisions that, if manipulated, would likely degrade performance.\n- Related, investigating the generalization of and effects of manipulating the editing priors at test time to exercise some control and/or diversify outputs would be an interesting ablation/demonstration.\n- Statements like \"editing processes can also be used to calculate the probability of only the final document while taking into account previous revisions, which is not possible in the traditional text generation setup\", are misleading. Under the model intermediate revisions are meaningless, and conventional LMs can evaluate likelihood in a single pass (i.e., having to go through a revision process for likelihood evaluation is a disadvantage of the model). Generally speaking, a more balanced and frank presentation of the strengths and limitations of the approach, properly situated within context with previous work, would improve the paper signficantly. \n- Equation 5 has errors, please correct.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster