Self-conditioned Embedding Diffusion for Text Generation
Authors: Robin Strudel
Corentin Tallec
Florent Altché
Yilun Du
Yaroslav Ganin
Arthur Mensch
Will Sussman Grathwohl
Nikolay Savinov
Sander Dieleman
Laurent Sifre
Rémi Leblond
Venue: Submitted to ICLR 2023 (2023)
Keywords: language models
diffusion models
generative models

Abstract: Can continuous diffusion models bring the same performance breakthrough on natural language they did for image generation? To circumvent the discrete nature of text data, we can simply project tokens in a continuous space of embeddings, as is standard in language modeling. We propose Self-conditioned Embedding Diffusion (SED), a continuous diffusion mechanism that operates on token embeddings and allows to learn flexible and scalable diffusion models for both conditional and unconditional text generation. Through qualitative and quantitative evaluation, we show that our text diffusion models generate samples comparable with those produced by standard autoregressive language models — while being in theory more efficient on accelerator hardware at inference time.  Our work paves the way for scaling up diffusion models for text, similarly to autoregressive models, and for improving performance with recent refinements to continuous diffusion.

Under review as a conference paper at ICLR 2023
SELF-CONDITIONED EMBEDDING DIFFUSION FOR TEXT GENERATION
Anonymous authors Paper under double-blind review
ABSTRACT
Can continuous diffusion models bring the same performance breakthrough natural language they did for image generation? To circumvent the discrete nature of text data, we can simply project tokens in continuous space of embeddings, is standard in language modeling. We propose Self-conditioned Embedding Diffu- sion (SED), continuous diffusion mechanism that operates on token embeddings and allows to learn flexible and scalable diffusion models for both conditional and unconditional text generation. Through qualitative and quantitative evaluation, we show that our text diffusion models generate samples comparable with those pro- duced by standard autoregressive language models while being in theory more efficient on accelerator hardware at inference time. Our work paves the way for scaling up diffusion models for text, similarly to autoregressive models, and for improving performance with recent refinements continuous diffusion.
INTRODUCTION
Continuous diffusion models (Sohl-Dickstein et al., 2015) have taken the world of image generation by storm, advancing the state of the art further than ever before (Rombach et al., 2021; Ramesh et al., 2022). Can the same framework encounter as much success on the text modality? Diffusion for language is indeed an attractive prospect. Compared to autoregressive (AR) models (Bengio et al., 2000; Sutskever et al., 2011; I; Austin et al., 2021; Hoffmann et al., 2022), diffusion models can predict all tokens sequence once. This allows for bidirectional, rather than causal attention increasing interactions between tokens, potentially leading to more coherent samples. Diffusion models can make a better usage of hardware accelerators during inference than AR models, since computations are parallelizable over the sequence axis. Yet AR models remain the mainstream ap- proach for modelling text. major obstacle to text diffusion is that diffusion processes typically operate in continuous space. While this naturally handle images, text is inherently discrete. Con- sequently, most previous attempts to apply diffusion to text have focused on discrete diffusion- approaches. These methods do not benefit from the refinements made to continuous diffusion in the image domain. Crucially, they cannot make use of guidance (Dhariwal & Nichol, 2021), which drastically improves diffusion models sample quality.
We address this gap by making simple observation: language models operate mostly in continuous space, with discrete tokens only inputs and outputs. A natural idea is then conduct diffusion directly in continuous token embedding space. For simplicity, we use a fixed embedding space, either random or stemming from a trained language model. Combined with the "self-conditioning" (Chen et al., 2022) refinement, this forms the basis of the method we propose, Self-conditioned Embedding Diffusion (SED). SED models rival mainstream AR models in both conditional and unconditional text generation. We make the following contributions:
In section 3, we introduce SED, the first continuous diffusion approach for text with good scaling properties (testing models up to 420M parameters). We analyze several continuous text diffusion settings, and identify self-conditioning and diffusion on small fixed embed- dings as key factors to make continuous text diffusion work.
In section 4, we apply classifier-fie free guidance (Ho & Salimans, 2022) to text data an orig- inal achievement. We show that SED rival AR models on generic language tasks, for similar models sizes. SED samples achieve a better likelihood-entrop trade-off compared to these models, and are deemed comparable (if slightly worse) by human raters.

---

Under review as a conference paper at ICLR 2023
2
RELATED WORK
We provide an overview of diffusion models with a focus on modeling discrete data, as well as AR models and sample-ban baspe metrics for evaluating text generation.
Continuous diffusion on continuous image data. Continuous diffusion has recently established itself as the method of choice for modeling continuous data such as images. While our main focus in this paper is on discrete data, we review some key works in continuous data modeling as this literature was the major source of inspiration for SED. The first continuous diffusion formulation was introduced in the seminal work by Sohl-Dickstein et al. (2015). Ho et al. (2020) improved and simplified this formulation, relating it to denoising score matching, and creating a new method called DDPM. Nichol & Dhariwal (2021) further improved upon DDPM, showcasing impressive diffusion results compared to GANs. Rombach et al. (2021, Stable Diffusion) introduced diffusion in latent space. Conceptually similar to SED, was specifically targeted at image modeling. Classifier-free guidance was proposed by Ho & Salimans (2022) as mean to improve image fidelity at the cost of reduced diversity. GLIDE (Nichol et al., 2022) scaled up the ideas of guided diffusion, while DALL-E 2 (Ramesh al., 2022) and Imagen (Saharia al., 2022) are the latest, most advanced image generation systems to date, combining most of the improvements proposed in previous works.
Discrete diffusion on discrete data. One cannot simply reuse the methods that are successful on continuous image data in the discrete text domain. A number of bespoke methods have been explored instead, forming the family of discrete diffusion approaches. In discrete diffusion, the data is corrupted by switching from one discrete value to another. This first proposed in the seminal work by Sohl-Dickstein et al. (2015), where was tested on simplistic binary heartbeat data. It was extended to multinomial text modeling (Hoogeboom et al., 2021) and further scaled up in the D3PM work (Austin et al., 2021). Most recently, a similar discrete diffusion approach was applied to image modeling in VQ-Diffusion (Gu al., 2022). In parallel, few diffusion-like approaches were proposed in the denoising autoencoders literature. CMLM (Ghazvininejad et al., 2019) tackled machine translation. SUNDAE (Savinov et al., 2022) was the first non-AR method to show strong results both in machine translation and unconditional text generation. MaskGIT (Chang et al., 2022) demonstrated excellent results in modeling VQ-discretized images. These approaches rely on training models to predict masked tokens from their context, and iterating this reconstruction step multiple times at sampling time. Despite those positive developments, the samples from discrete diffusion methods for text modeling remains less coherent than those produced by AR methods.
Continuous diffusion on discrete data. Fewer works try to tackle diffusion on discrete data from the same angle as SED starting by turning the data into continuous representations before mod- eling it with continuous diffusion formulations. Mittal et al. (2021) used a VAE to generate such representations for discrete music modeling, with exciting results. Closest to SED, Liffusion-LLM (Li et al., 2022) trains token embedding together with the diffusion model itself. Diffusion-I-M meets success on specific language applications. in low data regime and on constrained, very formatted textual data. Most recently, Analog Bits (Chen et al., 2022) introduced relficonditionnng, closely re- lated to step-unrolls in SUNDAE (Savinov et al., 2022), together with bit-level modeling to improve the generation of discretized images. While the qualitative results of those continuous methods on text modeling show promise, they have not been shown to scale to large realistic text datasets like C4 (Raffel et al., 2020) yet, or compare with AR approaches on generic language tasks.
Auto-regressive modelling on discrete data. AR models remain the method of choice for mod- eling discrete data. In combination with neural networks, they were first explored by Bengio et al. (2000) and later combined with RNNs (Sutskever et al., 201 I). Their breakthrough moment came with the advent of the Transformer architecture, introduced by Vaswani et al. (2017) for machine translation. Even more impressive results were shown with GPT-3 (Brown al., 2020), which trained a large AR language model unconditionally. and used few-shot prompting to adapt it to new tasks. A few works later improved upon the results of GPT-3, including Hoffmman al. (2022).
Sample-based evaluation of text generative models. There are traditionally two classes of met- rics for generative modeling: likelihood- and sample-base based. While the tikelhhoodbbase way is mathematically appealing, its usefulness for measuring progress is reduced by the fact that not
2

---

Under review as a conference paper at ICLR 2023
all models readily provide likelihood computation. Just like the sampled-base basddd-badddd-e FID metric im- portant for driving the progress of diffusion in image modeling, there a need for a sample-based metric which would be universally accepted for text modeling. Caccia et al. (2018) investigated fidelity/variance metrics for evaluating text GANs. Semeniuta et al. (2018) suggested using FID for texts. De Masson d' Autume et al. (2019) later used those previously proposed metrics to iterate ScratchGAN but did not provide conclusive guidance on which metric a practitioner should choose- essentially finding serious vulnerabilities in all investigated metrics. We opted for a middle ground, reporting both sample likelihood according to a strong AR model and human preferences.
3
METHOD
In this section, we outline the different components of SED: continuous diffusion in the space of token embeddings and condiiondi,,,, which form the basis of our approach for unconditional text generation; span masking and guided diffusion to enable conditional generation.
3.1 DIFFUSION MODELS FOR UNCONDITIONAL TEXT GENERATION
Diffusion models in continuous space. We consider diffusion models as introduced by Sohl- Dickstein et al. (2015) and improved by Ho al. (2020). A diffusion model aims at modelling a data distribution 2% E R ~ q E D(R") ) by estimating a sequence of latent variables æT, 21 of the same dimensionality as the data Tg. Starting from ø0. the latent variables are generated with Markov chain called the forward process: æ, ~ g(- (x,-1,t). It is defined by gradually interpolating the iterate with Gaussian noise according noise levels defined by schedule $1,..., By:
x; q( 1. WWVIIIIIIIIII 3,
This parametrization gives us closed form sample xi for any arbitrary 1 1, given zip:
Dt V + vI 0g4 Virao v1 -6,
where 0₁ := Be, a2 == II N((. and c~ (O,
We define generative model by approximately inverting the diffusion process of Eq. to obtain a reverse process. The reverse process starts from 2T ~ NN I) and is defined as a Markov chain with learned Gaussian transitions (parameterized by 0, the weights of neural network): æ;-1 pa(- (æ,) = '' (ua t), (t) We train a neural network to predict an estimate to 0) of the data zo and approximate the reverse process by using the following parametrization, with learnable means but fixed variances, and a fixed schedule Bi,...,
(o( 1 Co(2,, t,0) ot))² 1- 1-01-1 01-1
3:-
1 aie
While there exists a tractable variational lower-bound (VLB) on logo)( Ho et al. (2020) showed that better results are obtained by optimizing a simplified objective that re-weights the terms in the VLB. We follow this approach, which simplifies the loss to a sum of mean-squared errors between the ground truth data En and its estimates &o t,
Laiffusion = Ex (117)17))) L0 t,
(4)
Though this framework works out of the box on images, which are close to continuous, we can- not apply it directly to the discrete tokens of the text modality. To resolve this issue, we perform continuous diffusion in a continuous space in which we embed text tokens.
Diffusion word embeddings. We consider textual data a one-hot representation in R of a discrete token in (1,..., VJ. Each token w has an associated embedding e,, E R², with fixed norm vD to match the norm a random gaussian sample in dimension D used to noise clean data. We denote by E I2 DxV the matrix of all embeddings.
(W....., wN), where each is
3

---

Under review as a conference paper at ICLR 2023
We define our diffusion process in embedding space, rather than in token space. To that end, define a forward discrete-to discretetoo-onntnnen conntnnuuus step qv (zo|w) = (Ew, oi where dy is a constant scale factor with similar order of magnitude as 31. Conversely, we define reverse continuous- to-discrete step pn(w)a0)) (20) I Cat(wr E (x)))) where R E IR VVD is a learnable readout matrix initialized E and Cat(wall (t) is the softmax probability of token with logits I RV
To train the readout step, we add a reconstruction loss to Ldiftusion during training. Conveniently, it naturally arises when deriving the VLB of po(w) with this discretization step (Li et al., 2022), introducing a simple cross-entropy loss to maximise po(w) wo)::
Lrcon E D.w.D.aar-viiii (w) log loggp (Rr (zo)],
with
Limil Ldiffusson + Lrecon- (5)
Contrary to what is done in Li et al. (2022), we do not learn the embedding matrix E, as we identified that it was empirically unstable and could lead to drops in unigram entropy. The reconstruction loss freeon therefore only depends on the trainable readout weights R.
At sampling time, embedding To of size dembed. We multiply it by R to obtain logits in RR, and then use the index of the maximum component convert it to a token wi, with i arg maxisjev (RTg). (RRr)) This entails running T full forward passes which is quite expensive compared to cached AR sampling; however each forward pass computes all timesteps once which is naturally parallelisable. Further, we hope to benefit from many diffusion sampling improvements to get T down to low double-digits.
the reverse process for T 1000 steps, ultimately yielding continuous
Self-condit (Chen et al., 2022). In standard diffusion sampling, at each timestep the denoising network generates an estimate Ty Co( t, 0) of Zo given only It as input. Self- conditioning progressively refines To estimates by passing the estimate / obtained at the previ- ous sampling step as input to the denoising network; the colf-conditioned estimate is then defined as Tb # (æte, =b++ I,, 0), and sets the diffusion direction. In practice conditioning is performed by concatenating æ, and a on the feature axis. To approximate the inference behavior at train time while remaining computationally efficient, we compute a first estimate z = 00 0, f, 0) with the self-conditioning set to zero, then perform second forward pass using stop gradient on zú to obtain म = to (21,
Lte z f, 0). The denoising network is then optimized using the output from the
two forward passes in order to estimate Ty accurately with and without self-conditioning.
Equipped with these components we can train models to generate text, though only uncondition- ally. To add conditional generation our system's capabilities,
two additional methods.
3.2 SPAN MASKING AND GUIDANCE FOR CONDITIONAL TEXT GENERATION
By design diffusion models for text generation are flexible and can handle a wide variety of infilling tasks. This a key advantage over the predominant auto-regressive language models that typically generate text in a left-to-right fashion.
Span masking. We train our model on a rich set of infilling tasks with the following method. We optimize the diffusion loss from Eq. 4 only over æ while fixing the conditioning tokens c. Conditioning tokens e are defined by a binary conditioning mask m set to one on conditioning positions and zero on positions to be infilled.
We sample conditioning mask m randomly as follows. Given sequence of length T. and a max- imum number of spans M, we sample a number of spans n uniformly in [1, M). Span starting positions are defined by n - 1 integers (i1, in-1) sampled uniformly without re replacement and sorted in increasing order to satisfy 0 < i1 < < in-1 < L. The tuple (i1, i1 partitions the sequence of tokens in n spans satisfying E[ix|n] = L. The conditioning mask m is defined using even spans for conditioning and odd spans for infilling, and then m is flipped with a 50% probability. corresponds to unconditional generation; then set to 0 everywhere.
The case
This span masking strategy defines a collection of text generation tasks with a large variety of con- ditioning which average evenly splits the sequence between conditioning and infilling spans. It enables conditional generation, and opens the door for additional diffusion improvements.
Guided diffusion. Guidance (Dhariwal & Nichol, 2021) often improves the sample quality of con- ditional diffusion models. We use classifier-fre guidance (Ho & Salimans, 2022), which alleviates the need for separately-trained guide model. In the conditional case, estimator do is now
4

---

Under review as a conference paper at ICLR 2023
Table 1:
SED samples on unconditional generation, fillin-the-mmidle and several spans in-filling.
Task
Samples
Unconditional
We make use of the very best supplies and solutions to ensure that work is going to stand up to the test of time, and we help you save money with techniques that do not change the quality of your mission. We'll achieve this by offering you the best deals the field and avoiding pricey mistakes. If you want to spend less, Refrigerator Unit Repair Guys is the company contact.
Fill-in-the- middle
A year ago in Paris, I had the opportunity to take a field trip to La Rite-en-Laurenee Laurenees Interna- tional de France where met David Nigel Johnson, professor of social studies. What great trip and what great day!
Spans in-filling
There was no evidence, only fleeting glimpses of the killer and his fate, In fact, it seemed that there was evidence. It was all guesswork, and of the most unusual murder cases throughout history.
function io c, - 0). where c are fixed conditioning tokens. During training, with fixed probability the conditioning tokens used in the estimator 2o are dropped and set to null label Ø. During sampling, the model prediction is extrapolated the direction of t0 c, a .t, 0) and away from æo (z,, 0, 0, 1, 0) as follows:
(6) where: 2 is the guidance scale. Remark that we jointly drop conditioning and the self- conditioning a++1 concretely setting both values zero. Classifier-free guidance allows leveraging both the unconditional and conditional abilities of a model to improve its conditional generations.
z = 2o ze. 0, 0, t, + (z0 10 2u (z,,
4 EXPERIMENTS
4.1 TRAINING DETAILS
We train all our models on the C4 dataset (Raffel al., 2020), using SentencePiece tokenizer (Kudo & Richardson, 2018) composed of 32000 words. We use a non-causal transformer model (Vaswani et al., 2017) as our diffusion model (see Appendix A for details). SED models are trained with sequence length 256, while for ablations models are trained with sequence length 128. We insert uniformly, i.e. not necessarily at the end of the sequence, 10% of padding tokens in the training set to allow SED models to generate samples of varying size and provide more flexibility.
To generate word embeddings, we train BERT model of fixed size (150m parameters) and feature dimension dmuxeel = 896. The diffusion space is defined by the initial lookup table of this BERT model. We bottleneck the dimension of the word embeddings dembed and add a linear projection layer from dambed to dmodel at the beginning of the model. We found this helped diffusion (see section 4.4).
SED models are trained with cosine noise schedule (Dhariwal & Nichol, 2021), with 31 2.10-3 To = 10 and T = 1000. We use batches of 65. 35.536 tokens, thus for sequence length 256 the batch size is set 256. We use a maximum span count of 5 for all runs except for its speeific ablation. We train SED models at two different scales: SED-S (1 (135m parameters, 10 training steps) and SED-I. (420m, 2.10 steps). Their detailed architectures can found Appendix A.
4.2 VALIDATION
While optimizing the perplexity of AR models for text leads to improved language models, directly optimizing the ELBO of diffusion models for images does not correlate strongly with sample quality as observed by Nichol & Dhariwal (2021); Kingma et al. (2021); Ho & Salimans (2022). For images, the sample based metric FID (Heusel et al., 2017) has been introduced as measure of sample quality and is now widely adopted. Similarly, we need sample-based metric for text generation that is reliable and allows comparison between a large variety of generative models. To provide a fair comparison to AR models, we rely on three metrics.
The first metric measures how likely the samples produced by a model are according to an AR language model with 70B parameters, trained on 1.4B tokens (Hoffmann al., 2022); we denote
5

---

Under review as a conference paper at ICLR 2023
this metric AR NLL for auto-regressive negative log-likelihood. It provides continuous measure of sample quality that has proven useful when combined with a measure of sample diversity, e.g. in the development of nucleus sampling (Holtzman et al., 2020) for improved AR model decoding.
To measure diversity we rely on a second metric, the unigram entropy samples, which helps balance the AR NLL that can be gamed by unnatural repetitive samples. For both these metrics, target is the score of the validation set data. Deviating from the data unigram entropy in particular sign of degenerate modeling.
Though this initial combination has provided us with a reliable signal to iterate over our model design, it remains imperfect; it too be gamed, though it is harder do To address this limitation, we also report human preferences. We presented 6 colleagues with 20 pairs of samples for each comparison, asking them pick the best one. For all three metrics, we report results on two tasks: unconditional language modeling and suffix in- in-filling, the later a heavily conditioned task.
4.3 RESULTS
Samples. We present samples generated with our SED models in Table 10. We use single model to perform a wide variety of text gener- ation tasks, such as unconditional generation, seatnn-i--themmidee
5.0
4.5
SED
or filling several spans of text. We show strong performance in the un- conditional case, with samples that are syntac- tically correct and stay coherent long se- quences. the conditioned case, SED models able to infill spans with coherent transitions and links to the conditioning but also exhibit rich diversity. By design, SED yields flexible bi-directional masking models that can perform text generation diverse of conditioned task. To compare SED with AR baselines we next restrict conditioning to a prefix and con- sider a task of suffix in-filling.
SED
Data
4.0
AR NLL
3.5
2.0
6.5
6.6
8.8 Unigram entropy
6.7
6.9
7.0
Comparison AR models. To assess the gen- eration ability of SED, we compare against AR baselines of similar capacity and trained fol- lowing optimal scaling laws from Hoffmann sample
Comparison of sample quality and diversity SED versus AR models on suffix in-filling. SED uses guidance with scales in {1,2, 4, and AR uses nucleus sampling with a top-p in (1.00, 0.95, 0.90, 0.85). Top-right points are SED models with a guidance scale of or AR models with a top-p of
Figure 1:
al. (2022) on suffix in-filling. batch of sequences from C4 and use the first 128 tokens as conditioning given to the model generate suffix of 128 tokens. Figure I re- ports AR NLL and unigram entropy of the gen- erated suffixes for AR and SED models. As reference point, we compute the AR NLL and unigram entropy of the ground truth C4 suffixes and report it on the plot. Several methods can be used improve sampling quality at the cost of samples diversity; we use nucleus sampling (Holtzman et al., 2020) for AR models and guidance (Dhariwal & Nichol, 2021; Ho Salimans. 2022) for SED models. We show the impact guidance on sam- ples quality in Table 3. To our knowledge, we are the first to show sample quality improvement when using guidance for text generation.
As shown in Figure 1, both SED-S and SED-L perform strongly when compared against AR base- lines-ewee - even though we report metric favoring AR models on a task AR models have been designed to optimize. Similar to nucleus sampling for AR models, guidance has a strong positive impact on sample quality that is both observed quantitatively with improved AR NLL in Figure and quali- tatively in Table 3. We observe that using top-p nucleus sampling below 0.8 for AR models guidance scale above 4 for SED models leads to samples exhibiting a lot of repetitions, a degenerate case reflected by lower entropy of samples even though sample AR NLL improves.

---

Under review as a conference paper at ICLR 2023
Table 2:
We compare SED-L (guidance scale 2.5) and AR-L (nucleus sampling, p 0.95) samples.
SED I.
AR-I.
You're going to love wearing this traditional tee from our latest Wilson collection. Designed in scrapped floral styled knit with sleeve of asym- metric lines across the round solc. Lightly fluffy, the square pleats will take you right to
A Koda Ram 25 is presented in sedan and Max- ima saloon. Based on the Acenta car, the power- ful Koda 2014 hits Indian roads in the "Maxima" body-con shape. Being powered by Hyundai i20 1.4 litre diesel engine, the Koda 2014 is coupled
The beaver is interesting animal that lives in rivers and lakes. He is not mainly concerned with finding wolves and dolphin but also has great hunger for fish. The beaver has sharp legs, large eyes, and black coat
The beaver is interesting animal that lives in rivers and streams. is usually seen big num- bers in the fields or upstream, and is quite docile. On cold days when its pattern is perfect, the beaver will have some interesting, and sometimes
Once upon time in Spain, Leonardo Pueleva had the pleasure of meeting guests at Spanish restau- rant, Buva Casinos. While driving, he got a chance to get to know the people behind the restaurant and, course, how they made his experience very interesting, After his conversation, he got to
Once upon time in Spain, which seems pretty much the same way now, the question that was posed to each of the end of our interview was "would you like to see Froome one day?" In ret- rospect, after our interview, we have grown ever closer to that answer. As you will read in the arti- clc, I know that
Table 3: Impact of guidance on samples quality using our SED model.
Guidance 1.0
2.5
5.0
In the cold, cold night sky, a fairy princess sits in chair and surrounded by tea leaves in pond. Meanwhile, she bies back into the cold, with bluish hair her hips and el- bows on her forehead and her fingers numbed by the freezing temperature.
In the cold, cold night of November 2018, little girl sits in a chair hidden under light blanket on patio. Meanwhile, she bends back into the chair with bluish hair on her forehead, her hands on her face, her fingers numbed by the freezing tem- perature.
In the cold, cold night of December, my oldest daugh- ter sits in chair accentuated in cotton fabries and pil- low. Meanwhile, she yearns straight in the cold air, her wrists covering her neck, her eyes straight on her forehead, and her fingers numbed by the freezing temperature.
Barbara was of many wonderful women that really helped so I am blown off by her purpose, civility; and adversity. Once started inter- acting with her, it proved to me that no matter how hard this was, she always strove for excellence.
of the most gifted women in the world. She was creative and stood by her integrity and ci- vility; against adversity. though she placed herself higher than her peers.
Barbara
Barbara brilliant women in the world. She was amazing her heart, her spirit, her mind and in the soul. never turned people off in her absence. It proved to me that no matter how hard this was, she always strove for excellence.
of the most
proved to me that matter how hard this was, she al- ways strove for excellence.
Our human preference scores temper observations in Table 4. They show that our NLL and entropy metrics do not tell the whole story, as humans still prefer AR models at equivalent size. While SED-L performs slightly worse than AR-L (38% preference in suffix in-filling, 44% un- conditional generation), its scores remain comparable. SED-L is roughly on par with AR-S.
Finally, we compare SED and AR models' qualitative examples with short prompts in Table 2.
Table 4: SED-L vs other models human preference scores on conditional and unconditional tasks.
SED-S (cond)
AR-S (cond) AR-L (uncond)
AR-L (cond)
SED-L 63.4% 4.3%
51.0% + 5.0%
43.8% + 4.4%
37.7% - 4.4%
7

---

Under review as a conference paper at ICLR 2023
4.4 ABLATIONS
Table 5: Ablation of the proposed SED approach on unconditional generation. conditioning and embeddings pretraining play key role in the model performance.
Both self-
Diffusion space
Self-conditioning
Unigram entropy
AR NLL
Bits (Chen al., 2022)
X
6.97
7.01
7.47
6.05
Random embeddings
X 6.86
6.90
6.80
5
5.31
Pretrained embeddings
x
6.75
5.66
6.77
4.57
Data
6.70 + 0.04
1.81 + 0.15
Self-conditioning and embedding pretraining. Results from Table 5 and samples from Table 6 show the influence of both the diffusion space and self-conditioning. AR NLL decreases very significantly when using self-conditioning, regardless of the rest of the setup. Diffusing at the bit-level (Chen et al., 2022) yields very high NLLs. While using random embeddings performs markedly better, using pretrained embeddings results in further improved numbers.
Samples from Table 6 highlight that models trained on random word embeddings exhibit topic mod- elling abilities with the co-occurrence of words like child and mother even though the paragraph remains globally incoherent and meaningless tokens like gluc are generated. Self-conditioning dramatically improves sample quality; the diffusion model gets the low-level structure right and generates syntactically correct sentences, even though the global text is not intelligible. Combining self-conditioning and pretrained embeddings leads to globally coherent paragraphs that stay on topic with proper sentence structure.
Embedding dimension. An important design choice for SED is the word embeddings space. We study the influence of pretrained embedding size in Table 7. Surprisingly, there is a threshold af- ter which performance degrades when increasing the dimension of embeddings. We visualize the forward process for different embedding sizes by displaying the nearest neighbor of noised token while running the forward process. In high dimension we observe that the nearest neighbor a noised token remains the starting token itself until it switches token. In low dimension, we often observe that the closest neighbor of a noised token goes through several semantically related tokens (nearest neighbor of the starting token) before ultimately becom- ing random. We hypothesize that the random walks defined by diffusion are more likely to drift towards neighbors of the starting token low dimension. As a result, when diffusing low dimen- sion information is destroyed in more semantically meaningful fashion, which leads to easier learning problem for the denoising function.
completely random, unrelated
Number of spans. In order to enable in in-filling, train the model not only to do unconditional generation but also to conditionally fill spans of tokens. For each data point we sample span number uniformly at random and span delimiters to generate the span mask. Picking the maximum allowable number of spans has a significant effect on model performance, as we can see in Table 8. Somewhat counter-intutiiveyy adding span masking improves also appears that using relatively high maximum span number is optimal. We hypothesize that this results in varied mix of task difficulty at training time, between "easy", very conditioned problems on the one hand and "harder", unconditional ones on the other.
unconditional generation NLLs.
Scaling. We show encouraging results when scaling from SED-S (150m) to SED-L (420m). We train both models on sequences of 256 tokens and report AR NLL of 4.20 for SED-S compared 3.68 for SED-L. This improvement translates to improved sample quality, as is confirmed by our human preference scores, which are much higher for the larger model (63%, see Table 4).
5 LIMITATIONS
While our results are promising and show that continuous diffusion for text can be an exciting alternative to AR models, the current approach does present significant limitations.
8

---

Under review as a conference paper at ICLR 2023
Table 6: On unconditional generation, self-conditioning results in better sentence modelling, pre-
trained embeddings enhances topic modelling.
Random embeddings
Random embeddings with self-conditioning
Pretrained embeddings with self-conditioning
did the buildingroom granted a lighter distance. On though, salaries about clients that a child, which dispersed glue so many events and certainly wanted the Mother's project, discovered by their child would keep
Tree brings the sound. bearing features and capabilities that we are set in. For the first time, she uses customizable framework use that we help students pub- licly solve the weather conditions that only offer students
almost six decades ago 72 per- cent of Americans didn't feel they'd ' actually rent their own cars this year. Conversely, 90 per- cent of Americans feel that the decision to rent car is some- thing they feel it's impossible
Word embeddings with small di- mension have higher AR likelihood.
Table 7:
Table 8: Span masking tasks improves un- conditional text generation.
Embed. dim. 16 32
64
128
256
896
Max span count AR NLL
5 9 11
AR NLL 4.65 4.57
4.71
4.61 4.77 4.92
4.99 4.82 4.82 4.67
4.48 4.56
First, much more could be done terms of model tuning, including scaling to much bigger models to better understand SED's limits, and to be able to compare it with state-of-the-ar AR models. Our training regime in particular would certainly benefit from more hyperparameter optimisation.
Second, one compelling reason we chose to explore continuous diffusion for text is to leverage the improvements produced by the literature on image generation. While we have ported some (e.g. self- conditioning), lot more remains unexplored. The most obvious example is the sampling process itself, where the number of required steps has been considerably reduced for images (e.g. Karras et al. (2022) goes from 1000 to 35, and Salimans & Ho (2022) all the way down to 4 on simple images). Our current sampling is very inefficient, and this direction is one the first improvements to make over SED.
Third, SED crucially relies on diffusing in a pretrained embedding space. This means relying on second model, and using embeddings that may not be optimal for diffusion. Ideally, we'd we'd train the full model end-to-end, which could yield even better results. While Li et al. (2022) found some success with this approach, it was in specific setting at small scale; in practice we found it difficult to avoid competition between the diffusion and reconstruction loss.
Finally, work would benefit from improved metrics in the experimental section. Because the current state of the art involves AR models, the field lacks established benchmarks for tasks diffusion models are potentially better suited for, such as text in-filling. We opted for reasonable mix, evaluating the negative log-like ligeehood of generated samples according to a very strong AR model as well as their token entropy and complementing with human evaluation. However, both NLL and unigram entropy are gameable (e.g. AR models assign very low NLL to repetitive snippets, and long enough repetitions can fool even entropy). Further, our NLLL inherently tied its AR model and could thus be providing unfair advantage to AR models. All told, we still found both metrics quite useful for measuring research progress, and our human evaluation confirmed our results. Moving forward, defining clean in-filling benchmark would help produce even more convincing results.
6 CONCLUSION
We propose SED, the first generally-capable continuous diffusion model for text generation. SED models can perform both conditional and unconditional generation, and their performance rivals AR models while being more flexible in their use (e.g. enabling in-filling) We demonstrate their performance and study the impact of the main design choices.
Despite its limitations, this work lays the foundation for more exciting research. Promising direc- tions include speeding up the sampling following the lessons learnt in the image domain, devising better embedding spaces for diffusion and investigating new in-filling capabilities.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Jacob Austin, Daniel D. Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg. Structured denoising diffusion models discrete state-spaces. In NeurIPS, pp. 17981-17993, 2021.
Yoshua Bengio, Réjean Ducharme, and Pascal Vincent. A neural probabilistic language model. Advances in neural information processing systems, 13, 2000.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Massimo Caccia, Lucas Caccia, William Fedus, Hugo Larochelle, Joelle Pineau, and Laurent Char- lin. Language gans falling short. arXiv preprint arXiv: 1.02549, 2018
Huiwen Chang, Han Zhang. Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 11315-1 1325, 2022.
Ting Chen, Ruixiang Zhang, and Geoffrey E. Hinton. Analog bits: Generating discrete data using diffusion models with self-conditioning. CoRR, abs/2208.04200 2022.
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V. Le, and Ruslan Salakhutdinov. Transformer-xl: Attentive language models beyond fixed-length context, 2019. URL https: //arxiv. orgiiivorrgaaee abs/ 1 901 02860.
Cyprien De Masson d' Autume, Shakir Mohamed, Mihaela Rosca, and Jack Rae. Training language gans from scratch. Advances in Neural Information Processing Systems, 32, 2019.
Prafulla Dhariwal and Alexander Quinn Nichol. Diffusion models beat gans on image synthesis. In NeurlPS, pp. 8780-8794, 2021.
Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer. Mask-predict: Parallel decoding of conditional masked language models. In EMNLP/JCCNLP (7), pp. 6111-6120. As- sociation for Computational Linguistics, 2019.
Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and
Baining Guo. Vector quantized diffusion model for text-to-image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10696-10706, 2022.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus), 2016. URL https:/
arxiv. org/ abs 1606. 08415.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In NIPS, pp. 6626-6637, 2017.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. CoRR, abs/2207.1259t 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurIPS, 2020.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, al. Train- ing compute-optimal large language models. arXiv preprint arXiv:2203. 15556, 2022.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. ICLR. OpenReview.net, 2020.
Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, and Max Welling. Argmax flows and multinomial diffusion: Learning categorical distributions. Advances in Neural Information Processing Systems, 34:12454-1244 2021.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. arXiv preprint 2xXi::2206.00344 2022.
10

---

Under review as a conference paper at ICLR 2023
Diederik P. Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. CoRR, abs/2107.00630, 2021.
Taku Kudo and John Richardson. Sentencepiece: simple and language independent subword tokenizer and detokenizer for neural text processing. In EMNLP (Demonstration), pp- 66-71.
Association for Computational Linguistics, 2018.
Xiang Lisa Li, John Thickstun, Ishaan Guirajani, Percy Liang, and Tatsunori B. Hashimoto. Diffusion-Im improves controllable text generation. CoRR, abs/2205.1423 14217,, 2022.
Gautam Mittal, Jesse H. Engel, Curtis Hawthorne, and Ian Simon. Symbolic music generation with diffusion models. In ISMIR, pp. 468-475, 2021.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In ICML, volume 139 of Proceedings of Machine Learning Research, pp. 2021.
PMLR,
Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. GLIDE: towards photorealistic image generation and editing with text-guided diffusion models. In ICML, volume 162 of Proceedings of Machine Learning Research, pp. 16784-16804. PMLR, 2022.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21:11001-144:61 2020.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with CLIP latents. CoRR, abs/2204.061225, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. CoRR, Ahs/2112.11752. 2021.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kam- yar Seyed Ghasemipour, Burcu Karagol Ayan, Sara Mahdavi, Rapha Gontijo Lopes, Tim Sali- mans, Jonathan Ho, David J. Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffu- sion models with deep language understanding. CoRR, abs/2205.1 14477, 2022.
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint arXiv:2202.00511 2022. URL https::t////s. //arxiv///vvvv.///aa///a https////axavwv.oaaaaa/ae org/obrgss..... 2ttpp////22220.20...... 00512.
Nikolay Savinov, Junyoung Chung, Mikolaj Binkowski, Erich Elsen, and Aäron van den Oord.
Step-unrolled denoising autoencoders for text generation. In ICLR. OpenReview.net, 2022.
Stanislau Semeniuta, Aliaksei Severyn, and Sylvain Gelly. On accurate evaluation of gans for lan- guage generation. arXiv preprint 0e06.09996..0990 2018
Jascha Doch-DDcksttei, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. In ICMI., volume 37 of JMLR Workshop and Conference Proceedings, pp. 2256-2263 JMI.. 2015.
Ilya Sutskever, James Martens, and Geoffrey E Hinton. Generating text with recurrent neural net- works. In ICML, 2011.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- tion processing systems, 30, 2017.

---

Under review as a conference paper at ICLR 2023
A MODEL ARCHITECTURE
For both the AR and the SED models, we use the same transformer (Vaswani et al., 2017) archi- tecture, which are similar to those described in (Hoffmann et al., 2022), with relative positional encoding as described in (Dai et al., 2019) in the attention blocks, and with a 4x expansion and Gelu (Hendrycks & Gimpel, 2016) non-linearity in the ferwafo blocks. The architecture hyper-parameter are detailed in table 9. Noised word embeddings, a E I RND,, are first passed through a linear projection that operates on each embedding independently to get projected em- bedding whose feature dimension matches the width of the transformer, dmodee- At diffusion step I, we compute time embedding as sinusoidal position embedding (Vaswani al., 2017) of size dmodel. which is then passed into a dmodel X dmodel linear layer and added to the projected embedding. We add a linear output projection layer E' which takes the output the transformer y E R d and projects each element (y: Jisig N back to the same size as the word embeddings. When using ooffccondiiinningg we modify the input to the model by concatenating x and žn along the feature axis before passing them the input projection layer.
Table Model hyper parameters.
Model
number of layers
dmodel
number of heads
head size
S
12
896
16
64
12
1536
16
128
B FORWARD DIFFUSION PROCESS VISUALIZATION
To support the discussion on word embeddings dimension from Section 4.4, we present visualiza- tion of the forward diffusion process. Given starting tokens 2o. we project the noised tokens æ, of the forward process at step t to their nearest neighbor among word embeddings E to obtain wy. We then store the 128 nearest neighbors ((w,) of starting tokens wh zp and define the rank r, of w, at its index in N ((na). We display and highlight it in green if r; is close to zero (meaning a close neighbor of wa) and in increasingly red colors otherwise. We present the first 16 nearest neighbors of wo in Figure 2 and provide illustration of the color code used for highlighting. Fig- ure shows an instance of the forward diffusion process while diffusing on embeddings of with a high dimension of 896 and Figure 4 shows diffusion on embeddings with a lower dimension of 32.
We observe that in high dimension, the noised token's 's closest neighbour remains the original token up until the point where any token could be its closest neighbour. The diffusion random walk does not seem to pass through the neighbourhoods of semantically-relatee tokens. We hypothesize that the root cause of this issue is that the embedding space mostly empty (with only 32000 points in R896 as dembeed = 896); and that embeddings are potentially concentrating in a lower-dimensional space.
In contrast, in lower dimension we see reaningffull-relltted tokens appear as the corruption pro- gresses ('brown' becomes 'grey", 'quick' becomes 'swift', "over' becomes 'underneath" etc). We believe this more gradual information destruction is beneficial for the diffusion model.
C ADDITIONAL SAMPLES
12

---

Under review as a conference paper at ICLR 2023
00 the quick brown
fox
jumps over the lazy
dog
The fast
gray
wolf
swings around hungry puppy
02 the swift
grey
rabbit
moves
under their tired
03 The speedy
pink
spider kicks
boring horse
04 their easy
yellow
deer
spins
about our silly pup
simple
orange
lion
through the lonely dogs
06 their slow
black
bull
hikes in its bored
07 your quicker
purple
tiger
blows after your Impatient animal
08
those straightforward
red
frog
pulls within an weary Dog
these faster
superhero knots for his straw shark
blue
pig
10 its rapid
loops across those noisy
breed
11 my instant
dark
bird
curves off these crazy
pony
12 put nice
coloured crow
catches up this stupid goat
our brief
redd
pushes out my anxious
14 this short
white
duck
draws behind The awkward
15 Our
prompt
cream
elephant
gears
against any
miserable monkey
Figure 2: Nearest neighbors of tokens from the sentence "the quick brown fox jumps over the lazy
dog"
0 the
quick brown
fox
jumps
over
the lazy
dog
25 the quick brown
fox
jumps
over the
lazy
dog
50 the quick brown
fox
jumps
over the
lazy
dog
75 the quick
brown
fox
jumps
the lazy
dog
100 the quick
brown
fox
jumps
the lazy
dog
125 the
quick
brown
jumps
over the
lazy
dog
150 the
quick
brown
fox
jumps
over
the
lazy
dog
175 the
quick
brown
fox
jumps
over
treatment
lazy
dog
200 thel
quick brown
fox
flights
over
army
lazy
ole
225 stated
quick brown 25
flights
over
army
lazy
Rams
250 makers
quick tan
25
flights
over
army
Philip
Rams
275 furt
quick tan
herical
job
vol
third
Rays
ball
300 Search quick abdomen char
job
vol
third
pathway
thal
Figure 3: Visualization of the forward diffusion process up to 300 steps when diffusing on embed-
dings with dimension 896.
the quick brown 25 the quick brown fox
fox
jumps
over
the
lazy
dog
jumps
over the
lazy
dog
50 the quick
fox
jumps over the lazy
dog
75 the quick brown fox
jumps
over the lazy
dog
100 the
quick grey
fox jumps over the lazy
dog
125 the Pinterest gray
abduct jumps underneath changed lazy
animal
150 vivo
swift
gray
fox hikes over
No bingo
dog
175 rapid
eager grey
frog umbled
over
ired usi
collision
200 rapid
ggy
boy
mosqu uttering
underneath
proud
casters
diesel
225 inter
ggy
blonde mosqu enced
underneath Nor
gambling
tow
250 inter
tend
pink
illeg acious
elsewhere
casino
cargo
275 un
gray
erectile
dreaming
here
Nor
casino
bay
300 Al
gray
thir
dreaming
here
Nor
rare
Figure 4: Visualization of the forward diffusion process up to 300 steps when diffusing on embed-
dings with dimension 32.
13

---

Under review as a conference paper at ICLR 2023
Reverse process
10V
evins Drvakee P6
C4
120
11540
636
The (ortto
the
TAR
1972
The
cnte (bx ivar
1772
X36
The some piinn tio cad
180
Te
116
2010
70%
116
Platte plas
18)
716
1i0h
114
12200
116
1996
1982
ETG
1991
115
680
2003
Frrt
115
F14t
180
1141
1155
1569
1551
1221
1229
1547
1551
1304
180
Te.
loca emtil Di
15
6WA1
33%
METUSA
18
11000
(94)
1.0
1x
treak (ic mm
27.3
(5000)
15
De0m
im
pat:
10
leeil eari (Diiliv re rvas iiles
1471
131
301
130
The teccch entrn beet rvas und ldkes
190
91
Doar
110
Figure 5: Reverse diffusion process of SED-L with guidance scale 2.5.
Table 10: Samples from SED-L.
This course is essential for the environment in which students choose the curriculum option study at JHD. Geographical Integration is at the heart of this department. Along with the ur- ban infrastructural innovations of the mid-1990's and social issues in the 2 Century. People within the department regularly exemplify the concept of real integration.
That did trigger some rewarding words or insights to begin preparing their kid for their future. Luckily, Mikaela has been nice enough to tolerate my questions about what parents can do to help, even during her summer vacation.
went to college Boston University. After getting my degree. decided to make a change! enrolled in outdoor schools. After getting my degree, loved jet skiing, offshore fishing, and Kitesurfing. The list became growing. More importantly, started a career by fishing at sea. Now, can't 1 get enough of the Pacific Ocean!
The beaver is interesting animal that lives in rivers and waterfalls across Puerto Rico. It loves kayaking. fishing, and swimming. But, you want to know what are the animals behind the heaver?
A year ago in Paris, had the opportunity to take field trip to La Rite-en- Rite-en-Laurencee Laurences Interna- tional de France where I met David Nigel Johnson, a professor of social studies. What great trip and what a great day!
A year ago in Paris, my friends and I went on a dirt road trip to the city. remember walking through the church, its beautiful square, narrow streets lit with memorials and passing terrible Catholic Bishop I am used to what a sad day...
There was no evidence, only fleeting glimpses of the existence of cognitive disability. There was no luck and no solid science. It was all guesswork, the blinding prospect of pncumonia could spur imagination at the possibility of brain damage.
14


Reviews:
[{"review_id": "m0Xp-MjCj9Z", "reviewer": "Reviewer_gGpy", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper describes an application of continuous diffusion models (Sohl- Dickstein et al., 2015; Ho et al., 2020 to text. Methodologically, this work differs from previous continuous text diffusion (Li et al., 2022) by leveraging recent insights on self-conditioning (Chen et al., 2022) and classifier-free guidance (Dhariwal & Nichol, 2021). Empirically, this work advances the state of the art for text diffusion by scaling these methods to a larger dataset (C4; Raffel et al., 2020) with larger models (135M and 420M parameter models) than considered by Li et al., 2022.", "summary_of_the_review": "An interesting application of recent innovations in diffusion models to the text domain at larger scale than earlier work. The empirical evaluation of the resulting model is somewhat lacking.", "strength_and_weaknesses": "This is a timely result on diffusion modeling for text generation, applying recent innovations in diffusion modeling (self-conditioning, classifier-free guidance) to text-domain diffusion modeling at a larger scale than previous work. Demonstrating the effectiveness of these methods on text data at scale is a valuable contribution.\n\nThat said, the evaluation of the quality of these models is somewhat lacking. Evaluation of open-ended generated text is a difficult and active area of research, which this paper largely ignores: for example (non-exhaustive list): (various metrics) Holtzman et al, 2019, (HUSE) Hashimoto et al., 2019, (Mauve) Pillutla et al., 2021. Instead of following guidance from the literature on evaluation, this work uses the following metrics: sample perplexity, unigram entropy (not a standard diversity metric), and head-to-head human preference using a small sample with limited nuance: \"We presented 6 colleagues with 20 pairs of samples for each comparison, asking them to pick the best one.\" It is difficult to feel confident in the quality of these models based on these statistics and the limited samples presented in the text.\n\nThat said: here is what I took away from the evaluation. The human evaluation results (Table 4) do give me confidence that the SED models are somewhat reasonable: the SED-L (420M param) model is competitive qualitatively with the AR-S model. It is interesting (Table 5) that Analog Bits (Chen et al., 2022) performs poorly compared to the tailored solution for text (the performance differences are large enough that I believe them, despite concerns about the choice of metrics). On the other hand, I do not find the more subtle perplexity differences presented in Tables 7 and 8 convincing.\n\nOne question I have is: do the authors plan to release these models to the community? Releasing pre-trained text diffusion models of this scale would in itself would be a contribution, and to some extent mitigates concerns about the evaluation because members of the community would be able to use the model and reach their own conclusions.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "iI6C2DhmAB", "reviewer": "Reviewer_U4Mu", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors present a continuous diffusion model that achieves performance comparable to autoregressive language models for natural language generation. The authors identify that self-conditioning and small fixed embeddings are important for achieving good performance with diffusion models for text generation. ", "summary_of_the_review": "The authors successfully show how diffusion models can be used with large language datasets. However, the authors mostly use techniques that are introduced in previous papers (self-conditioning, guidance). The most important contribution of this paper is to use pre-trained embeddings for diffusion. The paper can also benefit from additional results on other NLP tasks like controllable text generation. ", "strength_and_weaknesses": "Strengths\n* The authors were able to scale diffusion models to work well when trained on large NLP datasets by using pretrained embeddings, self-conditioning (originally proposed in the Analog Bits paper) and a bottle neck for embedding dimension. \n* The proposed method SED-L achieved lower perplexity and better unigram entropy when compared to an autoregressive model of similar capacity.\n* The authors identified that classifier-free guidance can improve language generation results (in both the condition and unconditional setting).\n\nWeaknesses\n* It's not clear why a separate BERT model is trained from scratch, instead of using a publicly released checkpoint. Was it trained differently or on a specific dataset?\n* The authors mention that using randomly initialized embeddings lead to instabilities in training and therefore propose using pertained embeddings. An ablation that would be of interest is to initialize the word embeddings with pre-trained embeddings, but also further finetune the embeddings as the model is trained. \n* There are qualitative samples for different guidance values but it is hard to judge which is better and in what way. Presenting a quantitative evaluation with the different guidance value would be useful. Furthermore, the guidance explanation is missing some details. For instance, how are the conditioning tokens c provided to the model? Is x_t modified such that the tokens corresponding to the conditioning tokens are replaced with the ground truth?\n* For bottlenecking, it is unclear whether bottleneck means project down from initial embedding dimension and back up to d_model or reduce the size of d_model. \n* MAUVE (https://openreview.net/pdf?id=Tqx7nJp7PR) can be used as an additional evaluation metric to compare autoregressive models and SED models to increase confidence in the results. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "KvyflRLoovC", "reviewer": "Reviewer_avZf", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a method to train text based diffusion models which can generate text non-autoregressively modeling bidirectional context. Since diffusion models work in continuous domains only, the authors propose the following changes to the standard setup (1) represent a sequence of tokens as a sequence of pretrained embeddings (obtained from a BERT-based model) and compute a standard diffusion training loss (L2), (2) to recover the discrete tokens from predicted embeddings, add another loss to the diffusion training loss, which computes the cross entropy wrt the target token by projecting the predicted vector back to the vocabulary space using a learnable matrix, and (3) add self-conditioning to this framework following Chen at al 2022. Trained a large corpus (C4), the generated LM shows improvement over autoregressive LM on the computed quality and diversity metrics. Ablations have been conducted on the proposed changes and show the importance of each setup. ", "summary_of_the_review": "The paper presents a diffusion language model by applying diffusion on pretrained embeddings. The evaluation seems to suggest that the outputs from this model have better quality and diversity scores than autoregressive LMs, however some important metrics are missing so it's difficult to clearly say if the model is better than the baselines. ", "strength_and_weaknesses": "Strengths:\n1. This is the first work on diffusion LMs I have seen that show results on a simple unconstrained text generation baseline and seems to have positive results in terms of the metrics used. \n2. The writing is clear and the motivations seem sound. There are some clever ideas incorporated like padding in the middle of the sequence, self-conditioning which seem to give the results a boost. \n\nWeaknesses:\n1. There is a discussion in limitations on the inadequacy of automatic metrics but the authors do not evaluate a variety of them that prior work has done such as MAUVE, distinct-n, zipf score, repetition rate and many more. These metrics should be computed to a get better idea of the quality and diversity of the output. Unigram entropy is not what is usually reported but would provide a similar trend to dist-1 I would imagine. Please see these this paper for the description of these metrics: https://arxiv.org/abs/2202.00666.\n2. There are missing baselines such as Li et al 2022's diffusion LM which the authors describe as not being good for large datasets but this claim is not verified. The ablation in Table 5 with random embeddings is not the same loss function as far as I can tell as the diffusion LM paper.\n3. The model rely on a pretrained BERT token embeddings which just seems like an arbitrary choice. An ablation should be provided with different pretrained embeddings to understand the impact of this choice. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "7t8pDQOZGP", "reviewer": "Reviewer_Sxgy", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper explores the Diffusion method for Text generation similar to the Diffusion Language Model (DLM, Li et al. 2022). Both this work and DLM propose to use continuous diffusion on the word embedding space to leverage recent advances in image diffusion techniques. This paper goes one step further and add self-condition (Chen et al. 2022) to the diffusion process to improve generation quality. This self-condition idea added the previous predicted embedding output of the previous diffusion step as a condition (through input concatenation) to the next diffusion step. The paper also proposes to use binary span masking to train the diffusion model in an infilling task and further explore diffusion guidance, as proposed in previous works for image diffusion.\n\nAs one of the first few works on diffusion language model, the results compared to SOTA AR models are mixed. Although there are some improvements in terms of token diversity and log-likelihood, the model still perform slightly worse in human evaluation.", "summary_of_the_review": "The paper explores several recent techniques in image diffusions in text domain. The results are encouraging, but there are two key concerns with the paper. The first is the lack of novelty, the continuous diffusion has been explored in Li et al. 2022, and all the model improvements have been proposed in the vision domain. The second is the lower performance of DLM compared to SOTA auto regressive LM, even when trained on a pretrained embedding space. Having said that, the paper does show some interesting observation that can be a good foundation for future research.", "strength_and_weaknesses": "Strength:\n+ This is one of the first continuous DLM works, and the paper explores the effectiveness of several key ideas in image diffusion in the text domain, namely the diffusion guidance and the self-conditioning.\n+ The performance of DLM, even with minimal changes compared to its vision counterparts, is shown to be encouraging, about equal to or at worst, slightly worse than AR models.\n+ Compared to Li et al. 2022, the authors tested the ideas on a larger scale data.\n\nWeaknesses:\nMost of the concerns I have are similar to what the authors have acknowledged in their limitation section.\n+ The diffusion process requires a pretrained embedding space while having a similar results compared to AR models. This make the diffusion LM substantially more expensive and inefficient.\n+ The data/model size are not large-scale\n+ Most of the techniques used in the paper have been proposed perviously.\n\nThe author mentioned that \"in practice we found it difficult to avoid competition between the diffusion and reconstruction loss\". I would like to see more analysis on the difference between training the model from scratch and from the pretrained embedding space since Li et al. did start the diffusion without pretrained embedding. Does the authors find the training not convergent or the final model quality worse?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject