Latent Diffusion for Language Generation
Authors: Justin Lovelace
Varsha Kishore
Chao Wan
Eliot Seo Shekhtman
Kilian Q Weinberger
Venue: NeurIPS 2023 poster (2023)
Keywords: diffusion
language generation
TL;DR: We demonstrate that latent diffusion is effective for language generation.

Abstract: Diffusion models have achieved great success in modeling continuous data modalities such as images, audio, and video, but have seen limited use in discrete domains such as language. Recent attempts to adapt diffusion to language have presented diffusion as an alternative to existing pretrained language models. We view diffusion and existing language models as complementary. We demonstrate that encoder-decoder language models can be utilized to efficiently learn high-quality language autoencoders. We then demonstrate that continuous diffusion models can be learned in the latent space of the language autoencoder, enabling us to sample continuous latent representations that can be decoded into natural language with the pretrained decoder. We validate the effectiveness of our approach for unconditional, class-conditional, and sequence-to-sequence language generation. We demonstrate across multiple diverse data sets that our latent language diffusion models are significantly more effective than previous diffusion language models. Our code is available at \url{https://github.com/justinlovelace/latent-diffusion-for-language}.

Latent Diffusion for Language Generation
Justin Lovelace
Varsha Kishore
Chao Wan Cornell University, Ithaca, NY
Eliot Shekhtman
Kilian Q. Weinberger
Abstract
Diffusion models have achieved great success in modeling continuous data modali- ties such as images, audio, and video, but have seen limited use discrete domains such as language. Recent attempts to adapt diffusion to language have presented diffusion as an alternative to existing pretrained language models. We view dif- fusion and existing language models complementary. We demonstrate that encoder-decoder language models can be utilized to efficiently learn high-quality language autoencoders. We then demonstrate that continuous diffusion models can be learned in the latent space the language autoencoder, enabling us to sample continuous latent representations that can be decoded into natural language with the pretrained decoder. We validate the effectiveness of our approach for unconditional, conditiool,, and sequence-to to-eeuecee language generation. We demonstrate across multiple diverse data sets that our latent language diffu- sion models are significantly more effective than previous diffusion language models. Our code is available at /https://githu: com/justinlovelacee latent-diffusionn-oon -anguage
Introduction
Although originally introduced by Sohl-Dickstein et al. 61 in 2015, diffusion models did not see widespread use until Ho al. [2 demonstrated their viability for high-quality image generation in 2020. Since then, research has driven rapid improvements and they have recently surpassed generative adversarial networks on image generation benchmarks [12] and autoregressive models on density estimation benchmarks [30], outclassing generative modeling paradigms that have dominated those areas for the better part of a decade. Diffusion models are now, arguably, the most widely used class of generative models for continuous data modalities such as images, audio, and video [54] 32 23
The widespread success of diffusion models across a variety domains and applications makes them appealing for language generation. However, they have seen less use in discrete domains, where the gradual transition of discrete states to Gaussian noise (and vice versa) is not as natural as continuous domains. Prior work proposes to learn continuous diffusion models in the space of learnable word embeddings and decodes the continuous generations with a rounding step [36]. However, combining representation learning with the diffusion objective requires careful regularization to avoid collapse.
One breakthrough in image generation was the introduction of latent diffusion [1T], where diffusion models are trained to produce samples from the latent distribution of a pretrained autoencoder. This offloads the task of generating high-frequenc details to the autoencoder and enables the diffusion process to focus on the high-level semantics of images. In this paper, we explore the viability of latent diffusion for text generation. We claim that this approach is particularly well-suited for discrete modalities because it offloads the challenge of modeling a discrete distribution to the autoencoder and simplifies the diffusion process by restricting
the continuous, latent feature space.
We introduce Latent Diffusion for Language Generation (LD4LG), a method that leverages the latent space of a pretrained encoder-decoder network (e.g. BART 35), T5 50 learn high-quality
"Conespondence to @13353@cornell.ed
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

diffusion model for text. The latent representations from such models are high- seth-dmmmesionll and input-length dependent complicating the use of diffusion models [5] 66]. To address both issues, we learn an additional compression module that maps the high-dimensional encoder representations to lwver-dimensionl fixed-length representation. We also learn a corresponding reconstruction network to map these fixed-length features back to high dimensional features that guide the language decoder (via (ross-attention) to reconstruct the original language.
The low-dimensional representation is ideally suited for diffusion. For language generation, we use diffusion model to generate liwddimnniionll (fixed-length) latent, which is mapped into higher dimensional space with the reconstruction network. This high dimensional representation then guides the pre-trained decoder to generate natural language. Our approach naturally combines the continuous, fixed -eeetlength diffusion process with discrete, variable length text generation.
We demonstrate that LDDLL is effective for unconditional, class-conditional. and sequence-to- sequence language generation across a variety of datasets. Our approach significantly outperforms recent diffusion language models while using fewer sampling steps. For instance, we achieve MAUVE score [47] of .716 for the ROCStories dataset with 250 sampling steps while Diffusion-
LM 336 achieves MAUVE score of .043 using 2000 sampling timesteps. For the challenging XSum summarization benchmark, we achieve a ROUGE-L of 31.9 with 250 timesteps while the recently proposed DiffuSeq [16] achieves ROUGE-L of 14.1 with 2000 timesteps. We also find that the diffusion models offer some benefits over strong autoregressive baseline. In particular, we observe that our latent language diffusion less susceptible memorization and more effective for class-conditional generation.
2 Background
class of latent variable models that learn to iteratively transform Diffusion models 22 63
random Gaussian noise, which can be sampled analytically, to a sample from an unknown data distribution specified by collection of samples. This mapping is defined through forward diffusion process that iteratively adds Gaussian noise to samples, and a generative process that iteratively "denoises" samples from the Gaussian distribution obtain samples from the data distribution. We provide a formal description of diffusion models in the appendix.
The diffusion model consists of denoising network Xo trained with regression objective
[()) Ex,x, Mel (allllooaaaa + vI aye, x|12]
where x is the training data, t ~~ (0 1) is the timestep, (O, 1) is Gaussian noise, a defines the noise schedule, and A is a time-dependent weighting term. The denoising network is therefore trained to denoise a noisy latent, &e / + VI ate, to the clean data, x, with a regression objective that emphasizes certain times t. Sampling algorithms start from pure Gaussian noise, 21 (0, 1), and utilize the denoising network to iteratively generate latents Z; Zta .... Zt, where 1 t > t2 > ... > tr O, with decreasing levels of noise until Zg is drawn approximately from the data distribution.
3 Latent Diffusion For Language
Figure presents overview of Latent Diffusion for Language Generation. Our method consists of two main parts. We augment a pretrained incoder-decode language model with two learnable networks to develop high-quality language autoencoder with compact latent space. We then introduce continuous diffusion models that learn to generate samples from the latent distribution of our language autoencoders. These continuous samples can, by design, be decoded into natural language.
3.1 Language Autoencoder
We base our architecture on pretrained encoder-decoder language models (depicted in blue), such as BART (35) and T5 (50 (we present results with both). By default, we freeze the tre-tr models and learn only the autoencoding modules to accelerate training. The Language Encoder, E(-), maps variable-length language, represented sequence of tokens, w N4, latent representation of the same length, E E 4xdu.

---

Forward Diffusion Process
Prstrained
Gaussian
Auteencoding
Language
Doo
OD0
Networks
Encoder
Latent Ditlusion
Compression Network
B(w)
Generative Diffusion Process
Geussian
%a(-)
xe(-)
9c(-)
D(-)
Language
Decoder
Denoising Network
Reconstruction Network
(STTAR]
Figure Overview our proposed latent language diffusion framework.
Compression Network. The learnable Compression Network maps the encoder features to a com- pact latent space that is well-suited for diffusion. We adopt the Perceiver Resampler [2] architecture, originally developed to compress image features for vinion-language model, which is depicted in Figure The Perceiver Resampler, like the transformer, consists of a stack of alternating multi-hes head attention (MHA) blocks and feedforward (FF) layers. We refer the reader to Vaswani et al. [69 for detailed description of these components. We learn É latent queries Z E Redxxx that iteratively cross-attend to the language encoder features E(w) E R RLxdux to extract information. We follow Alayrac al. [2] and allow the latent queries simultancously attend to themselves and the frozen encoder representations. We can write the attention layer as
Z = Z + MHA(q = K,kv [(:)(w)])
where MHA(-) is the multi-head attention operation with queries, q. and keys/values, kv. This design compresses the encoder representations to fixed sequence length, e, of the latents. After each multi-head attention layer, feedforward layer is applied to the latent query representations.
After the Compression Network maps the input to a fixed sequence length, we reduce the dimensionality of the output to dimension due with learnable linear projection. The compression network therefore maps the variable length output of the frozen encoder to a compact latent space
fo(-)
Learnable Latents
x
MHA
E(w)
x = (f(Www) E Rxdd
Compression Network
(E(())
of fixed length e L and dimensionality due dLM where we will learn our diffusion model.
Figure 2: Architecture of our Compression Network.
To ensure that the latent space is appropriately scaled for diffusion, we can optionally constrain the norm of the latent space. Since (o,)) [|]] due (.]
we can normalize the latent vectors along the feature dimension so that ||x;1 due similar to prior work on text diffusion [33.
Reconstruction Network. The Reconstruction Network maps the compressed latent space the fea- ture space expected by the Language Decoder. To achieve this, we project x = fe(EWw)) e R fx back up to dimension d_M, add learnable absolute position embeddings, and pass through standard transformer model to obtain features g, (x) E R ExNN
The Language Decoder, D(-), cross-attends We train the compression and reconstruction networks to produce features that guide the decoder to reconstruct the input text
these features and generates text autoregressively.
(x)) D(g.(Jo(B(w)))
with the cross-entropy loss. This gives us a continuous, semantic latent space that can be decoded to natural language.

---

Implementation Details. We utilize BART-base and FLAN-T5-baal [TO] as the encoder-decoder language models throughout this work and learn language autoencoders for each dataset. During autoencoder training, we freeze the pre-trained language models and only learn the autoencoding modules. The autoencoder training could likely be amortized across datasets by training a general- purpose language autoencoder on large corpus of text, but we leave such explorations to future work. We train the autoencoder to reconstruct the input language with the cross-entropy loss. For the diffusion latent space, we set é = 32, dae 64 and utilize 3 layers in both autoencoding modules across all monolingual datasets.
For our machine translation experiments, we utilize MT5-base [72] to develop our autoencoder. We found it beneficial to jointly fine-tune the language model and the autoencoding modules, likely because the dataset is an order of magnitude larger than our other datasets and therefore benefits from the additional capacity. We use the same latent dimensionality, but only use a single layer for the autoencoding modules. We report full hyperparameter settings in the appendix. We constrain the norm of the latent space across models and datasets except when using FLAN-T5 because it led to a minor degradation in autoencoding performance and downstream generation quality.
3.2 Latent Language Diffusion
Figure 1] ooutlines our latent language diffusion framework. Given some dataset of natural language. D. we can now sample continuous data as x = fo(EEw) E fxxd where w ~ D. We then train a continuous denoising network, xt to recover x with the standard regression objective
(() E Ex,,, [Arl No (Vaix v1 are, () -x1
with some time-dependent weighting A In practice, the denoising network is often parameterized as e-prediction network 22 v-prediction network (57) where the velocity, v, is defined as v= VO,E vi X2x. These parameterizations can be interpreted as different weighting functions, Ae, for the regression objective above (see Salimans and Ho [57]. We adopt the v-parameterization in this work because it has been shown to be effective for latent image diffusion [51].
For generation, we sample latent variable, 21 R exd (O, I). that is iteratively denoised to produce a sample, x = Zo. from the distribution of the language autoencoder's latent space. We then generate natural language with the pretrained reconstruction network and language decoder w
(ggs (x)). We train our diffusion models with the cosine noise schedule a, = c05 by default. For our machine translation experiments, we employ a scaled cosine noise schedule (see subsection E.2 in the appendix for full details) 7 27). For generation, we use the DDPM sampler with 250 sampling timesteps. For text generation with the pretrained decoder, utilize beam search with 4 beams. We train all of our diffusion models with a single Nvidia A6000 GPU except for the machine translation models which are trained with 4 Nvidia A6000 GPUs.
45
17 55
Denoising Network Architecture. Our denoising network, Ro(2,, L), is a pre-LayerNorm trans- former 669 70 with 12 layers and dimension of 768. We utilize learnable absolute positional [59]. Bao et al. [4] adapted transformers to image diffusion and found that dense connections [28) between early and late layers are beneficial due to the dense nature of the denoising objective. We adopt this modification to improve the suitability of the transformer for diffusion. The autoencoder latent is projected the transformer dimension, processed by the transformer, and then projected back to dimensionality of the autoencoder latent to obtain the final prediction. Following prior work 6 56 Z] we utilize a-conditioning to condition the model on the level of noise. We map a, to a sinusoidal positional embedding [69] and pass it through an MLP with a single hidden layer to obtain time embedding. We add this time embedding to the input sequence and apply adaptive layer normalization [6 conditioned on the time embedding to the output of every feedfoward layer.
encodings and GeGL.U activations
Self-Conditionnng We utilize the self-conditionian technique introduced by Chen et al. S which has been shown to improve the quality of diffusion models 67). The denoising network typically conditioned on the latent variable and the current timestep as X, = tt) Self-conditioning proposes to condition the network on its estimate of the data from the previous timestep, & > L, to improve the prediction at the current timestep X, = *, l, x.). During inference, the sampling procedure is inherently iterative and at time I, we have already computed the output of the denoising network for the previous step. Therefore, it does not require any additional applications of the

---

network. We must, however, modify the training procedure so that the denoising network learns to utilize the estimate of the data, and we must define the inference behavior for the first timestep.
For each training step, we sample some time t ~ 4{ 1)) as before. With probability provide any estimate of the data for self-conditioning, denoted X Sy 2,, I, 0). With probability 1 1-p. p. however, we mimic the inference behavior by first computing X,,0 X (Ze, i, 0) and then computing an additional estimate x (z E, sg (g(x)))) where sg() the stop-gradient operation. This second estimate is then used to compute the loss. We follow Chen et al. and set p 0.5.
we do not
This training procedure also maintains the capacity for inference without self- ceff-condiioning which utilized to generate the first estimate during sampling. We condition on the previous estimate by concatenating it with the noisy latent along the feature dimension. When the previous estimate is not provided, we concatenate a learnable embedding with the noisy latent.
Mass-Conditional Diffusion. For conditionl diffusion, we have some dataset where each natural language utterance associated with one of C' class labels representing, for example, the topic of the text. We condition the denoising network on the class label, y, during training, X, 8a (Zr; y)- We replace the ground truth class label, yi, with null label, ya, with probability p = 0.1 to maintain choose some class y to guide the
the capacity for unconditional generation. At inference time, sampling process to generate text from the specified class. We condition on class labels by introducing learnable embeddings for all labels, including the null label, and add it to the time embedding.
Sequence-to-Sequence Diffusion. Given some seq2seq dataset consisting of source-target language pairs (Wa.. Wing) ~ D, we condition our denoising network on the source sequence and generate the target latent Xirg = fa(EE( (E(wng)). For news summarization, for instance, we generate latent representation of the summary by conditioning the network on the article text. To achieve this, we introduce a cross-attention layer after every self-attention layer the denoising network that attends to features from frozen language encoder.
In general, we can incorporate any language encoder, E-c extract features from the source text. By default, we use the same pretrained encoder used for our language autoencoder. For our machine translation experiments, we condition our latent diffusion models on representations from frozen MT5-XL encoder, which we found to be more effective than MT5-base representations. Therefore, given a sample from our seq2seq dataset, (Ware: Wug) D. we can compute Xug (e(()))))) and use modified seq2seq diffusion objective
[(0) (w........)).. ( + VI aje, l, Exse (Ware) Xug 112)--
We also utilize classifier-free guidance [2 to improve sample quality. We jointly learn an uncondi- tional network, & (z,,,, t), and a conditional network, (z t, ((Wsc)), by dropping the conditioning information with probability p- = 0.1 during training. When we drop the conditioning information, we cross-attent to a learnable embedding instead of the embedded source text. During sampling, we use guidance weight w and compute the prediction as
X wx (ze, 1, E(wwe)) + (1 - w))x(21,1
Setting w = 1.0 corresponds to the conditional diffusion model while setting w > 1.0 strengthens the influence of the conditioning information. We use 2.0 for the seq2seq tasks and ablate this choice in section 5
We can also generate multiple outputs S for each input by sampling different latents z; ~N I). We then select the most promising candidate with Minimum Bayes Risk (MBR) Decoding [15 In MBR decoding, we define a loss function L. such as the negative Rouge, and use it to select a candidate WMBR argmin WES [S] Ew'es L(W, w'). In our experiments, we use |S] 5 and denote the results from using MBR decoding as MBR-5, We also report results using the ground truth to select the best candidate Woracle argmin wes[(w, Way) provide an upper bound on the performance of our method given optimal sample selection. Because this requires knowledge of the ground-truth target text, we refer to this Oracle sampling.
4 Datasets
We evaluate I.DLLG on a variety of natural language datasets. ROCStories [42 a corpus of 98k five-sentence commonsense stories, that capture casual and temporal relations. The AG News Topic
5

---

Classification [60] dataset consists of news articles across four topics: World, Sports, Business, Sci/Tech with article titles and descriptions from 120k training instances. We focus on generating the article descriptions in this work. The XSum dataset consists of BBC articles from 2010 to 2017 covering wide range of topies (e.g., News, Politics, Sports, etc.). The training split has 204k instances and each example contains document and a summary. The QQP 9 dataset consists of 400k question pairs, where each example is two similar questions and a binary value indicating whether the two questions have the same meaning. The WMT 2014 English-Germat [S] dataset a widely used machine translation dataset consisting of roughly 4.5 million sentence pairs. We present detailed dataset statistics in the appendix.
4.1 Evaluation Metrics.
We use MAUVE Score 47 and Perplexity (Ppl) to evaluate the quality of our generated text. MAUVE Score metric for open-ended text generation that compares the distribution of generated text with that of reference text using divergence frontiers. We follow Pillutla et al. [47] and use the GPT-2-Large model 49 to embed the text. Perplexity measures how likely the generated samples are according to autoregressive language model; we use GPT-2-Large to compute perplexity.
We also want to quantify the Diversity (Div) of generations. We define diversity as
Div = II Jurique gramst (w.))) where {w:} } is set of generated samples [68]. The metrics dis- total o-gramst( w.
cussed so far can be optimized by generating samples from the training set. We measure the proportion of generated 4-grams that are found in the training set quantify the degree of Memorization (Mem).
To evaluate the performance for monolingual seq2seq language generation tasks, we utilize Rouge (37 and BERTSccre (75]. Rouge-1/2 1/2 measures the number of unigrams/bigrams in the refer- ence that appear in the generated text and Rouge-L measures the longest common sequence between the texts. BERTScore uses contextual embeddings from pretrained language model to measure the similarity between texts. We follow prior work and use the mi crosoft/ boosof/debbbrr deberra-aaee aalalaree--aaa mnli model [18] to extract contextual embeddings. For machine translation experiments, report SacreBLEU scores 5 to ensure fair comparison with prior work.
For our unconditional and class-conditional language generation experiments, we sample 1000 instances from the diffusion model. For the MAUVE reference text, we sample 1000 instances from the test set. We repeat this 5 times and report the mean and standard deviation meangtdev We also compute reference values for our metries with natural samples from the test set. The reference MAUVE, for instance, is computed between 1000 train and 1000 test samples. Qualitative samples from our models are in the supplemental materials.
5 Experiments
5.1 Language Autvencoder
We evaluate the effectiveness of our proposed language autoencoder using heldout examples from our datasets. As point of comparison, we also evaluate the default behavior of the language models that we use to develop the language autoencoders. A consequence of BART's particular denoising objective is that the pretrained model already generates a copy of the input language, although this is not true of other models such as T5 or FLAN-T5.
We present results for our two most complex datasets, ROCStories and AG News, in Table I and present the results for XSum, QQP, and WMT14-En-De, which show similar trends, in the appendix. We observe that our BART-base autoencoder is able to compress the feature space by a factor of 24x while improving the fidelity of the reconstructions. Our autoencoding modules are also effective at converting the pretrained FLAN-TS into a language autoencoder, even though that is different from the model's default behavior. Across both models and all datasets, language autoencoders able to achieve near-perfect reconstruction with a low-dimensional latent space.
5.2 Unconditonal Language Generation
Baselines. We evaluate our approach's 's capacity for unconditional language generation with the ROCStories and AG News datascts. We compare against the recently proposed Riffusion-LMM model

---

Table 1: Effectiveness of Language Autoencoder
Method
Latent Dimensions
Hidden Units
RocStories
AG News
Rouge-1/22 1/2/L.
BLEU
Mouuge-1/211
BLEU
BART-Basc
L 768
49.1252
2//9//
97.5
99.6/99.4/99.6
98.6
BART-Base Autnencoder
32 64
2048
99.2/98.5/99.22
97.6
99.7/99.4/99.7
98.8
FLAN-TS- Basc
L 768
$ 49.152
21.5/11.8/19.4
0.7
63.6/53.0/59.6
42.3
FLAN-TS-Base Autoencoder
2048
98.4//6.9/98.4
95.8
199./99.3/99.
96.8
36.. We also fine-tune the pretrained GPT-2-Medium model, which roughly 1.6x larger than our
denoising network, strong autoregressive baseline [49]. For sampling from GPT-2, we prompt with BOS token and utilize nucleus sampling (p 0.95) 24 We explore different sampling configurations in the appendix and find that they lead to similar conclusions.
We observe that our approach is significantly more effective than Diffusion-LM at modeling language distributions, demonstrated by the higher MAUVE scores, while requiring fewer sampling steps. Diffusion-L.M is unable to model diverse language distributions and exhibits poor diversity. Utilizing high quality latent spaces from pretrained language models improves the effectiveness of our diffusion model. We observe that both language models are highly effective for the AG News dataset, but using BART- leads to stronger MAUVE score for the ROCStories dataset. Across both datasets, FLAN-TS-base produces more diverse generations and exhibits less memorization.
Results. We present this comparison in Table 2
While GPT-2 generally achieves strong language generation metrics, it is more susceptible to mem- orization than LD4LG. For the AG News dataset, GPT-2 exhibits significant memorization and lower MAUVE score. We do find that GPT-2 samples have lower perplexity. However, measuring perplexity with pretrained GPT-2 model likely biases the metric towards the fine-tuned GPT-2 model. Moreover, MAUVE scores have a stronger correlation with human judgments of quality [47
Table 2: Unconditional Language Generation Evaluation. The fine-tuned language model is presented in gray.
ROCStories
News
Timesteps
MAUVE
Ppl
Div
Mem J
MAUVE
Ppl.j
Div
Mem
Reference
951.co
21.1.
.114.ca
.32 o0s
.951 o14
43.61.a
.658 o02
.385.0x
Diflusion-1 LM 36
2000
043..oo
47.3.
.128 oca
.434.om2
.012 001
67.11.2
.U43.on2
0886.
LD4LG (BART Base)
.716.119
30.6.
.331 acs
.441.0us
.866.0u
.540.0m
.203
LD4LG (FL
250
.88
37.5.
.389 aca
.377
.50
122.0a.
.024.00
.21
GPT-2-Modium
.788 025
20.0.2
.372 oca
688 o0o
8200.012
37.31.
.532.17
829 220
Benefits of Compression. Because the pretrained BART model already copies the input text, we can ablate the impact of learning a compact latent space by learning diffusion model directly in the encoder feature space. One complication of this setting is that the sequence length of the BART features vary. During training, the sequence length simply determined by sample. During generation, however, we must specify the length. To determine the sequence length for generation, we opt to sample a length from the empirical distribution of lengths the training set. We refer to this bascline
AART-Diffusion and outline full implementation details in the appendix.
We quantify the speedup by measuring how long it takes each approach to match the peak validation MAUVE of BART- Diffusion. We observe that learning a compact latent space is beneficial both terms of absolute performance and wall-clock time, reaching the peak MAUVE of BART-diifusion in quarter of the time. Compressing the latent space along the sequence dimension significantly reduces the overhead per iteration due to the quadratic cost of self-attention, and we also observe faster convergence.
We compare BART-Diffusion with our proposed approach in Table 3
Self-conditioning.
We find that significantly improves the MAUVE score and the perplexity of the generated text, but sacrifices some diversity.
We ablate the impact of self-conditioning Table 4
7

---

Table 3: Benefits of Compression (ROCStories)
Hidden Units
Relative Speedup
MAUVE t
Ppl
Div t
Mem 4
BART-Diffusion ID4LG (BART-base)
S 49,152
1.0x
.605.00 .716,019
16.8,7
.14.
.304,00 oo3
2048
3.86x
30.6,5
.311 .005
.441.00 000
Table 4: Impact of Self-Conditioning (ROCStories)
MAUVE1
Ppl +
Div f
Mem
LD4L.G (BART-basc)
.716.019
30.6.s
331..oo
.441.oo 0004
Self-cond.
.8000088
79.31.0
.427.004
.299.003
Table 5: Metrics for class-conditional generation.
LD4LG (BART-hase)
LD4L G (FL.N-TT--hss))
MAUVE1 f
Mem
MAUVE 1
Mem L
Conditioning
World
Sports
Business
Soi/Toch
World
Sports
Business
Sci/Toch
World
842..11
0055.002
o02
.220.oo2
.2966
.80,,sa
.013,00
on2
U22.002
.2333.00
Diflusion
Sports
013.00l
845,024
.011
.010.00
.305
.011
836
.009 000
.079,000
.219
Business
021.002
011.o01
.752 ceo
.068 005
363 o00
.011.001
765 016
0766.00
.214
Sci/Tech
0233.mz m01
.12..0m
.822 ous
.813.ss
.25 mm4
.024.uo
.01 1.00
.082 o1o
843.00
.169 us4
Conditional GPT-2
Reference
Conditioning
MAUVE
Mcm
MAUVE f
Mcm |
World
Sports
Business
Sci/Toch
World
Sports
Business
Sci/Tech
Comparsons
World
805,022
.012 .c0
.025.002
.021,02
4022.00
963..oo
oc1
.032.oos
.388 .00
Sports
.177.00
.810 oLo
.012.001
.013.001
369 .00
.018..01
.988
.011.001
.014.oo 002
.316
Business
.07 oos
.012 001
629 029
069
.79.o0r omr
_040 oos
.014 oc
968 p09
.125 o00
.411 oo0
Sci/Toch
.033 002
.013 001
1122.0i
097 ozr
434,004
.036.os
.016.0x
.133,om o1a
.955.11
.366 oms
5.3
Class-Conditonal Language Generation
Baselines. Conditional training with control tokens is one the most widely used methods for controlling autoregressive models [14 22 40 33). We prepend the class label to each sample as control token and fine-tune GPT-2-medium for class-conditional generation. Because memorizing the training instances associated with each class is trivial solution, terminate training when the model's 's memorization exceeds the reference values.
Results. We evaluate the effectiveness of class-conditioning with the AG News topic classification dataset. We sample instances for each class and compute the MAUVE scores between natural instances from each class. We report these metrics in 3] We observe that the MAUVE scores are highest when the conditioning and ground-trutt labels are aligned across all methods, demonstrating that the label guides the generation effectively. We observe that our approach is more consistently effective at class-conditional generation, particularly for the two most similar classes, business and sci/tech. The GPT-2 baseline is again more susceptible to memorization than our approach.
5.4 Sequeence-o-Seqqence Language Generation
Baselines. We compare against directly fine-tuning BART-base and FLAN-T5-base on the XSum summarization and QQP paraphrasing datasets. For diffusion baselines, we compare against the following continuous diffusion models learned in the space of word embeddings: DiffuSeq [16]. CDCD [3] DINOISER (73), and GENIE [38]. We also compare against the following discrete diffusion models which learn to invert discrete corruption processes (e.g. masking): Reparameterized Discrete Diffusion (RDM) 76 and DiffusionBERT [19 We compare directly against the metrics reported in prior work on our datasets. For XSum, we additionally train DiffuSeq model using the official implementation. We note that Gong et al. [66 typically train their models much longer than ours. The XSum DiffuSeq model, for instance, is trained for over 3 x more epochs than our approach.
8

---

For machine translation, we compare directly against the prior work that reported SacreBLEU scores ensure fair comparison 48
Table 6: Seq2Seq Evaluation QQP. Results from fine-tuned language models are in gray.
Table 7: Seq2Seq Evaluation on XSum. Results
Method
Sampling
Rouge- 1/27- 1
RFRTScom
from fine-tuned language models are in gray.
DifisSoy
Ronden
55.2/2922/52 2/527
62.4
Methed
Sampling
Reage-/M.. 1MM
BFRTSoae
RDM (DM-malliommill
TISI (BART-)aw))
LDALG (TLAN T3 Base)
RDMMaalocsbaig
Random
Randon
Random
62. 1/38.4/9.1
62.6/990696.1
157.
157.3
85.8
83.7
85.8
LD4LG -FLAN-T--BsS
LDAEG (AKT bak))
DiffuSey [B
Random
Raniom
Random
376/15.500088
361/155931.2
1890 3/1366
74,8
46.8
74.1
RDM muttinonial
DiffiSeq
RDM absorbing
78
76
MBR 10
MBR-11 10
MBR-10
159.5
159.5
158.8
83.7
$4.7
84.7
LD410 (RAKE)ce))
DufluSeg 16
LD4LG (FLAN TS Base)
MBR-5
MBR 5
MBR-5 5
38.7/166//119
982/16.//91.5
19.301.//1411
75.2
46,9
74.5
DiflssiBBRR 19
MBR-10
158.9
DiffuSeq 16
Ondee-5
23.52 23552.3//8..
47.9
LDSLG (BART-luse)
MBR-5
63.3:40.36111
GENIE 39
Ondle
37.3/15.3/29.4
MBR-5
GENTE of pepersining
15
Oncck-5
41.2/19.1/38
DifliSoy
Oraclo-5
67.4/4 965588
63.7
LD4LG (FLAN-T5-8)) T5-B
LDKLG MARRTbag
Oncle
Oncle-5
42/4/194/36.4
430/20.0:7.22
76.1
75.3
LDSLG (BART-lae)
Orade-55
68884666666.1
Oracle-S
BART Base
Naclous
35.1/13.2/27-7
73.1
BART-RIsG
Nacles
$1.528 1448.3
79.9
FLAN-TS-RENc
Nacless
34.6/12.9/2.2
72.7
BLAN-T5-Birc
Naclau
35.0/301/22.
FLAN- Base
Beam
39.2417,0/9233
35.3
BART-Bise
Beam
61 9199999599.
85.5
BLAN-T5-Base
Beam
63.0/40 18935
86.2
Results. We present our comparison on QQP and XSum in Table and Table Table 7. Our approach signifi- cantly outperforms recent diffusion language models across both datasets, especially for the more challeng- ing XSum dataset. For instance, DiffuSeq is reason- ably effective for QQP, but it struggles with XSum and fails to generate coherent text (see samples in appendix). Our method, on the other hand, is compet- itive with fine-tuning. L.4LGG narrowly outperforms fine-tuning on QQP with MBR decoding, but the fine- tuned models are slightly more effective on the XSum dataset. Across both datasets, LD4LG with oracle sampling outperforms all approaches (including di- rect fine-tuning methods) with just 5 random samples.
Table 8: Machine translation results WMT14-En-Dec Baseline results are from 73
Method
Sampling
SacreBLEEU
De-+En
CDCD [R]
Random
19.3
24.9
LD4LG (MIS-boe base)
Random
21.4
26.2
Diffusion-LMM 36
MBR-5
15.3
17.3
CDCD D3
MBR-10
19.7
25.4
DINOISER 1+
MBR-5
24.3
28.8
L.4LG (MTS-base)
MBR-5
27.0
This demonstrates that LD4LG has good coverage,
but MBR decoding does not consistently identify the best candidate. In our experiments. we use classifier-free guidance with guidance strength samples in Figure 3 and observe that such guidance meaningfully improves performance.
2.0. We ablate this choice with validation
We report our machine translation results in Table 8 We observe that LD4LG outperforms the Diffusion-LMN and CDCD baselines although it lags behind the DINOISER baseline. This demonstrates that our method can effec- tively take advantage of strong pre- pre-trained multilingual language models for effective multilingual generation.
Guidence (XSum)
18.5
Lotie mus
35.3
6
Future Work
Our experiments demonstrate that latent language diffu- sion models can generate high-quality natural language in variety of settings. In continuous domains, diffusion models are remarkably effective for applications ranging from image editing [41] to solving inverse problems [64]. We are excited explore the potential applications en- abled by effective language diffusion models. We expect that L.4L.G G is natural fit for applications such as lan- guage editing (e.g. style transfer) and controllable generation (e.g. mitigating toxicity).
Strength
Figure 3: Ablation of classifier-free guid- ance on the XSum summarization bench- mark.

---

Despite achieving good performance, LD4LG has important limitations. Sampling from diffusion models slow due the iterative generative process. LD4LG improves upon some prior continuous text diffusion models (that use 2000 steps) and only uses 250 sampling steps. However, speeding up the inference process of diffusion models an active area of research and techniques developed for image diffusion can likely be adapted for LD4LG [65 [5]] Song et al. [65]. for instance, distilled single step. We leave the
a trained image diffusion model to produce high-quality samples extension of such techniques to language generation future work. In subsectiio 5.4, we observe that diffusion models have excellent coverage but MBR decoding fails to identify the best candidate: developing improved sampling procedures candidate re-ranking methods would likely improve performance for tasks such as summarization and machine translation.
7 Related Work
Diffusion models. Diffusion models 61 22 are a class of generative models that have led to impressive results in image synthesis, recently surpassing Generative Adversarial Networks [7 12] These models typically operate directly in pixel-space, learning a distribution over images. Rombach al. [52] introduced latent diffusion for image synthesis and demonstrated that they can be learned in the latent space of a pretrained autoencoder, Latent diffusion has since been successful in other domains such as audio synthesis (32] symbolic music generation (58). and molecule generation 771.
Diffusion for Language. Prior work has focused on directly modeling discrete data by designing diffusion processes for discrete state spaces [25] 26]. Li et al. (36) train a continuous diffusion model in the space of token embeddings that are learned jointly with the denoising objective and decode generations with a rounding step. Strudel et al. [67] scaled up this approach and instead learn the diffusion model in the space of pretrained word embeddings and find that low-dimensional embeddings are better suited for diffusion. Gong et al. [16] extend Diffusion-L.M [36] to sequence-to sequence tasks by concatenating the source and target sequence and only performing diffusion for the target sequence. Chen et al. g map words to arbitrary binary strings, represented as sequence of real numbers. They then train continuous diffusion model and round the generated sequences to produce binary strings. The authors also introduce self-conditioning, which we adopt for our method.
8 Conclusion
In this work, we demonstrate that latent diffusion is an effective paradigm for language generation. To achieve this, we introduce a method for compressing the high-dimensional, variable-length language representations from pre-trained language models into compact, fixed-size latent representation that can be decoded into natural language. This compact latent representation is, by design, well-suited for learning continuous latent diffusion models. Our latent language diffusion models are effective cans-conniiional, and soguence-to-squenee language generation. They offer some benefits over fine-tuned auto-regressive language models and significantly outperform recent diffusion language models across a variety datasets.
for unconditional,
Acknowledgements
This research is supported by grants from the National Science Foundation NSF IIS- 1724822,, HDR-21183 10). The Cornell Center for Materials Research with funding from the NSF MRSEC program (DMR-1719875), DARPA, arXiv, LinkedIn, and the New York Presbyterian Hospital.
and
References
[1] Chi-square Distribution, pages 70-72. Springer New York, New York, NY, 2008. ISBN 978- 0-387-32833-1. doi: 10. 00.10007978--0388--83333 3000--38773233--1--103133333333-
URL (RI.H https:://oi. http://ii.../1111....1..100....0 1007/
[2] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Lue, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: visual
10

---

language model for few-shot learning. Advances in Neural Information Processing Systems, 35: 23716-23736, 2022.
[3] Jacob Austin, Daniel D. Johnson, Jonathan Ho, Daniel Tarlow, and Rianne den Berg. Structured denoising diffusion models in discrete spacess In A. Beygekzimer, Y. Dauphin, Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, net/ pett/forumiid=h// -xxxPAA
2021. URL Wtw....t.t//////ttt
[4] Fan Bao, Shen Nie, Kaiwen Xue, Yue Cao, Chongxuan Li, Hang Su, and Jun Zhu. All are worth words: A vit backbone for diffusion models. In CVPR, 2023.
[5] Ondrej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Leveling, Christof Monz, Pavel Pecina, Matt Post, Herve Saint-Amand, Radu Soricut, Lucia Specia, and Ale s Tamchyna. Findings of the 2014 workshop on statistical machine translation. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 12-58, Baltimore. Maryland, USA. June 2014. Association for Computational Linguistics. URL http: //www
aclweb. org/ anthalogytyaaaaoaaoaooooott /ttp/////1/// 21//////W14-3300
[6] Nanxin Chen, Yu Zhang, Heiga Zen, Ron J Weiss, Mohammad Norouzi, and William Chan. Wavegrad: Estimating gradients for waveform generation. In International Conference on Learn- ing Representations, 2021. URL htttpps:/oopereeiiek http////wwwww.wee//eooereeeeeeee wwt.:/eeenn/eeee/ iudummmiiiiii 1d-NsMLjcFaDBB
[7] Ting Chen. On the importance of noise scheduling for diffusion models. arXiv preprint vrXiv:2301.1097. 2023.
[8] Ting Chen, Ruixiang Zhang, and Geoffrey Hinton. Analog bits: Generating discrete data using diffusion models with self-conditioning. arXiv preprint arXiv: 2204002200 2022.
[9] Zihang Chen, Hongbo Zhang, Xiaoji Zhang, and Leqi Zhao. Quora question pairs. 2017.
[10] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph. Yi Tay, William Fedus, Eric Li, Xuezhi Wang. Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finet finetuned language models. arXiv preprint urXiv:22100 10. 1416, 2022.
[11] Mostafa Dehghami. Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, et al. Scaling vision transformers to 22 billion parameters. arXiv preprint arXiv:2302.05442 2023.
[12] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 8780-8794. Cur- wwtpeeps.ccpppprr/////e/ Associates, Inc., 2021. httpshhttts:::pooceeeeee pdf
[13] Sander Dieleman, Laurent Sartran, Arman Roshannai, Nikolay Savinov, Yaroslav Ganin, Pierre H Richemond, Arnaud Doucet, Robin Strudel, Chris Dyer, Conor Durkan, al. Continu- ous diffusion for categorical data. arXiv preprint wrXiv:2211.15085 2022.
[14] Jessica Ficler and Yoav Goldberg. Controlling linguistic style aspects in neural language generation. EMNLP 2017, page 94, 2017.
[15] Vaibhava Goel and William Byrne. Minimum bayes-risk automatic speech recognition. Computer Speech & Language, 14(2):115-135, 2000. ISSN 0885-2308. doi: https://doi.org//1 1006/csla.2000.0133. Utt.Litttee///....eeeedddeeeet
htthh:tt///wwtt wwwencedirect.......... com ctt.///iiieeeee//tiiii////iiiii
$0885230800901388!
[16] Shansan Gong, Mukai Li, Jiangtao Feng, Zhiyong Wu, and Lingpeng Kong. Diffuseq: Sequence to sequence text generation with diffusion models. The Eleventh International Conference on Learning Representations, 2023. URL /ttpps://operreveer wttp:/wwwwwineeeoonn/eeeee jtt://ommmmmdddjj----
rLVXsj
[17] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger, editors, Advances in Neural Information Pro- cessing Systems, volume 27. Curran Associates, Inc., 2014. URL htt.httss: pdf [18] Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Decoding-enhance bert with disentangled attention. In International Conference on Learning Representations, 2021. http::ttt////eerre wttpeeiiiwwweeeerreieeeee www..oorumiid--xPlaaattttto
11

---

[19] Zhengfu He, Tianxiang Sun, Qiong Tang, Kuanning Wang, Xuanjing Huang, and Xipeng Qiu. DiffusionBERT: Improving generative masked language models with diffusion mod- els. In Proceedings of the 61 Annual Meeting of the Association for Computational Lin- guisties (Volume 1: Long Papers), pages 4521 Toronto, Canada, July 2023. Asso- ciation for Computational Linguistics. doi: 10. 11116665/1//222332c-oog222222 URL https
org/2023. //aclanthology. org/ 2023. 2023.ac1-oom. acl-long. 248
[20] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint
arXiv:1606.084155 2016.
[21] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint 1:Xiv:227..1259, 2022.
[22] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffuusion probabilistic models. CoRR, abs/2006. 1 1239, 2020. URL httpttt//axxxii.. www.:oggg//.a.....
11239
[23] Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao. Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.0238 2022.
[24] Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In International Conference on Learning Representations, 2020. URL
https://openreviier wwt. http://rrmmmmdddd00111
[25] Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, and Max Welling, Argmax flows and multinomial diffusion: Towards non-autoregresssvv language models. CoRR, abs/2102.05379, 2021. URI. htttpt://axxiiii org. /bt.////...2. 2102.053799
[26] Emiel Hoogehoom, Alexey A. Gritsenko, Jasmijn Bastings, Ben Poole, Rianne van den Berg, and Tim Salimans. Autoregressive diffusiion models. International Conference on Learning Representations, 2022. URL https:t://ppeeeei..eeeee net/ wtt.:/oorm/ee wwwtorrrnmniiniiiee ?1d=Lm8T39vLDDE
[27] Emiel Hoogeboom, Jonathan Heek, and Tim Salimans. simple diffusion: End-to-end diffusion for high resolution images. arXiv preprint arXiv:2301. 1093, 2023.
(28) Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700-4708, 2017.
[29] Nitish Shirish Keskar, Bryan McCann, Lav Varshney, Caiming Xiong, and Richard Socher. Ctrl: A conditional transformer language model for controllable generation. arXiv preprint arXiv:1909.0588 058588 2019.
[30] Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. Advances in neural information processing systems, 34:21696-2111 21707, 2021.
[31] Diederik P Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. On density estimation with diffusion models. A. Beygelzimer, Y. Dauphin, P. Liang. and Wortman Vaughan, editors. Advances in Neural Information Processing Systems, 2021. URL.
forum?id-2LaBgxcIYy
[32] Zhifeng Kong, Wei Ping, Jiaji Huang. Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. In International Conference on Learning Representations, 2021. URL wtR...ttts::////t/ //operrvveeeeeeeeeeeee net/ wwwprrrmmmmddaaaaaaaeaaae
[33] Tomasz Korbak, Kejian Shi, Angelica Chen, Rasika Bhalerao, Christopher I. Buckley, Ja- son Phang, Samuel R Bowman, and Ethan Perez. Pretraining language models with human preferences. arXiv preprint arXiv:2302.00 085822 2023.
[34] Shankar Kumar and William Byrne Minimum Bayes-risk decoding for statistical machine translation. In Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics: HLT-NAACL 2004, pages 169-176, Boston, Massachusetts, USA, May 2 May 7 2004. Association for Computational Linguistics. URL htttts::/aclaathllllooooooo
org/ 1022
[35] Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension, 2019. URI.
https://arxiv. org/ absa htts//990..1.0 13461
12

---

[36] Xiang Lisa Li, John Thickstun, Ishaan Gulrajani, Percy Liang, and Tatsunori B. Hashimoto. Diffusion-Im improves controllable text generation, 2022.
abs/
2205. 14217
[37] Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81, 2004.
[38] Zhenghao Lin, Yeyun Gong, Yelong Shen, Tong Wu, Zhihao Fan, Chen Lin, Weizhu Chen, and Nan Duan. Text generation with diffussin language models: A pre-training approach with continuous paragraph denoise. 2023.
[39] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019, URL
htrpeerrrriieenee//oouum net/forum?
1d=Bkg6RiCqy7
[40] Ximing Lu, Sean Welleck, Jack Hessel, Liwei Jiang, Lianhui Qin, Peter West, Prithviraj Am- manabrolu, and Yejin Choi. QUARK: Controllable text generation with reinforced unlearning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. URL https: wwwpperreveeeeeeettoooe 5HaIds3ux50
[41] Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. SDEdit: Guided image synthesis and editing with stochastic differential equations. In International Conference on Learning Representations, 2022. URL (ttps:tp::/. //pprreviei
[42] Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. A corpus and cloze evaluation for deeper understanding of commonsense stories. In Proceedings the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pages 839-849, 2016.
[43] Shashi Narayan, Shay Cohen, and Mirella Lapata. Don't give me the details, just the sum- mary! topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1797-1807, Brussels, Belgium, October-Novemble 2018. Association for Computational Linguistics. doi: 10. 1..18653/1/D18. 1206. URL
org/ 1206
[44] Shashi Narayan, Shay B Cohen, and Mirella Lapata. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. arXiv preprint arXiv: /808.08745 2018
[45] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 8162 8171. PMLR, 18-24 Jul 2021. Solpatttss//pprceedigg mlr. press/v139/ nichol
[46] William Peebles and Saining Xie. Scalable diffusion models with transformers. arXiv preprint arXiv:2212.09748, 2022.
[47] Krishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, and Zaid Harchaoui. Mauve: Mcasuring the gap between neural text and human text using divergence frontiers. Advances in Neural Information Processing Systems, 34:4816-4828, 2021.
[48] Matt Post. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186-191, Brussels, Belgium, October 2018. Association for Computational Linguistics. doi: 10. 10.18653/v1/118--3111 URL https://
aclanthology.o .rgg wrgg/11---319
[49] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, al. Language models are unsupervised multitask learners. OpenAl blog. 188):9, 2019.
[50] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485-555) 2020.
13

---

[51] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models, 2021. URL htttps://axxi.. org/
abs/ 20772
[52] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10684-10695, June 2022.
[53] Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. Palette: paaaggett-image diffusion models. In ACM SIGGRAPH 2022 Conference Proceedings, pages 1-10, 2022.
[54] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Raphael Gontijo-opps,, Burcu Karagol Ayan, Tim Salimans, Jonathan Ho, David J. Fleet, and Mohammad Norouzi. Photorealistic rext-to-image diffusion models with deep language understanding. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. URL htttss//openrreviee net//forumidd-0BVe n512A1
[55] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Den- ton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Sali- mans, Jonathan Ho, David J Fleet, and Mohammad Norouzi. Photorealistic text-to- image diffusion models with deep language understanding. hamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neu- ral Information Processing Systems, volume 35, pages 36479-36494. Curran Associates, Inc., 2022. URL httpps::/pprcceednngge www..ps..ccpppp...r.r.e... wwwpppaap__aiiieeeeeeeeeeeeeeeee Saaapeerffleespppe/////20 /pppppp//22///1le//// arnbaaaddabb7d3333333ccroff.o
In S. Koyejo, S. Mo-
Paper- Corr-CCofeeenncee pdf
[56] Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
[57] Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffussin models. In International Conference on Learning Representations, 2022. URL
net//orrum22d-TiIXXII
[58] Flavio Schneider, Zhijing Jin, and Bernhard Schölkopf. Mo!," usai: Text-to-music generation generaatioo with long-context latent diffusion. arXiv preprint arXiv:2301.11757 2023.
[59]
Noam Shazeer. Glu variants improve transformer. arXiv preprint arXiv:2002.05202, 2020.
[60]
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning. Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference empirical methods in natural language processing, pages 1631-1642, 2013.
[61] Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics, 2015. URL https://arxiv org/abs/1503 13585
[62] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. CoRR, abs/2010.02502. 2020.
wrtp:////0aaa..00.../. 02502
[63] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
[64] Yang Song, Liyue Shen, Lei Xing, and Stefano Ermon. Solving inverse problems in medi- cal imaging with score-based generative models. In International Conference on Learning Representations, 2022. URL http:ttts////preeeeeee
pett/forummid-vaRccyy
[65] Yang Song. Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models, 2023.
[66] Robin Strudel, Corentin Tallec, Florent Altché, Yilun Du, Yaroslav Ganin, Arthur Mensch, Will Grathwohl, Nikolay Savinov, Sander Dieleman, Laurent Sifre, et al. Self-conditioned embedding diffusion for text generation. arXiv preprint (.ti::2/11.042.:00 2022.
[67] Robin Strudel, Corentin Tallec, Florent Altché, Yilun Du, Yaroslav Ganin, Arthur Mensch, Will Grathwohl, Nikolay Savinov, Sander Dieleman, Laurent Sifre, and Rémi Leblond. Self- conditioned embedding diffusion for text generation, 2022. https httee://aaxxa //arxiv....////////// www.y.rg/aaao.... /
2211. 04236]
14

---

[68]
Yixuan Su, Tian Lan, Yan Wang, Dani Yogatama, Lingpeng Kong, and Nigel Collier. A contrastive framework for neural text generation. arXiv preprint arXiv:2202.06417, 2022.
[69]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, tukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proceedings of the s1 International Conference Neural Information Processing Systems, NIPS 17, page 6000-6010, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 978 17815I0 10860964.
[70] Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang. and Tieyan Liu. On layer normalization in the transformer architecture. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 105333 PMLR, 13-18 Jul 2020. URL https: procee/d/ooceeenggs/ mlr. prrrs/ v119/xiong20b.htm
[71] Minkai Xu, Alexander Powers, Ron Dror, Stefano Ermon, and Jure Leskovec. Geometric latent diffusion models for 3d molecule generation. arXiv preprint arXiv:2305. 01 01040, 2023.
[72] Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami AI-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mT5: A massively multilingual pre-trained text-to-text trans- former. In Proceedings of the 2021 Conference the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 483-498, Online, June 2021. Association for Computational Linguistics. doi: 10. 20115866/122222 nnacc--miin44 URL hwt.oolloaaootogoo.oooooooooooo. org/ 2021 w.a.aalllminn....
[73] Jiasheng Ye, Zaixiang Zheng, Yu Bao, Lihua Qian, and Mingxuan Wang. Dinoiser: Diffused conditional sequence learning by manipulating noises. arXiv preprint arXiv:2302. 10022202..1025,, 2023.
[74] Biao Zhang and Rico Sennrich. Root mean square layer normalization. Advances in Neural Information Processing Systems, 32, 2019.
[75] Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv: 1904.09675, 2019.
[76] Lin Zheng, Jianbo Yuan, Lei Yu, and Lingpeng Kong. A reparameterized discrete diffusion model for text generation. arXiv preprint arXiv:2302.05737, 2023.
15

---

A Diffusion Models
We present formal description of diffusion 22 62 30). Diffusion models are latent variable models with latents z = {zz]t E [0, }l} that are given by the forward diffusion process g(z x), with the data, X p(x), being drawn from an unknown distribution.
The forward process is a Markovian process that iteratively adds Gaussian noise to the data over time
g Mizz Va,x (1 at) ((zi (N(z varl.Ze: (1 (t(ss
where (ijs 0t aa, and 0 S < S 1. The noise schedule, specified by ae [0, 1]. decreases
with until the final latent becomes approximately Gaussian, p(z1) ~ (Zz1; 0, I) - independent of
the original data. The forward process therefore defines a transition from the data distribution to Gaussian distribution.
Given access the original data x, the forward process can be inverted analytically. For t <>> have
(2, x) NN(uqq s, (1,
where
x, Va,( amja)) Vokele
ats) Zz og(s,t) (1 - a.))) a1asss))))
asiss)
-tt
at
Cke
We utilize this define our generative process. Because is unavailable during generation, train a neural network to approximate the original data given some noisy latent and the timestep, {o(2,,) 1) as The denoising network is trained utilizing regression loss
(()) xx, &o (illoo(arr) (Va,x + /1 a,c, x111
with some time-dependent weighting A This loss function can be motivated as the weighted variational lower bound of the log likelihood of the data under the forward diffusion process [22 31
In practice, the denoising network is often parameterized as an 6-prediction network 22 or v-prediction network [57] where the velocity, v, defined as Vage vI axx These parameterizations can be interpreted as different weighting functions, λ,, for the regression objective We adopt the v-parameterizztioon throughout this work.
[5]]
With a trained denoising network, we define our generative process as
po(ze|zi) N (izz Me s, t), o',
where
(z s,t) (z, X0 (ze, s,t, (s, = Oidss
We therefore substitute our estimate of the clean data into the posterior distribution of QZz,/zz. x) to parameterize the mean of our generative process po pg(2)/z,). We follow Ho et al. [22] and set the variance of po(za|ze (2:) to (s, = 1 atjss a choice given by the variance of the forward process.
For generation, we utilize the standard DDPM sampler, also known as the ancestral sampler 22 We
sample some initial noise Z
(O, I) and iteratively apply the update rule
1 Ho (z., 1++1, t.) + (t441,
where (O, 1) and the intermediate timesteps t 12 ... tr 0 linearly interpolate between and 0. We use = 250 sampling timesteps by default.
B Additional Language Autoencoder Results
We present results for our language autoencoders on XSum, QQP and WMT14 in Table 9 We observe that our proposed language autoencoders are similarly effective for these datasets.
We also ablate the performance as we vary the dimensionality of the latent space in Table 10, We observe, as expected, that the reconstruction performance improves as the dimensionality of the latent space increases and degrades we decrease the size of the latent representation. We found our default dimensionality of 32 64 to generally effective for high quality reconstructions across datasets.
16

---

Table 9: Effectiveness of Language Autoencoder
Method
Latent Dimensions
Hidden Units
XSum
QQP
Rouge- 1/2/L
BLEU
Rouge- 1/2/L.
BLEU
BART-Basc
L 768
49,152
9
99. 99.9/99.9/99.9
99.8
BART-Base Autnencoder
32 64
2048
99.8/99.6/99.8
99.3
99.9/99.8/99.9
99.1
FLAN-TS-BB Base
L 768
$ 49.152
65.7/51.2/59)
45.7
26.9/14.1/24.3
10.8
FLAN-TS-Base Autoencoder
2048
99.6/99.3/99.6
99.8/99.5/99.7
98.5
WMT14 English
WMT14 German
Rouge- 1/2/L
BLEU
Rouge-1/2/1
BLEU
MTS-Base Autoencoder
2048
99.7/99.2/99.7
99.8/99.4/99.8
99.1
Table 10: Ablation of Autoencoder Latent Dimensionality
Method
Latent Dimensions Hidden Units
RocStories
Rouge-L
BLEU
BART-Base
L 768
Σ 49,152
98.8
97.5
32 x 32
1024
97.0
92.4
BART-Base
32 x 64
2048
99.2
97.6
Autoencoder
64 x 64
4096
99.2
97.7
C Impact of Sampling Steps
We present the results from different sampling configurations for ROCStories dataset in Table We also report the wall clock time needed to generate the 1000 samples across the different numbers of sampling timesteps while batching the generations with a batch size 128.
We find that the number of sampling steps introduces tradeoff between the diversity and the quality of the text, with more sampling steps leading to more fluent but less diverse text and fewer sampling steps leading to less fluent but more diverse text. When using BART-base, the MAUVE score is maximized when utilizing only 100-250 steps, demonstrating that it achieves a reasonable balance between diversity and quality. When utilizing FLAN-T5-bard on the other hand, we find that the MAUVE score improves monotonically with increased sampling steps. This suggests that latent distribution of the FLAN-TS-base autoencoder may be more challenging to learn. Increasing the capacity of the denoising network or the language autoencoder may therefore be beneficial when using FLAN-T5-baa
We observe that the sampling time scales with the number of sampling steps expected, although there is also fixed cost from the reconstruction network and the autoregressive decoder that is independent of the number of sampling steps.
Table 11: Evaluation of different sampling configurations. We use 250 steps by default.
ROCStories
Sampling Steps
MAUVE1
Ppl.
Div
Mem
Wall Clock Time (1000 samples)
Reference
.91..oo
21.1.a
.14.000
.362
50
.844.oa1
32.6.a
.07.004
.337 .00
Im27s
100
.719
38.5.a
368
339
Im55s
LD4LG (BART-baar base)
250
.716.019
30.6.s
.331 oss
.441 nos
3m20s
500
.704.cm
28.1.a
.313.00
.62
Sm444s
1000
667
25.9.1
295
481
10m30s
50
.31.cca
67.9.;
.456.00
.233
Lm34s
100
.42..012
48.7.7
.423 on2
.3/
2m02s
LDALG (FLAN-TS -N--T----ss)
.481.co
37.5.1
.389.om
.387 o02
3m29s
500
.455
32.8.6
.370.oo
.413 008
SmS1s
1000
.522,ca 028
30.6.a
.360.004
.132 005
10m38s
17

---

Table 12: Evaluation of different nucleus sampling configurations.
ROCStories
Sampling Parameter (p)
MAUVE
Ppl
Div t
Mem +
.90
.762.027
19.6.3
.362.0ox
.718 .066
.95
.788 .25
20.0.2
.372 002
.688
GPT-2-Medium
.98
.782
20.2.3
.378 .002
.666 ons
1.00
.793,024
20.5.4
.385,004
.637 .00
D GPT-2 Sampling Ablation
We report an ablation of the nucleus sampling parameter, p. in Table 12 The memorization does exhibit some sensitivity to the nucleus sampling parameter, but the memorization is consistently higher than the LD4LG models across all sampling configurations.
E Implementation Details
All of the models presented in this work are trained on single Nvidia A6000 except for the DiffuSeq XSum baseline which was trained with two Nvidia A6000s.
E.1 Language Autoencoders
We adopt the pre-LN design 70 for both the compression and reconstruction networks and therefore apply layer normalization before all attention and feedfoward blocks. We also adopt query-ket normalization [II and apply RMSNorm [74] to the queries and keys before computing the dot product similarities
attention mechanism. We found that this enabled training with a larger learning rate which accelerated training. We present the hyperparameters for our language autoencoders across all datasets in this work in Table 13
We also report additional details such as the number of trainable parameters and training time. The training time similar across datasets because we use the same hyperparameters, so we simply report the training times for the ROCStories dataset for the monolingual models. For the MT5-base base autoencoder, we report the training time for the German autoencoder which is similar to the English autoencoder. We note that our implementation is not optimized for runtime and that pre-computing and caching the language encoder representations would significantly accelerate training.
E.2 Latent Diffusion For Language Generation
We present the training details across the different datasets in Table 14 We tuned hyperparameters using the validation MAUVE scores for the ROCStories dataset and found that they generally transferred well across datasets. We therefore used the same hyperparameters across datasets, except that we utilized the LI loss instead of the L2 loss for the Seq2Seq tasks. Consistent with prior work image- to- to-image diffusion models [53]. we observed that the LI loss improved the fidelity of the generations at the cost of sacrificing some diversity. This improved fidelity translated to improvements in metrics of interest, although the I.2 loss may still be desireable for settings where diversity is of greater importance. For the unconditional and class-conditional language models, we did not observe overfitting to be problem and simply use the final checkpoint for evaluation. For the monolingual Seq2Seq tasks, we utilize the checkpoint with the best validation ROUGE-L. For machine translation, utilize the checkpoint with the best validation BaCreBLEU.
For the machine translation experiments, we observed benefits from rescaling the noise schedule to emphasize training higher levels of noise. This idea introduced by Hoogeboom et al. 27 and
Chen 7 to improve regouuesolttion image diffusion models. Both Hoogeboom et al. [27) and Chen 7 shift an existing noise schedule by some scale factor, increase the time spent at higher noise levels. Given a noise schedule a, with SNR X
of the shifted noise schedule, 0g,a E [0. 1]. is
1-
18

---

Table 13: Training details for our language autoencoders.
Language Model
BART-hase
FLAN-TS-base
MT5-Base
Trainable Params
26M
26M
591M
Compression Architecuree
Perceiver Resampler
Perceiver Layers
3
1
Perceiver Dimension
768
768
768
Self-Attention Heads
12
12
Autoencoder Latent Length (6)
32
32
32
Autoencoder Dimension (dae)
64
64
64
Reconstruction Architecture
Transformer 66
Transformer Layers
3
Transformer Dimension
768
768
768
Self- Att-nnttntionn Heads
12
12
12
Activation Function
GELU 20
Max Seq Length
64
128
Optimizer
Adam W
Learning Rate
le-4
lc-4
lc-4
(8,, (2)
(0.9, 0.999)
(0.9, 0.999)
(0.9, 0.999)
Batch Size
256
256
128
Warmup Steps
1000
1000
1000
Learning Rate Schedule
Linear Decay
Weight Decay
lc-2
lc-2
Ic-2
Gradient Clipping
1.0
1.0
1.0
Training Steps
50k
50k
50k
Training Time
12h38m
20h17m
20h29m
defined so that
07
Xt.5 x
Given and the scale factor the scaled noise schedule ar,, can be computed in closed-form. Using the relationship that af = sigmoid(log(A:)) (see Kingma et al. (30), the new noise schedule can be computed
2 i sigmoid(log(A...)
sigmoid(log(A) A, 2)) sigmoid(log(A)) + log(s)).
We employ shifted cosine noise schedule with s 0.1 for machine translation. Past work on text diffusion for machine translation observed that training at higher levels of noise improves the models utilization of the conditioning information (i.e. the source sentence) [73
During the inference process, image diffusion models typically rescale the estimate of the data the range of pixel values (i.e. [-1,1]) I) at each sampling step. When we restrict the latent space so that |xx 2 dac- we similarly rescale the intermediate estimates of the data enforee this constraint. This design decision is not critical and similar performance is achieved without this rescaling, We did, however, observe that this made the generative process more robust large guidance weights which may be important in some settings. This observation is consistent with prior findings from text-to-image diffusion
We also report the wall clock times for training the models, although our implementation could be further optimized improve training times. The primary cause of the slowdown for AG News compared to ROCStories, for instance, stems from additional validation sampling and logging for class-conditional generation during training.
When decoding the sampled latent vectors, we utilize beam search with beam size of 4, a repetition penalty of 1.2 [29]. and prevent generations of duplicate trigrams.
E.3 BART-Diffusion
For our BART-Diffusion baseline, we utilize the same denoising architecture as our L.4L- method. As discussed in the main paper, the sequence length of the BART features vary with the length of
19

---

the input text. During training, the sequence length is simply determined by the training instance. To select the length the Gaussian noise during generation, we sample a length from the empirical distribution of lengths in the training set.
We observed that the v-prediction parameterization prediction parameterization was unstable. We therefore adopted the x-prediction parameterization. This is consistent with past work that has found the x-prediction parameterization be more effective for high-dimensional data [36 8
less effective in this setting and the
longer control the scale of the latent space. We therefore follow common practices from latent image diffusion and normalize the latent space to have unit variance [5T] When normalizing the latent space, utilize the first batch of training data to compute the mean for each feature dimension, averaging across the samples in the batch and the sequence lengths of the samples. Therefore, we compute the mean vector jı Σs,e X6,t, f Rd where x E fxx is some batched data. We then compute the global variance across all dimensions the centered latent space a² 1 (x.e,a i2))² , o² E R to rescale the latent space to have unit bea variance, We otherwise train this baseline with the same hyperparameters as LD4LG.
Another challenge is that
E.4 Diffusion LM
We train our Diffusion-LM models utilizing the public implementation by Li et al. 33 We utilize the provided command and hyperparameter settings for the ROCStories dataset. To adapt to the AG News dataset, we increase the batch size from 64 to 128 and set the number of training steps to 250k match our training configuration. We otherwise utilize the same hyperparameter settings as the ROCStories model. We attempted to double the learning rate from le-4 to 2e-4 to account for the doubled batch size, but observed training instabilities and therefore used the original learning rate of le-4.
E.5 GPT-2
We present the default hyperparameters for the GPT-2-Medium baseline in Table 13 For sampling from GPT-2, we prompt it with BOS token and utilize nucleus sampling (p 0.95). We use the same repetition penalty of 1.2 [29] that we use for the LD4LG language decoders and similarly prevent generations of duplicate trigrams.
E.6 DiffuSeq
For the QQP dataset, we compute the metrics with the model generations released by Gong al. [16]. We utilize the official implementation from Gong et al. [16 to train DiffuSeq model on the XSum dataset. In their work, the DiffuSeq models were trained with the same hyperparameters across all datasets considered, except for the number of training steps which varied across datasets. We therefore adopt their default hyperparameters for the XSum dataset.
We observed that the DiffuSeq models were trained for much longer than our models. The official implementation also utilized gradient accumulation with microbatches of 128 to achieve a large effective batch size of 4094 We trained the XSum DiffuSeq model for 960k iterations which is significantly longer than the 250k iterations used by our LD4LG XSum model. Due the of gradient accumulation, this corresponds to 30k gradient updates. The XSum DiffuSeq baseline was therefore trained for over 3.8x more epochs than our method.
limitation of the DiffuSeq model compared to LD4LG is that it concatenates the source and target sequences the input to their transformer model. DiffuSeq therefore scales quadratically with respect to the combined length of the source and target sequence. Our denoising network, on the other hand, operates upon fixed sequence length of f 32 latents and only cross-attends to the
Sstps://gittrn htt.ggitt......heeeeeeeeeeeeeeee Mttbaarr---N//Dffufseee
We note that the original DiffuSeq implementation had a bug in its implementation of distributed training (see /tttps://giiie wwtp:/oo...oo///////e implementation.
com/ Shark NLP Wtt.DIII///ifeg////occtesl htsp/////////ssss//e//eeeeee /3 We describe the behavior of the original
20

---

source representations. As result, our method scales linearly with respect to the length of the source sequenca This enables LD4LG more efficiently incorporate long contexts than DiffuSeq.
By default, the official DiffuSeq implementation limits the combined length of the source and target sequences maximum length of 128. This could put which incorporates up to 256 tokens of the source sequence. To ensure fair comparison, we also experimented with increasing the maximum sequence length for the DiffuSeq model to 256 tokens, which significantly increases the training overhead. After training the model for 640k iterations, which took days with two Nvidia A6000 GPUs, we observed worse performance than the model using the default length of 128.
disadvantage compared to model
E.7 Encoder-Decoder Language models
We report training hyperparameters for fine-tuning the pre-trained encoder-decoder language models models on the Seq2Seq datasets in Table 16 We perform early stopping with the validation ROUGE- L.
E,8 Evaluation Metrics
For the MAUVE, ROUGE, BLEU, BERTScore, Perplexity, and SacreBLEU metrics, we utilize the implementations provided by the Huggingface evaluate library (attps://huggingf @ocs/evaluatee For BLC,,BLEU, we follow prior work and use the intl tokenizer if the target language is German and use the 13a tokenizer if the target language is English.
co/
For the n-gram metrics, we utilize the en_core_web_as tokenizer from Spacy pettps:/sspacy io/) split the generations into tokens.
F Dataset Statistics
ROCStories [42]. The dataset consists of 98,161 instances. We hold out 1,000 instances for validation, 4,000 instances for testing, and utilize the remaining 93,161 instances for training.
AG News Topic Classification [60]. The dataset consists of titles and short descriptions from news articles. We discard the titles and focus on generating the descriptions in this work. The official train/test splits have 120k training instances and 7,600 testing instances. We hold out 1,000 instances from the training set for validation. We therefore utilize 19k training instances, 1,000 validation instances, and 7,600 test instances.
XSUM 443.
The dataset consists of BBC articles from 2010 to 2017 covering wide range of topics (c.g., News, Politics, Sports, etc.). Each example in the dataset consists of a news article and summary, It has 204,045 training instances, 11,332 validation instances. and 11,334 test instances.
The dataset consists of 400k question pairs, where example consists of two similar questions and binary value indicating whether the questions have the same meaning. The semantically similar questions can be utilized as paraphrasing dataset. We use the version released by Gong et al. (16) to enable direct comparison. has 144,715 training instances, 2,048 validation instances, and 2,500 test instances.
QQP 91.
WMT 2014 English-German [5]. The dataset consists of roughly 4.5 million paired English and German sentences for training. The validation and testing splits each have roughly 3k paired sentences.
G Qualitative Examples
We present random unconditional samples from the diffusion models for the ROCStories Table 17) and AG News (Table 18) datasets. We note that because the Diffusion-LMM learns token representations from scratch and cannot model rare words, Li et al. (36] replace rare words with an UNK token. We observe that these tokens are often generated, leading to incoherent text. This problem is particularly
For LD4LG, the frozen language encoder still scales quadratically with the source sequence length, but the source representations can be pre-computed and cached prior to training.
21

---

Table 14: Training details for LD4LG across different datasets.
ROCStories
AG News
XSum
QQP
WMT14-En-Dc
Trainable Params
188M
190M
217M
217M
218M
Sampling Timesteps
250
Noise Schedule
Cosine 1.2
Cosine L.2
Cosine LI
Cosine
Shifted Cosine (s 0.1)
M1
Regression Loss
L.1
L1
Transformer T Layers
12
Transformer Dimension
768
Self-Attention Heads
12
Dense Connections
3
Activation Function
GeGLU
59
Optimizer
Adam 39
Learning Rate
2e-4
2e-4
2e-4
2e.4
4e-4
(8,, 5y)
(0.9, 0.999)
Batch Size
128
512
Warmup Steps
1000
Learning Rate Schedule
Cosine Decay
Weight Decay
le-6
Dropout
0.1
0.1
0.1
0.1
0.0
Gradient Clipping
1.0
1.0
1.0
1.0
0.2
EMA Decay
0.9999
Training Steps
250k
250k
250k
250k
500k
Max Seg Length (Source)
n/a
n/a
256
64
128
Training Time (BART-base)
Id 11h
Id 20h
22h
Id 20h
Training Time (FLAN-TTS-ase)
1d 17h
1d 21h
4d
2d 7h
Training Time (MTS-basee
9d 16h
Table Training details for our autoregressive baseline across different datasets.
ROCStories
AG News
Model
GPT-2-Medium
Trainable Params
355M
Max Seq Length
64
Optimizer
AdamW W (39
Learning Rate
8c-5
(8,, 82)
(0.9, 0.999)
Batch Size
Warmup Steps
500
Learning Rate Schedule
Linear Decay le-2
Weight Decay
Dropout
0.1
Gradient Clipping
1.0
Training Steps
100k
Table 16: Training details for our Seq2Seq baselines model.
XSum
QQP
Model
BART-base 139M
FLAN-T5-EE ---sc-basc
BART-base 139M
FLAN-T5- ------
Trainable Params
220M
220M
Max Seq Length (Source)
256
64
Max Seq Length (Target)
64
Optimizer Learning Rate
Adam W
5e-5
le-4
5e-5
5e-5
(3,, 3z)
(0.9, 0.999)
Batch Size
32
Warmup Steps
500
Learning Rate Schedule
I Linear Decay 1c-2
Weight Decay
Gradient Clipping
1.0
Training Steps
100k
22

---

Table 17: Random samples from ROCStories dataset.
LD4LG (BART-base)
LD4LG (FLAN-TS-base)
Diffusion-LM
After a long line in line, Amy was ready to carry her cart. She asked if she should put the money in bag- The cashier gave her quarter and she opened the bag. She was happy to see that she paid for the amount on the line. The checkier checked when she
Emma was playing with her doll doll. She was having good time when suddenly she slipped! The doll doll shattered in many places! Emma was so upset she cried and cried! Her mother took her home and got her
Tom was going to cat with friends. But it was stressed out. So He de- cided to go to the local bar. But when he realized his friend was too much. The police allowed home to pull him home.
band-aid.
Max wanted to build a tree in his backyard. He researched guides on what kind of plant to plant. He went online and cut trees so he could see one that would cover large. He bought all his supplies and drove to the farm dealership. They had planted beautiful backyard in his neighborhood.
Rita was about to go out in the UNK. UNK was the UNK and Rita was very nervous. She took out the ball was beginning
Barry was a popular high school stu- dent. Hc always got good grades in school. Barry's 's friends all met up. He arrived at his new job with big grin. Barry decided he would start the new job as a teacher.
UNK. She kicked the ball still and knew she was a good kid. She looked in her shoulder and immediately ran to the sound.
John and Molly thought it would be fun to go to Europe. They decided to take their little child to go swim- ming. The child had wonderful time playing in the waves. They also had ta lot of local food. They were exhausted when they still had re- turn home.
Michael had a crush on a girl. He finally had the courage to talk to her. Michael went over to her and she walked down hallway. They chat- ted for hours. Michael wished he had never asked another girl.
bought a new UNK. It was a UNK. My friends asked it for some money. We didn't 't listen. was declined.
Todd was walking his dog with his dog. The UNK hit by minutes close to check something out. There was a small UNK and UNK off of the ground. He got to the UNK's ' house to find UNK UNK. Todd's dog started to listen to the UNK of it.
Yesterday lulu went to the theme park. To her surprise her phone fell out at the park. She was so disap- pointed. But thankfully no one was looking for it. She had to walk home fast she could to get
Ed got a chihuahua. It escaped its cage. Ed was able to free the chi- huohua. He wanted to keep him so he let it alone. Ed is able to keep the rest out.
Maria was getting ready for her trip. She wanted a specific bathing suit, and went to the mall. She tried on many different outfits, but none fit. Maria realized she had found great deal while shopping. She bought herself nice suit.
Anna had been friends with her fam- ily for years, but curious. Later, Anna's 's mom told her she might be sick after a bad age. Anna broke up with this, and swore that she would not get sick. That night, Anna threw up all over the house
Stacy wanted to learn how to ride horse. She found a long one near her UNK. She decided to UNK on. Finally she was able to ride a UNK. Stacy was happy to be her own horse.
pronounced for the AG News dataset which has a more diverse vocabulary and uses many proper nouns such as names that are out-of tit---ff-oocbularyy. We also present random class-conditional samples for the AG News datasets (Table 19 and Table 20) for all the classes.
We present examples of sequence-to-seque -esuueence generations for QQP in Table 21 and XSum in Table 22 While the DiffuSeq generations are somewhat reasonable for the simpler QQP paraphrasing dataset, the model completely fails to produce coherent summaries for the challenging XSum dataset. This is the case even though DiffuSeq is trained for significantly longer than LD4L.G and uses 8x as many sampling timesteps.
23

---

Table 18: Random samples from AG News dataset. HTML entities are decoded for readability.
LD4LG (BART-basc)
LD4LG (FLAN-T5-base)
Diffusion-LM
What could have been a decisive role in Disney's ss merger of a lead- ing media group, but not only it appears have been. Last night, the founders of the media conglom- crate's leading stock management unit, introduced legislation capping the
Sachin Tendulkar has found himself fit to India's batting squad ahead this weekend's final and final session of the first Test against Bangladesh in East Oval.
UNK UNK UNK UNK UNK UNK, UNK UNK UNK UNK UNK. the UNK UNK UNK, the UNK UNK of UNK UNK UNK UNK.
A federal grand judge has reached a new $ UNK stake in UNK for the $ 35 billion, UNK leading investors the UNK. & <FONT face="verdang, MS Sans Scrif, arial, helvetica sizc="-2 Washington UNK
America Online and Ask Jeeves set- tle over file-swapping technology that could lead to lawsuits against hundreds of online businesses and result fraud.
The startup provider will provide CRM-based services for small and midsize businesses its offices.
Reuters U.S. oil and gas compa- nies will try develop and develop a new greenhouse gas system over the next two years in a bid to give more cautious sense of environ- mental conditions for the economy. senior US Energy Department of- ficial said on Wednesday.
Real Madrid moved to the top of the Bayem Munich's Premier League standings Wednesday, with Atletico Atletico charge fol- lowing a 2-0 draw against Porto.
North Korea's UNK UNK and UNK UNK UNK UNK the UNK of UNK, UNK UNK, UNK UNK UNK 6 4. 6 at the $ UNK US UNK today.
Reuters Three more Americans will be able make cloning cloned research make medical research innovation that brings them to the massive victims of tuberculosis vaccine, the British government an- nounced on Friday.
Andre Agassi upset Carlos Moya 4, 6-2, 6-3 to reach the Stockholm Trophy for the first time on Sunday and wrapped up his first grand slam title.
LONDON Reuters ) It was UNK but the European team's 's UNK man's 's Davis Cup game was upheld in the last week due to their start into the semi finals, manager said.
Gary Sidson wants the Miami Heat step down from the Dallas Maver- icks the end of the offseason after undergoing medical proceedings to relieve him. Sidson told The As- sociated Press. "Every wife has a choice" to make him head with
Canada's 's rules federal audits pose a threat to Canadian compa- nies, some wanting to keep journal- ists out of their jobs, Sen. Thomas Powell warned during his annual meeting of the Securities and change Commission last summer.
UNK UNK UNK, the UNK of the UNK UNK has decided to stop UNK the old UNK UNK UNK UNK a UNK, of his father UNK of UNK.
24

---

Table 19: Random conditional samples from AG News dataset. HTML entities are decoded for readability.
LD4LG (BART-base)
World
Sports
Business
Sci/Tech
Australia have dropped their suspension for next week's game cricket series against In- dia, but it seems to have emerged. Last night, owners of the Inter- national Cricket Coun- cil's leading governing body, announced plans for scrapping the
What could have been a role in digital's business of the record industry, but it already makes message to investors. Last year, the mu- sic industry's 's leading record companics, pro- posed legislation scrap- ping the
President Bush's re- election has been number of central issues of the Middle East, but it appears to have happened. He headed the US intelligence committee's 's re-election yesterday, prompting end of the
What could have been the final step in Dis- ney's merger of lead- ing media group, it's likely to happen. Last year, the longtime media conglomerate's 's (NYSE: news research) re- search and cable empire topped the $10
first
founders
AP The Bush ad- ministration has agreed change its portfo- lio af redeploying addi- tional U.S. troops to Iraq to prevent possible de- ployment of many U.K. troops there, the White House and Democratic lawmakers said Friday.
You go home, sit uni- form and work in four times. Every day now the Expos are starting to change, along with many people. want to get us about the goals and fore- casts of their playoff
The internet phone provider
The upstart will provide satellite-based phone ser- vices and more services help customers man- age their Web applica- tions.
charge fee-hased phone calls more closely to keep consumers in their hands.
Real Madrid could be in trouble to disrupt Bay- Munich's Champi- ons League clash with Porto at the San Siro on Sunday. Although the recent hat-trick from Ronaldinho has helped Brazil
government stepped up a programme yester- day to monitor Russia's s school siege, including a school Beslan in school where more than 400 people have fled Russia.
The Cassini-US space- craft continues moni- tor Saturn' 's largest moon Titan, Saturn' 's larger moon Titan. A region where the swirling of dust dust have trig- gered Saturn
The Kremlin asked order yesterday punish Irina Yukos' CEO, chairman of Rus- sia's 's state gas monopoly and the beleaguered oil firm Yukos.
AP Terrell Owens will be suspended indef- initely for the Kansas City Chiefs due Hur- ricane Frances, and he will return to the team Sunday when they face the Tampa Bay Bucca-
Reuters stem cells used make cloned cells for medical re- search experiments and innovations that could open the open future women cloning, British researchers reported on Friday.
Reuters Thatcher will be re- leased in Cuba after undergoing a brain surgery, her father said on Friday as she flew to the Middle East help patients evacuate her.
Prince
Where you' ever seen any wireless lending us- ing the Internet you see in your house, or brings it to great shift when you teleport to the net?
NEW DELHII: Prime Minister Manmohan Singh asked the gov- ernment to do more reviving the Kashmir peace process with India. Singh, a spokesman for the Association of Ase Nations Monday said the Indian Government willing to post off
NewsFactor (NYSE: IBM) is ac- quiring its WebSphere division in a deal valued US $160 million cash. Mcanwhile, the companies said the deal valued of US $1.5 billion for Sybase (Nasdaq: ADABECKs) (Rasdaq
IBM
Paul Hamm made surprise exit from the Athens Olympics favor of the Olympics after entering the Games but plans to give him, his spokesman said Monday. "No one has mistake" to the US gymnastics body.
ATHENS
MOSCOW (CBS.MW) Supporters of Rus- sia's beleaguered oil firm Yukos have an- nounced they will file
for bankruptcy this week press ahead with $1.1-billion back-tax bill.
25

---

Table 20: Random conditional samples from AG News dataset. HTML entities are decoded for readability.
LD4LG (FLAN-TS-Base)
World
Sports
Business
Sci/Tech
LeBron James did all of them they needed, leading the Sacramento Kings to 92-80 victory Friday night against the Toronto Raptors at Air Canada Center.
US Airways has reached an agreement with its pilots representing the nation's biggest airline, just three weeks after emerging from bankruptcy protection.
The first close-up images of Saturn's largest moon Titan have been seen eral times long on August 28th this image captured by the Cassini Space Telescope.
British Prime Minister Tony Blair met secu- rity officials at Queen's 's Palace in London just weeks after the US- led invasion of Iraq.
AFP-based Southeast Asian countries agreed to negotiations on a job-cutting deal that could result in trade fiasco in the world's biggest. economy a solution to post-war conflict.
SAN FRANCISCO (CBS-MW): Cingular Wireless is close to talks over possible US $11 billion purchase of US storage firm Veritas Software.
AFP-Ajax Amsterdam coach Rafael Nedved set aside to order shake- up of Spanish pre- mier league club after a month of speculation.
America Online and Ask Jeeves continue work on file-swapping tech- nology that could lead to hundreds of online busi- nesses of revolution.
WASHINGTON
(Reuters) U.S. health and health officials agreed Wednesday begin testing a new biodegradable drug that tests questionable risks of heart disease, with newly published report from the Food and Drug Administration (FDA) only the only one likely have
Reuters's web search leader Yahoo will begin testing new desktop search engine with Web scarches next month in bid to maintain Google's remaining foothold the Internet world, a company executive said Wednesday.
AP The University of Washington announced Wednesday that its football coach, Tyrone Willingham will be serv- ing as head coach at Notre Dame for its only second run in the va- cancy.
WASHINGTON
(Reuters) U.S. chain store sales grew by 0.2 percent the latest week, held back by the biggest year-carlier pace history, the Commerce Department said on Wednesday.
Voters had to cast their ballots in Afghanistan's landmark presidential election with an easy and easy win over Hamid Karzai its first popularly elected leader.
Miguel Jimenez shot a 4-under 66 to take one-stroke lead over Car- los Moya after third round the PGA Tour tournament.
'We're all wanted win one I all want to do that, "Tiger Woods' had already had a very cru- cial victory.
Grand Theft Auto needed only 200,000 get on Xbox; although they just wanted to do so, it's just very impressive.
The new Securitics and Exchange Commission rules will lead to deep defence of corpo- rate business and the company will continue its oversight of account- ing practices, US U.S. Senator Thomas Powell warned during brief with the Security and Ex- change Board last night.
AP Records for-food market and job embargoes produce yet another political stale- mate Democratic Sen. John Kerry's ef- fort to end the lawsuits of journalists, including the ones carried out by the White House, at the summer weekend
oil-
file-sharing services are legal, the Federal Court of Justice (FTC) not liable copyright infringements in the UK, which was brought in by the Federal Trade Commission (FTC) rul- ing, brought yester- day.
The Boston Red Sox are old, but the slugger of baseball old, de- spite expected flurry of diplomacy and out- rage from fans that the holidays will bring their way at Fenway night, Sunday night end
26

---

Table 21: Samples from QQP Paraphrasing Dataset.
Source: What creative and innovative business ideas with less investment in India?
Reference: What best business ideas with minimum investment?
LD4LG (BART-Base)
LD4LG (FLAN-T5-basc)
DiffuSeq
What are some innovative business
What are some creative business
what are best business ideas with less class investment?
ideas with less investment?
ideas with lesser investment?
What some business ideas with minimum investment?
What are new business ideas
what some business ideas with less investment india?
minimum investment?
What some business ideas with lesser investment?
What are new business ideas with lesser investment?
what some business ideas with available in india?
Source: Can height increase after 25?
Reference: Can someone increase their height naturally after 19?
LD4LG (BART-Basc)
LD4LG (FLAN-TS-basc)
DiffuSeq
it possible increase their height after the age of 25?
Is possible increase height after the age of 21?
how increase taller after 21?
it possible to increase the height after the age of 21?
Is it possible to increase height after the age of 25?
how can increase height after 21?
there any way to increase the height after 21?
Is possible increase height after
how girls is increase our height? can be his, 21 18 years?
age of 21?
27

---

Table 22:
Samples from XSum Summarization Dataset. Parts of the articles are omitted for brevity.
Article: Two-year-old Lane Thomas Graves had been playing in the sand near the resort's Seven Seas Lagoon when he was dragged underwater by creature... The lighthouse has been installed near to where the attack occurred... A Disney spokesperson said they hoped the monument would spread awareness for the Lane Thomas Foundation, which also uses the lighthouse as its logo. Who is liable for alligator boy's 's dcath? "The lighthouse sculpture has been installed help spread awareness of the Lane Thomas Foundation, which was established to provide assistance and support to families whose children need organ transplants, 'Walt Disney World said in statement.
Reference: Walt Disney World has unveiled lighthouse memorial for young boy who was killed by alligator while on holiday at the Florida theme park.
LD4LG (BART-Base)
ID4LG (FLAN-TS--ase)
DiffuSeq
giant lighthouse in memory of a US boy who was fatally attacked by alligator has been installed at Walt Disney World.
A sculpture lighthouse has been in- stalled in memory of an Ameri- can boy killed by an alligator at Florida theme resort.
teenager has been ap graves claimed in after the in the over killing sand of an resort.
national lighthouse in memory of a US boy who died when he was attacked by alligator has been in- stalled at Disney World.
A US boy has been killed by al- ligator while playing in sand Florida theme resort, officials say.
a woman has died from s over a a caused parents in county accident a.
giant lighthouse has been in- stalled at Walt Disney World in Florida in memory of a US boy killed by alligator.
A Disney lighthouse has been in- stalled in memory of an Ameri- can boy killed by an alligator on Florida holiday resort.
a speech of man who killed the us- ing his leicester has by a resort been some been has resort playing by raised.
The Sky Blues currently play in Coventry's Ricoh Arena but had a long dispute with the stadium's 's previous owners... In a statement, Rughy Borough Council said its leader and the council's executive director and head of planning had met with Coventry City in March. "The club requested the meeting to understand how the council would deal with any planning application for potential stadium sites in the borough of Rugby." it said, It said the plans would need to be finalised by September to be included in the council's 's local plan, but added that site had yet be identified. Peter Ward, from Sky Blues Supporters' Consultative Group, said he was pleased hear that things were "moving" with the club's search for new home. "It's good that finally there is some evidence things
Article:
Reference: Planners in Rugby have revealed they have been in talks with Coventry City Football Club about building stadium in the borough.
LD4LG (BART-Bar Base)
LD4LG (FLAN-TS-base)
DiffuSeq
Premiership club Rugby Football
Coventry City have held talks with Rugby Borough Council to consider plans for stadium.
coventry city will have midfielder barack their for as has a poor his at side the club one back until.
Club have met with Coventry City Council to discuss a potential new stadium in the city.
Coventry City's Rugby Football Club have met with local authori- ties to discuss the potential site of new stadium for next scason.
Coventry City have held talks with Rugby Borough Council to discuss the search for new stadium.
owners city's iity's david stadium been has council been sky to league by talks from the club" at london.
Coventry City football club has met with Rugby Borough Council to dis- cuss potential sites for a new sta- dium in the city.
coventry city have set stadium new league over cup death following after deal was league their club club.
Coventry City FC has held talks with the borough council to discuss a new stadium.
28


Reviews:
[{"review_id": "JODZEyt0Gx", "reviewer": "Reviewer_SqZp", "summary": "This paper introduces a latent diffusion model for natural language generation, utilizing a continuous diffusion framework. The authors leverage pretrained encoder-decoder models, such as BART and Flan-T5, to establish a continuous feature space. They propose a Perceiver Resampler as a compression module to handle variable-length and high-dimensional challenges, allowing effective learning of the diffusion process and parameterized denoising. The generative process includes a reconstruction module to map the low-dimensional latent space back to the feature space and is supported in unconditional, class-conditional, and seq2seq settings. The proposed approach demonstrates its effectiveness through extensive experiments.", "questions": "See weaknesses. \n\nI am happy to raise my rating upon the authors' responses to my concerns.", "limitations": "n/a", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The utilization of pretrained encoder-decoder models to establish a feature space is a clever and efficient approach, distinguishing it from previous work that relied on less-capable embedding \"encoder-decoder\" for latent diffusion.\n2. The paper conducts extensive experiments in various settings, including unconditional, class-conditional, and seq-conditional text generation. The evaluation metrics cover aspects of quality, diversity, and other relevant factors.", "weaknesses": "1. Evaluation concerns arise from the lack of a multilingual setting in the benchmark datasets investigated. The benchmark datasets investigated in the paper were various, which was good, but lacked of multilingual setting, making the evaluation not fully convincing. Several previous diffusion-based text generation studies experimented on machine translation, e.g, CDCD [2], Difformer [2], and DINOISER [3] studied on WMT14 English-to-German dataset, which is most commonly-used setting for non-autoregressive text generation. Can the proposed LD4LG model also be applied in a multilingual setting?\n\n2. The paper lacks a comprehensive literature review, hence comparison and discussion on its contributions with relevant studies. In the experiments, only diffusion-lm and diffuseq were compared. Additionally, the related work section primarily cites papers from 2022, while significant progress has been made in text diffusion research this year. What are the advantages of LD4LG compared to other continuous diffusion models, such as CDCD, Difformer, and DINOISER? How does it compare to discrete diffusion models like DiffusionBERT, reparameterized discrete diffusion model (RDM), and DiffusER?\n\n---\n[1] Continuous diffusion for categorical data\n\n[2] Difformer: Empowering Diffusion Model on Embedding Space for Text Generation\n\n[3] DINOISER: Diffused Conditional Sequence Learning by Manipulating Noises\n\n[4] DiffusionBERT: Improving Generative Masked Language Models with Diffusion Models\n\n[5] A Reparameterized Discrete Diffusion Model for Text Generation\n\n[6] DiffusER: Discrete Diffusion via Edit-based Reconstruction", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\n**Multilingual Evaluation:**\n\nWe thank the reviewer for their thoughtful feedback on our paper. We agree that evaluating our approach on a multilingual dataset would further demonstrate the versatility of our approach. The extension of our method to such settings is straightforward given the availability of strong pre-trained multilingual language models. We conducted additional experiments on the WMT14 German->English translation task following the same setup as our other Seq2Seq datasets utilizing a Multilingual T5 model. We report the reconstruction performance below for both English and German for the MT5-base language autoencoder. We observe that it can similarly be adapted to highly effective autoencoders across diverse languages.  \n\nModel | Data | ROUGE-1/2/L | BLEU |\n|-|-|-|-|  \n| MT5 Autoencoder | WMT14 English | 99.7/99.2/99.7 | 99.2 |\n| MT5 Autoencoder | WMT14 German | 99.8/99.4/99.8 | 99.1 |\n\nWe can then train our latent diffusion model for machine translation similarly to our other Seq2Seq datasets. We present our preliminary results below and compare against the additional provided references that report results for that dataset. Unfortunately, the Difformer and DiffusER papers do not open source their implementation or report sufficient details about their evaluation setting to ensure an apples-to-apples comparison. For instance, the Difformer paper reports a tokenized BLEU score as their primary metric but does not provide their tokenizer, or sufficient details to reproduce their tokenization. It is well established that such choices significantly impact BLEU scores and must be controlled for valid comparison [1] [2]. We therefore compare against the methods that follow best practices and report reproducible SacreBLEU scores on de-tokenized generations to ensure that the comparison is valid. We also report the training durations for each method based on the reported training settings (or based on the official codebase in the case of DiNOISER).\n\nThe WMT14 De-En dataset, with approximately 4.5 million training instances, is orders of magnitude larger than the other datasets we examined. Given its immense size, maximizing performance typically requires significantly more computational resources than we utilized for training the models in our work. We began training our model as quickly as we were able, but our method is still significantly undertrained compared to the related work due to the limited time available for our response. We present preliminary results below for WMT14 De->En.\n\n| Model | SacreBLEU |\n|-|-|  \n| Diffusion-LM (n=5) | 20.3 |\n| CDCD (n=1) (~227.6 epochs) | 24.9 |   \n| DiNOISER (n=5) (1000 epochs) | 28.8 |\n| LD4LG (MT5) (n=1) (14.4 epochs) | 24.1 |\n\nOur method again outperforms the Diffusion-LM baseline (adapted for MT in the DiNOISER paper). Even when training our model on 6.3% as much data as the CDCD model, its performance is within a single BLEU point. This demonstrates that our presented framework can be effectively extended to multilingual tasks such as machine translation. Our model does lag meaningfully behind DiNOISER with MBR decoding, but given that it is trained on less than 1.5% as much data, it is difficult to compare the approaches directly. \n\nOur model is still improving steadily with training, so the presented result is a loose lower bound on the effectiveness of our approach. We will include finalized results for WMT14 En-De for both translation directions in our camera ready. \n\n[1] A Call for Clarity in Reporting BLEU Scores\n\n[2] Scientific Credibility of Machine Translation Research: A Meta-Evaluation of 769 Papers\n\n**Literature Review:**\n\nWe thank the reviewer for providing additional references. We focused our comparisons on the state-of-the art diffusion approaches published when we began the project, but acknowledge that several preprints describing concurrent language diffusion methods have since been posted. We would like to note that our DiffuSeq baseline was published at ICLR 2023, which occurred the same month as the submission deadline. We, however, acknowledge that rapid progress is being made in the area and will expand our related work discussion to include the provided references and compare directly against these methods when possible. For the methods that utilize the same paraphrasing dataset, QQP, we report the performance comparison below, and we will add these results to the camera-ready. We note that these methods use a different BERT model to compute BERTScore, so we recompute BERTScore with that model.\n\n| Method | MBR | ROUGE-L | BERTScore |\n|-|-|-|-|\n| DiffuSeq | n=10 | 58.80 | 83.65 |\n| RDM-absorbing | n=1 | 57.89 | 83.74 | \n| RDM-multinomial | n=1 | 57.25 | 83.66 |\n| RDM-absorbing | n=10 | 59.45 | 84.72 |\n| RDM-multinomial | n=10 | 58.86 | 84.66 |\n| DiffusionBERT | n=10 | 58.45 | - |\n| LD4LG (BART) | n=1 | 60.3 | 85.82 |\n| LD4LG (T5) | n=1 | 59.7 | 85.79 |  \n| LD4LG (BART) | n=5 | **61.1** | **86.18** |\n| LD4LG (T5) | n=5 | 60.7 | 86.07 |\n\nWe observe that LD4G outperforms all other methods using both BART and FLAN-T5, even when n=1. We believe that our approach has two key advantages over these baselines. First, by utilizing existing pre-trained language models to establish a high-quality continuous feature space, we are still able to take advantage of the strong language representations obtained from self-supervised pre-training. Second, our hybrid approach better exploits the strengths of the two complementary generative modeling paradigms. By offloading the challenge of modeling a discrete distribution to the autoencoder, we simplify the diffusion process by restricting it to the continuous, latent feature space.\n\nIn light of our extended comparisons and experiments, we would like to ask if you are willing to reconsider your score. We are happy to discuss any additional concerns or questions during the discussion period!\n"}, {"review_id": "rd4mwitrq2", "reviewer": "Reviewer_5zaG", "summary": "This paper introduces a novel approach to learning diffusion models in the latent space of an encoder-decoder language autoencoder. The proposed latent diffusion model utilizes continuous latent codes to generate text through sampling, leveraging a pre-trained decoder. The effectiveness of this approach is demonstrated across various unconditional and conditional sequence generation tasks.\n", "questions": "1. In line 107, if the latent vector is normalized by its norm, then the latents are not Gaussian anymore but become a point on the hypersphere. Will it make more sense to divide the latent with its standard deviation estimated from the data?\n\n2. From the encoder to latents and the latents to the decoder involve two cross-attention, will this cause training instability when optimizing things end to end? On the other hand, how does the model avoid copying all encoder information to the target side?\n\n3. How are the length and dimensions of the latents decided? Will it be possible to design variable lengths of latents to account for the actual sequence length?", "limitations": "The limitations were addressed.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea presented in the paper is straightforward and easy to understand. It extends the concept of latent diffusion models (LDM) used in other domains, such as images, to the language domain, showcasing the versatility of this approach.\n2. This paper stands out by incorporating an autoregressive decoder into diffusion models, which combines the strengths of both methods. This integration significantly enhances the quality of text generation, distinguishing it from existing diffusion-based language models.\n", "weaknesses": "The basic idea of learning diffusion models in the latent space has been well explored in the image domain. Adding a self-conditioning trick has also been used in previous diffusion-LM literatures. Therefore the overall novelty is relatively limited.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Novelty**\n\nWhile we agree latent diffusion is commonly used for images, the adaptation to language is non-trivial and presents unique challenges given the discreteness of the domain. Latent diffusion models are reliant on the strength of the underlying autoencoder, and we introduce a method for effectively and efficiently adapting existing pre-trained language models into strong language autoencoders. The discreteness of the modality is another challenge and, as the reviewer acknowledges, we are distinctive from other diffusion language models in developing a hybrid approach utilizing the complementary strengths of diffusion and autoregressive generation.\n\n**Latent Normalization**\n\nIt is not critical that the samples from the data distribution are approximately Gaussian. For instance, diffusion models are commonly trained in the pixel-space of natural images. It is, however, beneficial to control the scale of the data samples to ensure that common noise schedules can be applied without adjustment. The reviewer is correct to point out that re-scaling by the standard deviation would serve a similar purpose as our L2 normalization. \n\nIn our preliminary experiments, we experimented with both enforcing L2-normalization during training and dividing the latents by an empirical estimate of the standard deviation as the reviewer suggests; we found that both were effective. A key technique employed by image diffusion models is to rescale pixel values after each diffusion step to the range of allowed values (i.e. [-1,1]). Such techniques have been shown to be critical for utilizing large guidance weights during inference with text-to-image diffusion models (e.g. Imagen). One benefit of using L2 normalization is that it enables an analogous practice of clamping the estimated data to the manifold by re-normalizing the estimate of the data after each diffusion step. \n\nWe observed that the L2 normalization approach did make the downstream diffusion model much more robust to strong guidance during generation as expected. To illustrate this we report the perplexity of the generated XSum summaries across guidance weights for the two methods from some of our earlier experiments exploring this choice.\n\n|  | Autoencoder w/ STD Re-Scaling | Autoencoder w/ l2 Normalization |\n|-|-|-|\n| Guidance Scale | Perplexity | Perplexity |\n| 1.0 | 75 | 78 |  \n| 2.0 | 69 | 77 |\n| 5.0 | 136 | 94 |\n| 10.0 | 351 | 123 |\n\nWe observed that large guidance scales break the generated language when using STD re-scaling, but this is mitigated by using L2 normalization with the clamping trick from image diffusion. In practice, we found that relatively modest guidance (w=2.0) maximized performance for the datasets we examined (see Figure 3), so this particular design decision did not end up being critical for the performance of our approach. We will add an ablation of this decision to the appendix with discussion of our observations; it will be relevant for settings where strong guidance is necessary and would likely be of interest to the community.\n\n**Cross-attention/ Training stability**\n\nThe Perceiver Resampler, which we utilize for the language autoencoder, was originally introduced to compress image features for integration into the vision-language model, Flamingo [1]. Similar to our setup, the compressed image features cross-attend to the output of a ViT encoder and are then cross-attended by the language decoder. Flamingo therefore has a similar sequence of cross-attention layers and is stable at scale (80 billion parameters). \n\nEmpirically, we did not observe training stability to be a problem in our setting. We tuned the initial parameters such as the learning rate for the ROCStories dataset and were able to apply them directly to all other datasets without adjustment.\n\nDuring the autoencoder training, the Perceiver Resampler bottlenecks the variable-length encoder feature representation to a 2048-d latent, an over 20x reduction of the encoder features. This prevents directly copying the encoder features and forces the model to learn a compact representation of the language that can be reconstructed by the decoder. Our strong performance across diverse datasets indicates that the autoencoder is effectively generalizing beyond memorization. \n \n[1] Alayrac, Jean-Baptiste, et al. \"Flamingo: a Visual Language Model for Few-Shot Learning.\" (2022).\n\n**Length of the latents**\n\nFor the size of the latents, our goal was to maximize compression while maintaining strong reconstruction performance with the auto-encoder. This accelerates learning for the diffusion process while maintaining the capacity for high-fidelity language generation. We observed that 32x64 dimensional latents led to high quality reconstructions across all datasets and the reduction in sequence length accelerated model training. When ablating the size of the latents for the auto-encoder, we observed that further decreasing the dimensionality of the latents (e.g. 16x64 or 32x32) began to degrade reconstruction performance. We will add this ablation to the supplementary material. We would also like to re-emphasize that our method can generate sequences of different length; even though the size of the latent that is used to condition the decoder is fixed, we generate text using the decoder until the EOS token is generated. \n\n"}, {"review_id": "ZT1MVphFs9", "reviewer": "Reviewer_T1gz", "summary": "The paper demonstrates that encoder-decoder language models can be used to efficiently learn high-quality language autoencoders and to learn continuous diffusion models in the latent space of language autoencoders so that continuous latent representations of natural language can be sampled using pre-trained decoders.", "questions": "1. Is the autoencoder joint training with the denoising network? \n2. What is the influence of different VAE models on the final generation performance?", "limitations": "Yes, the authors clearly give the limitations of their work.", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The paper proposes an additional compression module which maps high-dimensional encoder representations to low-dimensional fixed-length representations. And the corresponding reconstruction network is trained to map these fixed-length features back to high-dimensional features, guiding the frozen language decoder to reconstruct the original language.\n2. The paper verifies the effectiveness of the model's unconditional, class-conditional, and sequence-to-sequence language generation methods on several different datasets.", "weaknesses": "1.  The motivation of the paper is somewhat weak. Why do we need to apply the latent diffusion model on text generation? What are the advantages over autoregressive and non-autoregressive models? It seems that the performance (MBR-5 in Table 6) is worse than the original AR model. Maybe it is faster? Overall, the lack of clear motivation makes the paper read like a technical report.\n2. I have reservations about the ability of the variational paragraph embedder to learn effective representations of sentences with different lengths. What would happen if there is a significant difference in sentence lengths, such as one being very short (e.g., 5 tokens) and the other being much longer (e.g., 512 tokens or more)? Moreover, what is the influence of fixed length number, $l$ in line 102?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Motivation:**\n\nAs noted in our overall response, a key motivation of our work is exploring how diffusion models can provide complementary strengths to autoregressive generation for text. Our approach aims to combine the benefits of both methods by learning the diffusion model in the continuous latent space of a pretrained autoencoder.\nAs the reviewer rightly asks, the diffusion process is critical for enabling enhanced sampling, controllability, and diversity compared to directly using the discrete token features. By smoothing the features into a continuous latent space, sampling from the prior becomes more flexible and expressive. The autoencoder handles converting the samples back into the discrete text domain. In this way, our hybrid approach benefits from both the diversity and controllability of diffusion as well as the discrete text modeling capabilities of autoregressive decoders. We hope our work helps lay the foundation for realizing the immense potential of diffusion models to advance text generation.\n\n**Sequence length:**\n\nOur framework relies on the autoregressive decoder's capacity for generating sequences of different lengths. If the decoder struggles to model a dataset where the lengths are bimodally distributed, then this is a limitation that will be shared by auto-regressive models and our model. In contrast to other diffusion models that need to model the length of sequences directly, we offload that part to the decoder and our diffusion model just generates samples in the fixed sized latent space.\n\nWe develop our language autoencoders across 4 diverse datasets (6 including our machine translation experiments conducted in response to Reviewer SqZp), each with input sequences of varying length, and observe empirically that our proposed autoencoders obtain near perfect reconstruction (i.e. BLEU and ROUGE scores of ~99). The autoregressive component of our language autoencoder is therefore effective at mapping the fixed-length latent representations to variable length natural language utterances with high fidelity.\n\nThe reviewer raises a good point that there must be some limit to this sequence length capacity, although we did not encounter it in our experiments. Investigating the use of our framework for open-ended generation that requires maintaining long-term coherence is interesting future work but beyond the current scope.\n\n**Length of the latents:**\n\nFor the latent space, our goal was to maximize compression while maintaining strong reconstruction performance with the auto-encoder. This accelerates learning for the diffusion model while maintaining the capacity for high-fidelity language generation. We observed that 32x64 dimensional latents led to high quality reconstructions across all datasets and the reduction in sequence length accelerated model training. When ablating the size of the latents for the auto-encoder, we observed that further decreasing the dimensionality of the latents (e.g. 16x64 or 32x32) began to degrade reconstruction performance. We will add this ablation to the supplementary material. \n\n**Training:**\n\nWe follow the practice established by latent image diffusion models and train a high quality auto-encoder prior to learning the diffusion model. We then freeze the autoencoder and train the diffusion network to generate samples from the latent feature space of the frozen autoencoder. We will clarify this in the final version. \n\n**Different Auto-encoder Models:**\n\nIn our experiments we utilize both BART and FLAN-T5 to develop language autoencoders. We observed that they performed similarly in most cases. When ablating the dimensionality of the latent space for the autoencoder, we observed strong performance as long as the language reconstruction quality was high. Please see our response to Reviewer LWig (the **Backbones** section) for further discussion on this point.\n"}, {"review_id": "oNm4rnsJUn", "reviewer": "Reviewer_LWig", "summary": "The paper proposes a latent diffusion model for text generation, by stacking pre-trained powerful language models and fine-tuned projection layers and diffusion networks together. The input is first encoded into hiddens, then the diffusion model generates the hidden used as the condition of the language decoder. The final results are generated then. Experiments are conducted on both conditional and unconditional language generation tasks. ", "questions": "Please answer the questions in Weaknesses. \n\nWhat are the purpose and benefits of utilizing diffusion in such a framework?", "limitations": "The authors have discussed limitations.", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is well-written and easy to follow.\n\n- The method part is in detail which makes readers understand the model immediately. ", "weaknesses": "- The contribution is weak. \n   1) the diffusion model is only utilized to generate the condition of the LM decoder, which means that the model just provides another projection of the original LLM encoder output. Why is this step necessary? As a cost, the generation speed is very slow. I saw in the appendix that generating 1k samples needs 3 minutes. What will the original BART cost?\n   2) The performance of the model heavily depends on the backbone LM. For example, BART outperforms FLAN-T5 by a large margin. What if training from scratch? What is the performance with other architectures? What if the performance of backbones varies from task, for example, in some tasks BART is better but in others FLAN-T5 is better? This kind of dependence strictly limits the generality of the model.\n\n- The experiments lack some new baselines, such as GENIE, Difformer, etc. The choice of datasets is also wired, why not conduct experiments on exactly the same ones as Diffuseq? ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We discuss our contribution in our overall response.\n\nWe would like to emphasize that at inference-time, there is no encoder output. Therefore the diffusion model is not providing a projection of the encoder output, but sampling novel latent codes. The diffusion network iteratively refines a latent code (starting from noise) to model high level text semantics and the decoder produces fluent discrete text conditioned on the latent code.\n\n**Dataset Selection**\n\nWe explore unconditional, class-conditional, and Seq2Seq language generation settings, while DiffuSeq focused only on extending Diffusion-LM to Seq2Seq tasks. We thus utilized a wider range of datasets than just those employed by the DiffuSeq paper.\n\nFor Seq2Seq tasks, we use the QQP paraphrasing dataset (also used by DiffuSeq) and the XSum summarization dataset. XSum is a well-established seq2seq benchmark widely adopted by the research community, as evidenced by its use across influential works such as BART, FLAN-T5, PaLM, and others. We found the dataset selection in the DiffuSeq paper, beyond QQP, to be non-standard and observed problems with data quality. \n\nFor query generation, the DiffuSeq authors use a dataset derived from a QA dataset with string-matching heuristics. Specifically, the context paragraphs are split into sentences and sentences that contains the answer are paired with the original question to form training instances. This often removes critical context needed to answer the question, resulting in incoherent pairings. Here are some representative examples: \n\n| Source Sentence | Target Question |\n|-|-|\n| The year 1921 in science and technology included many events , some of which are listed here | Rudolph Valentinos movie premiere was in which year |\n| The butterfly tattoo is pretty and feminine without being overly flowery or sentimental . | The is no known language without a word for this creature - what |\n\nIn these instances, the answer is in the sentence (1921, butterfly), but the context occurred in surrounding sentences. \n\nThe DiffuSeq paper also explores dialogue generation and text simplification. The Commonsense Conversation Dataset contains a lot of profanity and sexual content, and is non-standard benchmark. Furthermore, the authors discard the conversation history and use only utterance/response pairs which leads to disjointed instances. Text simplification, while reasonable, lacks the level of adoption as summarization and is related to paraphrasing, which we evaluate.\n\nWe aim to provide results on established benchmark datasets that have been widely used and vetted for quality. The datasets used in both DiffuSeq and DiffusionBERT are QQP and QG (the query generation dataset discussed above). See our response to Reviewer SqZp for QQP comparisons. We ran additional experiments with the QG dataset and report results below.  We outperform the baselines on this dataset as well and will add this comparison to the camera-ready. We will also include machine translation results on the WMT14-En-De dataset to further demonstrate the versatility of our approach (see response to reviewer SqZp).\n\n| Method | MBR | ROUGE-L |\n|-|-|-|  \n| DiffuSeq | 10 | 36.65 |\n| DiffusionBERT | 10 | 34.20 |\n| LD4LG (bart-base) | 5 | **37.07** |\n\n**Speed**\n\nAs the reviewer mentioned, using our approach will be slower than the base LM. However, there is a significant body of work focused on reducing the inference time of image diffusion models that would be applicable to our continuous, latent language diffusion model. For example, Salimans et al. [1] and Song et al. [2] propose methods to distill diffusion models to perform single-step generation. Adapting such techniques to text diffusion will drastically improve the speed of text diffusion models. See our response to Reviewer TpnG for additional timing comparisons.\n\n[1] Salimans and Ho. \"Progressive Distillation for Fast Sampling of Diffusion Models.\" \n[2] Song, Yang, et al. \"Consistency models.\" (2023).\n\n**Backbones**\n\nWe thank the reviewer for raising this important point. They are correct that the performance of our approach depends on the backbone LM used. However, this dependence is not unique to our framework - it is also true for standard fine-tuning, prompting, and in-context learning with LMs. \n\nWe focus on BART and T5 as they are well-established LMs shown to be generally effective across a diverse range of language tasks in prior work. We do observe that BART outperforms T5 on some datasets and vice versa. This highlights the value of exploring different backbone models to determine the best fit for particular applications and tasks. However, this exploration is also beneficial when fine-tuning LMs for a specific task.\n\nOur LD4LG framework can flexibly incorporate improved base models over time, which is one of the benefits of incorporating pre-trained LMs. Our experiments in response to Reviewer SqZp, for instance, demonstrates that our framework can be extended to multilingual settings by utilizing a multilingual LM.\n\nWhile we leave investigation of this to future work, we expect training from scratch to result in significantly worse performance compared to leveraging pretrained LMs, as is commonly observed. We thus leverage strong pretrained LMs to provide a high-quality starting point.\n\n**Additional baselines**\n\nWe thank the reviewer for the references. GENIE uses the XSum dataset, so we present that comparison here. The GENIE paper uses our oracle setting in their work so we report our oracle performance. We present the results for their method when using the best of 5 to match our evaluation.\n\n| Method | Timesteps | ROUGE-1/2/L |\n|-|-|-|\n| GENIE w/ pre-training (n=5) | 2000 | 41.2/19.1/33.4 |  \n| GENIE (Oracle; n=5) | 2000 | 37.3/15.3/29.4 |\n| LD4LG (BART-base) (n=5) | 250 | 42.4/19.4/36.4 |\n| LD4LG (FLAN-T5-base) (n=5) | 250 | **43.0/20.0/37.2** |\n\nWe observe that LD4LG significantly outperforms GENIE even when GENIE is pre-trained on a large external corpus.\n"}, {"review_id": "sqYqQlcTTV", "reviewer": "Reviewer_TpnG", "summary": "This work proposes a new type of diffusion language model based on latent representations. The proposed approach leverages the continuous representation space produced by pre-trained encoder-decoder language models (BART and FLAN-T5). To embed the variable-length encoder features, the author(s) propose to adopt Perceiver Resampler to construct a fixed length latent representation. After obtaining the diffused representation, a reconstruction network is used whose output is later taken by the frozen language decoder to generate the output.\n\nExtensive experiments are performed on a few types of tasks across different benchmarks. The experimental results verify that the proposed approach outperforms existing diffusion-based methods and performs comparably with directly fine-tuned language models.\n\nGood discussions on future work are included in this work.\n\n", "questions": "1. Could you elaborate more details of the language autoencoder experiments in Section 5.1? Do you train the compression and reconstruction networks to let the model replicate the input in its output?\n2. In line 268, why do you need to specify the length for inference. I think the generation length is determined by the emission of EOS token of the language decoder. Please correct me if I am wrong.", "limitations": "Please see the sections of weaknesses and questions.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper is generally well-written and the presentation is clear.\n* The proposed model is novel in terms of architectural design and motivations.\n* Extensive experiments are conducted to sufficiently evaluate the method and the results are convincing.\n", "weaknesses": "* One major weakness of this work is its potential impact to the research community in the long-term. From the presented results (Table 6), simply fine-tuning the language model still yields the best results across the board. In addition, the existing autoregressive language model such as ChatGPT and GPT-4 have proven their exceptional capabilities in various applications. On the other hand, it is hard to judge the performance of diffusion-based approaches, especially when the model is scaled up. I believe scaling up the proposed model is out-of-scope of the this work, but it worth considered by the author(s).\n* **An important comparison is missing**. From the first impression, the encoding of language encoder and the sequential decoding of language decoder are shared by both the proposed method and the standard autoregressive language model. The additional diffusion process would inevitably increase the inference latency. While the author(s) discuss the downside of inference latency in Section 6, it is important to compare both approaches systematically.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Re: Long-term Impact**\n\nWe appreciate the reviewer raising the important concern about the long-term impact of our approach. The reviewer is correct that autoregressive transformers have continued to advance rapidly, demonstrating exceptional capabilities in a variety of language tasks. \n\nHowever, we believe language diffusion models provide complementary strengths in areas like sample diversity and controllability. Our latent diffusion approach was more effective at generating novel text and more effective at generating text from a specified topic than GPT-2, an autoregressive model with nearly double the trainable parameters as our latent variable model. For the Seq2Seq tasks, our oracle evaluation demonstrated that drawing just 5 samples from the diffusion language model consistently produced a higher-quality generation than that produced by the fine-tuned language model. This demonstrates that the diffusion model has excellent coverage over the hypothesis space and that further exploration of re-ranking techniques could lead them to consistently outperform the fine-tuned language model.\n\nWhile we agree that observations made in our compute-regime cannot be extrapolated to that of state-of-the-art GPT models, promising results in modest compute-regimes are critical for motivating serious attempts at scaling. To return to the point of long term-impact, we believe that language diffusion is still in its early days as a paradigm, and further research is needed to realize its full potential. As a point of comparison, diffusion models did not surpass GANs on image synthesis benchmarks until 6 years after their original introduction. They have since become the dominant paradigm in that domain, though whether language diffusion models will follow a similar trajectory remains to be seen. \n\nWe hope our work provides a foundation for further research into language diffusion as a promising new paradigm. In this work, we are the first to demonstrate that diffusion models can be integrated with existing pre-trained language models for high quality language generation. We are particularly interested in scaling up hybrid approaches such as the one demonstrated in this work to recent open source autoregressive language models such as LLama.\n\n**Re Auto-encoder training details:**\n\nYes, we train the auto-encoding modules such that the compact features produced by the compression network are mapped to features that guide the language decoder, through the cross-attention mechanism, to generate the input language. We train the autoencoder with the standard cross-entropy loss to re-generate the input language from the bottleneck representation.\n\n**Re line 268:**\n\nFor the BART-diffusion baseline described in 268, we explore learning a diffusion model directly in the feature space produced by the pre-trained BART model. This enables us to explore the benefits of learning our language autoencoder. Because language varies in length, the size of the BART latent diffusion space also varies with the sequence length of the text. For this baseline, a sequence length l must therefore be selected to generate samples. In contrast, our method LD4LG has a latent space of a fixed size. As the reviewer mentioned, LD4LG is therefore able to generate sequences of different lengths by solely relying on the decoder’s emission of the EOS token. The comment about specifying the length for inference only applies to the BART-diffusion baseline.\n\n**Re Inference Latency:**\n\nWe appreciate the reviewer raising this important point. They are correct that our method introduces additional overhead compared to directly using the pretrained language models. The reviewer rightly emphasizes the need to quantify the inference latency trade-off through direct comparison to the fine-tuned language models. \n\nOne of the strengths of our approach is that it enables the adaptation of advanced techniques for accelerating inference from other continuous domains such as images. There is already a significant body of work focused on reducing the inference time of image diffusion models. Improved sampling algorithms such as DPM-Solver++ [1] also enable the use of fewer timesteps at inference than the traditional DDPM sampler used in our work. To illustrate this, we report the time needed to generate 1000 validation summaries for the fine-tuned language model, FLAN-T5, and our approach on the XSum dataset using the traditional DDPM sampler employed in our original submission as well as the newer DPM-Solver++.\n\n| Method | Sampler | Timesteps | Seconds | ROUGE-L |\n|-|-|-|-|-|\n| FLAN-T5 | Beam | n/a | 26.6 | 31.7 |\n| LD4LG (FLAN-T5) | DPM-Solver++ | 10 | 73.3 | 29.6 |\n| LD4LG (FLAN-T5) | DPM-Solver++ | 15 | 82.5 | 29.9 | \n| LD4LG (FLAN-T5) | DPM-Solver++ | 20 | 90.8 | 30.2 |\n| LD4LG (FLAN-T5) | DDPM | 250 | 547.6 | 30.7 |\n\nWhile the added latency during generation remains a limitation of our approach, we observe that the recently developed DPM-Solver++ sampler can accelerate our framework by over 5x, although performance does degrade slightly. We are excited about extending other advanced techniques, such as progressive distillation [2] or consistency models [3], which have successfully distilled image diffusion models to perform high-quality generation in a single step models to language.\n\n[1] Lu, Cheng, et al. \"Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models.\" (2022).\n[2] Salimans and Ho. \"Progressive Distillation for Fast Sampling of Diffusion Models.\" \n[3] Song, Yang, et al. \"Consistency models.\" (2023).\n\n"}]

Author rebuttal:

We would like to thank the reviewers for their thoughtful feedback on our paper. In particular, we appreciate Reviewer SqZp highlighting that the “utilization of pretrained encoder-decoder models to establish a feature space is a clever and efficient approach”. We also thank Reviewer 5zaG for noting our submission “stands out by incorporating an autoregressive decoder into diffusion models, which combines the strengths of both methods” and “significantly enhances the quality of text generation, distinguishing it from existing diffusion-based language models.” As reviewers T1gz and TpnG point out, we perform comprehensive experiments across unconditional, class-conditional and sequence-to-sequence settings to demonstrate the effectiveness of our method.

We have responded to each reviewers' valuable critiques in our individual responses and look forward to incorporating their feedback into the final version. We hope to continue this valuable discussion during the discussion period. 

We also present a general comment here regarding the motivation, long-term potential, and contribution of this work to address points brought up by Reviewers T1gz and LWig. 

We believe diffusion models provide complementary strengths to autoregressive generation in areas like sample diversity and controllability. Our latent diffusion approach was more effective at generating novel text and more effective at topic-guided generation than GPT-2, an autoregressive model with nearly double the trainable parameters as our diffusion model. For the Seq2Seq tasks, our oracle evaluation demonstrated that drawing just 5 samples from the diffusion language model consistently produced a higher-quality generation than that produced by the fine-tuned autoregressive language model. This demonstrates that the diffusion model has excellent coverage over the hypothesis space and that further exploration of re-ranking techniques could lead them to consistently outperform the fine-tuned language models.

Although latent diffusion is commonly used for images, its extension to language presents unique challenges given the discreteness of the domain. Latent diffusion models are reliant on the strength of the underlying autoencoder, and we introduce a method for effectively and efficiently adapting existing pre-trained language models into strong language autoencoders. As multiple reviewers acknowledge, we are distinctive from other diffusion language models in developing a hybrid approach utilizing the complementary strengths of diffusion and autoregressive generation.

As for the potential for impact, we believe that language diffusion for language is still in its early days as a paradigm, with research needed to realize its full potential. As a point of comparison, diffusion models did not surpass GANs on image synthesis benchmarks until 6 years after their original introduction. They have since become the dominant paradigm in that domain, though whether language diffusion models will follow a similar trajectory remains to be seen. We hope our work provides a foundation for further research into language diffusion as a promising new paradigm. In this work, we are the first to demonstrate that diffusion models can be integrated with existing pre-trained language models for high quality language generation.

Meta-review: This paper presents an interesting approach to combine diffusion models with encoder-decoder language models in order to learn high-quality language autoencoders. The authors demonstrate the effectiveness of continuous diffusion models in the latent space of the language autoencoder, resulting in continuous latent representations that can be decoded into natural language. The proposed method is validated across various language generation tasks and diverse datasets, showing superior performance compared to previous diffusion language models. For reviewer T1gz's concern, I totally understand it but think the motivation question is a common challenge for the research on diffusion model topics. This does not make it less interesting for a wider group of audience in the community. Therefore, I would recommend to accept this paper.
Decision: Accept (poster)