PLANNER: Generating Diversified Paragraph via Latent Language Diffusion Model
Authors: Yizhe Zhang
Jiatao Gu
Zhuofeng Wu
Shuangfei Zhai
Joshua M. Susskind
Navdeep Jaitly
Venue: NeurIPS 2023 poster (2023)
Keywords: Text generation
diffusion model
NLP
TL;DR: A latent text diffusion model which improves the generation diversity and reduces repetition in generation.

Abstract: Autoregressive models for text sometimes generate repetitive and low-quality output because errors accumulate during the steps of generation. This issue is often attributed to exposure bias -- the difference between how a model is trained, and how it is used during inference. Denoising diffusion models provide an alternative approach in which a model can revisit and revise its output. However, they can be computationally expensive and prior efforts on text have led to models that produce less fluent output compared to autoregressive models, especially for longer text and paragraphs. In this paper, we propose PLANNER, a model that combines latent semantic diffusion with autoregressive generation, to generate fluent text while exercising global control over paragraphs. The model achieves this by combining an autoregressive "decoding" module with a "planning" module  that uses latent diffusion to generate semantic paragraph embeddings in a coarse-to-fine manner. The proposed method is evaluated on various conditional generation tasks, and results on semantic generation, text completion and summarization show its effectiveness in generating high-quality long-form text in an efficient manner.

PLANNER: Generating Diversified Paragraph via Latent Language Diffusion Model
Yizhe Zhang, Jiatao Gu, Zhuofeng Wu, Shuangfei Zhai, Josh Susskind, Navdeep Jaitly
Apple
(yizzhang, jgu32, zhuofeng_wu, szhai, jsusskind, mjaitly)oapple.com com com
Abstract
Autoregressive models for text sometimes generate repetitive and low-quality output because errors accumulate during the steps of generation. This issue is often attributed exposure bias the difference between how model is trained, and how it is used during inference. Denoising diffusion models provide an alternative approach in which model can revisit and revise its output. However, they can be computationally expensive and prior efforts on text have led to models that produce less fluent output compared to autoregressive models, especially for longer text and paragraphs. In this paper, we propose PLANNER, model that combines latent semantic diffusion with autoregressive generation, to generate fluent text while exercising global control over paragraphs. The model achieves this by combining an autoregressive "decoding" module with a "planning" module that uses latent diffusion to generate semantic paragraph embeddings in coarse-to-fin manner. The proposed method evaluated on various conditional generation tasks, and results semantic generation, text completion and summarization show its effectiveness in generating high-quality long-form text in an efficient manner.
1 Introduction
Horrible
hotel. Serioulay,
1-gram
2-gram
3-gram
FT (Gready decoding):
The hotel was nightmare. The rooms were fitty, the bathrooms were disgusting. disgusting. The food was disgusting. The food was disgusting. The food was disgusting...
filthy, was disgusting,
disgusting.
100000
100.00
FT (Top p sampling): Terrible, awful hotel, Horrible, awful hotel, I'm going to the awful hotel. I'll never get in. What hell this hotel? awful. don't Where? It's awful. think they have a bethroom there. I'm going to the ewful hotel. I'll never get in. want out of here. horrible, going
U113 sox
ioarible like bordl Munrisle
Tar Me tutel 8.5%%
Homrillc hutel is 2.t
8.2%%
This 8i2 2::2.22%
Terribe hotel Tembber:2 17%,
Fer Terikees 5.1%%
Hote 1. 61%
Tentille hotel Hon a01
get
4.33%
Imeam 2.1%%
Lsculd
Ours (Gready decoding): Staff was unhelpful. So rude from the front desk. The hotel tried to lose the reservation of family which had with the hotel. This hotel way overpriced. Hellways smelled bed. Carpet was disgusting. I'll never go back. don't recommend the hotel. stay anywhere elsel!
E 55%
Finoi 1.565
Feeroom wax 1o0k
Tre:
The vea
kcalled 6.78%)
2.72%
Icaless: 2.17%
Bookeed throughtSS.o 5.70%
Figure I: Left: With repetitive prompt, the finetuned GPT-2 large model (774M, FT) still attracted to self-reinforeed repetition (highlighted text) even under top-p sampling (K=50, p=0.92), Right: the most frequeett first n- grams of the generations for each method, derived from 512 generation roll-outs. Our proposed method results in more diversified generation robust to the ill-composed prompt.
Autoregressive models trained with a teacher foreing strategy (Williams Zipser 1989 are con- sidered the gold standard for text generation. However, significant drawback of this approach is that it lacks the ability to correct any mistakes made during the generation process which can lead to emror that accumulate as the generation progresses. Previous work (Ot et al. 2018 Holtzman et al.) 2019 Welleck et al. 2019 Xu et al. 2022) has observed that deterministic decoding methods have
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

tendency to generate consecutive repetitions at the word, phrase and sentence levels. For example, with repetitive prompt, the model can enter an absorbing state where produces repetitive outputs with higher and higher confidence by self-reinforcing the pattern (Xu et al. 2022) (Fig. I Through our experiments, we have observed that such degeneration is more prevalent in open-ended tasks that allow the model greater freedom for creativity. Even for large language models, the generation can drift away from the desired semantics, especially when the model is poorly prompted or has high initial probabilities (Xu et al. 2022)
Why does using maximum likelihood decoding lead to repetitions during generation which is significantly different from the training data distribution? One possible explanation for this is "exposure bias" (Bengio et al. 2015) arising from the discrepancy between the training and inference phases in the teacher forcing traming strategy. During training phase, the model focuses only on predicting the next token. However, during inference, predicting the next token alone can myopic because the model may not have enough foresight to anticipate its impact on future generation. This can also be seen as the "distribution shift" issue of behavior cloning (de Haan et al. 2019), where the model is trained to mimic the expert's s actions training data. However, small differences between the model and the expert can compound over multiple steps, leading the model to states it never encountered during training, rendering unreliable and undesirable predictions.
the states encountered by the expert in the
Although many approaches have been proposed to address this issue, such as adversarial models Yuu al. 2017| Lamb al.) 2016 Zhang et al., 2017). reinforcement learning (Li al. 2016b) or
repetition penalties (Xu et al. 2022 they attempt to improve the global aspects of the generation by making local adjustments which still follow the autoregressive generation recipe. Diffusion models provide an alternative solution the model can revisit and revise its output iteratively, potentially rendering more global control of the generation in a non-autoregressive manner. However, these text diffusion models can generate less fluent text compared to autoregressive ones (Gong et al. 2023). Also, when generating long text, the diffusion process involves multiple passes of the underlying denoising model over a long generation length, making it computationally expensive. The discrete nature of text also presents a challenge for diffusion models, which can suffer from "rounding errors" "
when converting between the text token and its embedding (Li et al. 2022 Lin et al. 2022).
Instead of performing diffusion on the original text or the corresponding word embeddings, we propose to apply diffusion techniques to the latent semantic space (Rombach et al. 2022 Lovelace et al. 2022). To achieve this, we learn a fixed number of continuous semantic tokens that encode salient information at the paragraph level. These tokens can then be used to reconstruct the original text. The latent diffusion can be additionally conditioned on an external signal to generate the semantic tokens. Finally, a decoder maps the obtained semantic tokens back to the raw text space. This process combines non-autoregressivve semantic diffusion approach with an autoregressive decoding technique. The semantic diffusion process handles the "planning", enabling the modification of semantics in a coarse-to- to-e manner, while the decoder handles the "decoding" by translating the semantics into raw text, with less flexibility in controlling the meaning. We call our proposed method PLANNER (Paragraph-level.. DiffusioN model for Embedding Representation).
Our contributions include: (1) We propose a latent semantic diffusion model for paragraphs that incorporates both aon-eetoreeressivve semantic diffusion and autoregressive generation. This allows us to generate fluent text while being able to exercise global control inherited from diffusion model. ii We study the essential requirements for a good latent space for paragraph diffusion models. We evaluate the effectiveness of our proposed method on various conditional generation tasks. Thanks to the iterative refinement of desnoising diffusion, our method enjoys less repetitive and more diverse generation, while maintaining good fluency and relevancy, comparing with autoregressive and text diffusion basclines LLi et al. 2022
Lin ct al. 2022).
2 Preliminary
Diffusion Probabilistic Models The standard diffusion model (DM) (Ho et al.) 2020 Song &
Ermon 2019) learns the data distribution p(x) by gradually denoising normally distributed variable in Markov chain of length T. The diffusion process can be viewed as continuous-tine stochastic process (ong et al. 2021b Kingma et al. 2021) where the initial data point & e R.N is progressively corrupted by noise according to predefined signal-noise schedule (at, ot), resulting in time-
2

---

dependent corrupted data {æylt [0, x0 xo = æ). The transition distribution is given by:
<x,))x,)) (z;;; 0tp oQ.
(1)
where Orti = =0/0as lls 0z a and t. When æ, æ. the marginal distribution (<>)(a)))
is given as (=)) N (x;; a 1).). The diffusion model relies on a parametric function
optimized to reverse the diffuusin process by denoising 2 the clean input æ. The model is trained
using a weighted reconstruction loss:
((0) = E En (=.))))))) [ F t) x: x=[3]
(2)
where wt = a (s.t. of + of 1) is the signal-to-noise-raul (SNR) and (-).) denotes the backbone denoising function. Sampling from the learned model can be performed using either ancestral sampling (DDPM) (Ho et al. 2020 or deterministic DDIM sampler (Song et al. 2021a). While the DM is capable of generating high-quality samples, the fact that the corrupted data 2; shares the same space as the input results in inefficient training (Jing et al. 2022) and difficulty in learning abstract and semantically meaningful latent spaces (Preechakul et al. 2022h.
To improve the efficiency, the Latent Diffusion Model (LDM) (Rombach) at. 2022) introduces an explicit separation between the compressive and generative learning phases of training diffusion models. It employs an autoencoding model consisting of an encoder E-) and a decoder (-) to learn a low-dimensional latent space that is perceptually equivalent the image space when decoded, but with reduced computational complexity, while retaining the perceptual quality of generated samples. The reweighted objective for training LDM is given by:
Latent Diffusion Models
((0)= =2)) [we [ =1]]]
(3)
where z is obtained from & during training. The generated can be decoded to image using D.
3 Related Work
Text diffusion models Early attempts on using diffusion models for discrete data used noising
processes which masked or randomly mutated the discrete tokens (Austin et al., 202 [] Hoogeboom
al. 202T). Recently, Diff-LM (Li et al., 2022) and DiffuSeq (Gong et al. 2023) have instead used a continuous token embedding space, converting the continuous token embeddings to text via "rounding". Analog Bits (Chen et al.) 2022) converts raw text into a set of bits and models them as analog bits with continuous diffusion model. (Lovelace et al. 2022) performed diffusion model on the contextualized BART embeddings rather than on the word embedding space. Zhu et al. 2022) has applied text diffusion to magge-captionigg and achieved good performance.
However, existing text diffusion models present several issues: (i) The varying length of the input text necessitates the prediction of additional length or superfluous paddings, and (ii) token generation in parallel may result in disfluent text and/or frequent repetitions especially when the generation is long. We instead employ the diffusion model to learn paragraph embeddings that contain fewer fixed-sized tokens, which allows for computational benefits and improved fluency.
Text Variational Autoencoders Text VAES (Bowman et al. 2016 Kim et al. 2018 Li et al. 2020 have been particularly useful for learning smooth and interpretable representation space, as well as for generating diverse text. However, one of the challenges is the KL vanishing problem (Bowman et al., 2016), which results in the decoder disregarding the latent code sampled from the prior distribution during the inference stage. Our approach can be perceived as to address this issue by leveraging more flexible prior distribution ensure the codes can strongly influence the output text distribution.
4 PLANNER: A Language Diffusion Model on Paragraph Embeddings
We use latent diffusion to improve the diversity and fluency paragraphs generated from the model. Our model comprises two parts (Fig. 2 a paragraph embedder via variational autoencoder (VAE) that learns meaningful and smooth latent space that corresponds to the original text space, and diffusion model that generates latent codes corresponding to the semantics of longer text.
3

---

Variational Paragraph Embedder
Latent Diffusion Model
2
2
N
Input
Time Embed
Norm
t-0.9
Transformer Model
As
Scalo,
%
X
X
21
2,
Multi head Attention
Scalle
Transformer Model
2
BOS
%
V2
AV
Layer Norm
BODDD X
Scali,
D
D
Pointwise FFN
Scale
X
EOS
bork avoarene
The Ront of Scottand has, srssunced IE close 23 branches Sentlend.
Model overview. Left: variational paragraph embedder is learned to encode paragraph into a fixed amount of latent codes. Right: the latent diffusion model based on transformer block is applied to generate the latent codes. The decoder finally translates them into the text. (BOS: Begin of Sentence token, EOS: End of Sentence token)
Figure 2:
4.1 Learning a Variational Paragraph Embedder
Instead of applying diffusion to tokens directly generate long text, we propose to learn set of latent codes = {z(1),.., 2²()) } Rkxh, which we call paragraph embeddings, that capture the semantics in the target text (of length up to 512 tokens), where h denotes the embedding dimension. These paragraph embeddings have shorter length, such as k 16, than the original text.
To obtain such embeddings z, we train a transformer-basee encoder-decoder model. The architecture used for the autoencoder is shown in Fig. 2 The encoder E and decoder D construct bidirectional mapping between the discrete data space and the latent code space. The paragraph embeddings = are extracted by taking the first & hidden state vectors of dimension h from the final layer of E, which fed into the initial steps of the decoder which is trained to reconstruct the original text. It's worth noting that the paragraph embeddings share the same hidden dimension h as the word embeddings, and forming a manifold in the word embedding space. Pretrained BERT and GPT-2 models are used initialize & and D. respectively. The manifold of the learned embeddings ideally possesses several desirable properties, including low conversion error, local smoothness and distributional smoothness.
Conversion error Ideally, the original input æ can be perfectly reconstructed via å = Exx), and modeling the lower-dimensional continuous space p(z) is equivalent to modeling p(x)). However, in practice loss of information can occur when converting raw text into paragraph embeddings or when doing the reverse. We assess the conversion loss by computing the BLEU score (BLEUcIean) between the input æ and the reconstruction ét
Local smoothness To generate target text that is fluent and consistent with the corresponding paragraph embeddings, it is essential to achieve certain level of local smoothness in the paragraph embeddings space. Ideally, slight variation in the input vector would not cause a significant change in the resulting encoded vector Ex)). Similarly, a small perturbation in the latent vector should not lead to significant change in the decoded vector D (()). Otherwise, the error accumulated in the diffusion process when generating z could result in an inaccurate realization of the desired semantics. To accomplish this, the denoising autoencoder is trained by substituting (Sub) input tokens with random tokens with probability p. The local smoothness measured using the BLEU score (BIEU between the input æ: and the denoising output from corrupted input ((E())) where Sub(ar,p= = 0.3). The level of injected noise will affect both the conversion error and the local smoothness, and it is important to strike balance between the two.
Distributional Smoothness The diffusion model may face difficulty in learning a distribution, p(2).
that is highly multimodal, or has density that are associated with large Lipchitz constant (i.e., has abrupt changes). Therefore, we employ a text VAE (Bowman et al. 2016 Li et al. 2020) to encourage the posterior to take on a form close a Gaussian distribution. Specifically, we parameterize yf (=(xz) to be (E,, E,) and maximize the objective L(E, D; .)) (e [log p()[=)] 3 KL ((2)))) ()m)))((=)). Here En and E, represent the posterior mean and variance predictions of the encoder E, while the

---

hyperparameter 3 controls the strength of regularization. It is typically set to a small value to alleviate the notorious posterior collapsing issue (Bowman al. 2016) in text VAE. To gauge the distributional smoothness of the paragraph embedding space, we select two examples, x and æ' at random from the training set and interpolate their embeddings to compute Slur = _[(x)) + "=('). We then evaluate the perplexity (PPList) of the decoded interpolation D ((IIT) using a GPT-2 model.
4.2 Planning then Decoding: A Latent Diffusion Model for Paragraph Embeddings
Training phase We now use the learned mean paragraph embeddings z = En (xx) to train a continuous-time latent diffusion model as in Fig. 2 while keeping & and DD frozen. We conducted experiments using two types of conditioning signal: (i) class labels, such as positive or negative sentiment labels, and (ai) raw text, such as preceding context or the document to be summarized. For class labels. we learned label embedding y R to represent each class. For the raw text, applied a conditional feature encoder 5 to the input and used the hidden states at the last layer as y e Roxh where e represents the number of feature embeddings.
During training, we gradually add noise to z via a cosine scheduler (Ho et al. 2020). and use a signal prediction scheme as the training objective (Kingma ct al., 2021). For our denoising backbone model F()). we use a transformer block similar to the one in the DiT (Peblles & Xie 2022 model. Specifically, we fed y and the time embedding t é IRAA into the model through two channels, namely cross-attention and adaptive layer norm (adaLN) (Peebles & Xie, 2022). For the cross-attention, the conditional embeddings and y are concatenated into a sequence of length e++ 1. The transformer block is modified to enable multi-head cross-attention to the conditional embeddings.
For the adaLN, we flattened and projected y to a vector of R using a linear projection layer. We then added the projected y to the time embedding t. Instead of directly learning dimension-wist scale and shift parameters (33 and 7) in the standard Layer Norm (LN), these parameters are regressed from the sum of the embeddings. In addition, dimension-wis wise scaling parameters a are regressed and applied immediately before any residual connections within the transformer block. This has been shown to be efficient and effective in image diffusion models (Peebles & Xie 2022).
Inference phase During the inference process, we start with random Gaussian embeddings and use a fixed number of steps I" to generate the final z. The resulting embeddings are then used as inputs for D to generate the text using a deterministic decoding method like greedy decoding We provide discussion and ablation study on using stochastic decoding in the App- C
We applied the classifier-free guidance (CFG) (Ho & Salimans) 2021 during the inference steps.
After each step, we apply a dynamic thresholding technique that was introduced in Imagen (Saharia al. 2022) for post-processing. However, we do not use the rescaling step in Imagen because rescaling step can completely alter the underlying semantics, as we have not imposed any constraints to ensure that the generated output remains the same after rescaling the latent code. In contrast, for Imagen, where the generation takes place in the raw pixel space, rescaling will predominantly retain the shape information while altering only the contrast and brightness.
Experimental Setups
We tested the effectiveness of our model in three different conditional generation tasks including sentiment-guided generation, long-form text completion, and summarization. The tasks can require generating text of hundreds of tokens in length, making them suitable to assess model performance.
Datasets For the Sentiment-guided generation task, used the Trip dataset provided by et al. 2014). By exclusively selecting reviews with rating of I or 5 and balancing the two ratings via subsampling, we acquired 218,509 reviews. For the text completion task, our model was assessed on two datasets: 1) the aforementioned TripAdvisor review dataset with postprocessing to remove reviews that are less than 20 or more than 512 tokens, result in 690,862 samples, and 2) one-tenth of the overall C4 datasets (Raffel et al.] 2020). which contains 36.5M samples. For each sample, we extracted the initial two sentences from a paragraph as source context, and predicted the remainder of the text as target. The datasets were partitioned into training, validation, and test in the ratios of
'The aim of D is to accurately convert the into meaningful text, thus deterministic decoding is desirable.
5

---

(0.96, 0.02, 0.02). For the summarization task, we use CNN/DailyMail XSum Narayan et al., 2018).
(Hermann et al. 2013 and
2023), we assess the fluency of the generation by computing the perplexity (PPL) under a GPT-2 large model. We use Ent-n Zhang et a 2018) and DIST-n (Li ct al. 2016a) and self-BLEU (S-BL) (Zuu et al. 2018 to evaluate lexical diversity. We present DIST-n and Ent-n metrics solely at n 1 owing to their strong correlation of the varying n values. We use REP-n to assess the extent of repetition in the generation
Automatic Evaluation Metries Following previous work (Gong et al.
following previous work (Welleck et al. 2019; Xu et al. 2022). For relevancy we use standard metrics following (Gong et al. 2023). including Sacre (BL) (Post 2018), ROUGE-L (R-L)
Lin 2004) and BERTSscore (Score) (Zhang al. 2019). Details are provided App.
AuBLEU: Evaluating Denoising Capability Our proposed model is a latent diffusion model, which differs from text diffusion models that operate directly on text or text embedding space. To comparing the denoising ability across different text diffusion models, we introduce a novel metrie, named AuBLEU (AuBI). To compute the AuBLEU score, we first add varying levels of noise to each input text æ by performing diffusion at T different time steps to < t1 <... < tr, corresponding to a series of SNR w > Wty. Next, we pass each corrupted input under different model and obtain the predicted output è;- = F(x,,) ). We then compute the BLEU score between each (i; 2) pairs and plot a curve with the x-axis representing a² -ي where a² is monotonically increasing with w and ranges from (0, 1). and the y-axis indicating the corresponding BLEU score. Finally, we compute the area under curve to obtain the AuBLEU score (see App. for more details).
the denoising backbone
Itw
Model Setups We used the BERT-large and GPT-medium models as initialization for the encoder E and decoder D respectively. The embedding dimension h was 1024, and the number of paragraph embeddings & was set to 16, as increasing the number did not result in significant improvement in performance. We provide more analysis to 2e - 4, and (3 was set to 5e - 6. For the latent diffusion model, the channel size was set to 1024 to match the embedding dimension h, and the number of heads was set to 16 with 28 transformer layers. The total size of the latent diffusion model was 533M The feature encoder T was also jointly learned, and was initialized with a T5-large encoder. We use DDIM throughout our experiments as it shows better performance than DDPM. In all our experiments, we use 30 diffusion steps to generate the final z which strikes a good balance among the efficiency, diversity and relevance. In comparison, Diff-LM (Li et al. (022) and Genie (Lin et al., 2022) report to use 200 steps and 2000 steps respectively to generate high-quality text. We set the CFG weights to be and 5 for text completion and summarization tasks, respectively, based on generation performance on validation set. For summarization task, we also incorporate a shift noise scheduler based on (Hoogeboom et al., 2023). More details, including ablations on DDPM, number of diffusion steps and noise scheduler, are provided in App. F
the impact of l: in App. A.2 The learning rate was set
Baselines We compare our method with several baseline methods trained under Teacher Forcing scheme, includ- ing deloder-only Autoregressive LM finetuned on GPT-2 (FT), encoder-decoder (Enc-Dec) transformer model, and Varitional Information Bottleneck (VIB) (Alemi et al. 2016). We initialized the FT model using GPT-2 large (774M), whereas encoder and decoder in the Enc-Dec/VIB models (695M/697M) are initialized with bert-large and GPT-medium, respectively. All the considered models are finetuned on the target datasets. We follow Li et al. (2022) to report the FT basclines with two decoding strategies, top-p sampling (K=50, p=0.92) and beam search (beam width 4), denoted as FT- FT-sample and FT-search. We use top-p sampling for Dec/VDe generation. For summa- rization tasks, we finetune a Ta-large model (770M) on the target datasets as baselines. We also compared two text diffusion models Diff-LM and Genie using their suggested configuration from the official repository. More details are in App. F
100
100
80
80
Perplexity
60
60
40
BLEU clean BLEU robust
int
0.0
0.2
0.4
0.6
Figure 3: Impact of the proportion of in- jected noise for learning Paragraph Em- beddings on XSum dataset. Large substi- tution noise results in worse BLEU but better BLEU and PPL

---

6 Results
6.1 Paragraph Representation Learning
It is essential to learn a paragraph embedding space that is both accurate and smooth. To this end, we examined the effect of various substitution probabilities to the input tokens æ. Our findings, presented in Fig. 3 reveal that a smaller p results in a lower conversion error, as indicated by a higher reconstruction BLEU (BLEUctean). albeit at the expense of local smoothness (BLEUrobuat) and distributional smoothness (PPLint). We performed grid search of p with 0.1 increment based on 512 samples. Empirically, we observed that weighted score Sowerll 0.5BLEU + O.8BLEU robust O.3PPL correlate well with downstream diffusion performance, leading to fluent and accurate generation for PLANNER (see App. A for more details). We finally opted for = 0.3 for most datasets which strike a balance between conversion error and smoothness.
is worth noting that there is an inevitable conversion loss, indicated by the fact that the BLEU between 77 87 when generating hundreds of words (App. D). We observd that most of the lexical mismatch still maintain the similar semantics, with the exception of some name entity swapping. We show some paragraph reconstruction and denoising examples from our paragraph embedder in the App. A We also include examples of interpolated generation from random paragraph pairs in the App. A In general the transition of semantics is natural, indicating reasonable distributional smoothness of the paragraph embedding space.
6.2 Sentiment-Guided Generation
For sentiment-guided gen- eration experiments, fol- lowing previous works (Li
Arch.
PPL
ACC1
DISTVENTY
S-BLI
Rep-41
Len
FT Fa-sapple
20.86
70.2%
0.13/6.154
0.96
5.86%
113
al. 2022 Hu et al. 2017 Keskar et al. 2019). we use
Diff-LM
101.97
83.6%
0.15/5.115
4.05
6.23%
66.2
Ours
51.12
94.9%
0. 0.16/6.360
0.77
2.37%
161
a trained classifier to assess
Human
47.94
96.7%
0.17/6.737
0.48
2.17%
157
if the given sentiment is well-controlled in the gen- eration. The trained classi- fier initialized with BERT- large and finetuned on the training set, which yields an accuracy of 96.75% on the held-out test set. The results are provided in in Tab. [ PLANNER outperforms the baseline approaches in generating long reviews at higher levels of accuracy. Although PLANNER using greedy decoding mode is at least comparable with FT with top-p sampling in terms of diversity, and has lower repetition as assessed by Rep-4 in generation.
Table 1: PLANNER achieves high success rate (ACC) and diversity with less repetion in generating hotel reviews conditioned on sentiment.
The perplexity of the text generated by PLANNER is close that of examples of the generated text in App. E Interestingly, as shown in App. E with the same random seed but different controlling sentiment, PLA LANNNRR generates text with similar contents but different sentiments, suggesting the diffusion model may be able to disentangle the semantic space to certain extent. Unlike the autoregressive generation, the nature of the diffusion model allows the model to "regret" and iteratively refine the current generations. In App. B we demonstrate how the generation evolves over multiple time steps in a coarse-to-fine manner in PLANNER.
text. We provide
6.3 Long-form Text Completion
We further evaluate our model on the long-form text completion tasks. For text diffusion baseline, we compared our method with Diff-LM (Li et al. 2022) on hotel review dataset. We could not perform comparison on the C4 dataset with Diff-LM due to the significant amount (thousands) of GPU hours required to train Diff-LM adequately. A Diff-LM running time estimation is available in App. F The results are provided in Tab. 2 FT-search performed poorly in this open-ended generation task as its generation exhibited high repetition levels, consistent with findings in previous research (Holtzman ct al. 2019; Xu et al., 2022). Although our approach also employs a deterministic decoding method, observed that it produces text with low Rep-4 metric, signifying that PLANNER is effective in
"except for CNNDM dataset where p 0.5
7

---

Arch.
PPL
DIST/ENTI
S-BLI
Rep-41
BLÍ
R-L1
Scoret
Len
AuBLi
Hotel Review dataset
FT-search
1.87
0.03/4.865
3.50
86.60%
0.62
5.2
0.39
179.51
Fa-sapple
15.51
0.14/6.455
0.88
4.49%
0.78
6.8
0.53
164.50
Enc-Dee
33.82
0.18/6.379
0.57
3.25%
0.47
7.3
0.54
94.03
VIB
36.89
0.19/6.481
0.54
3.15%
0,45
7.1
0.54
86.11
Diff-LM
178.30
0. 0.13/5.560
3.57
4.54%
0.84
8.8
0.43
175.10
26.16
PLANNER
47.36
0.17/6.602
0.52
1.55%
0.77
7.9
0.55
168.08
38.55
Human
47.60
0.20/7.023
0.60
1.46%
181.29
subset dataset
FT-search
1.927
0.07/6.245
0.14
79.54%
0.77
5.2
0.37
154.88
FT-sample
12.244
0.25/7.136
0.44
7.01%
1.59
5.9
0.47
122.55
Enc-Dec
23.095
0.24/7.077
0.16
2.27%
1.92
7.5
0.5
118.07
VIB
19.701
0.24/7.003
0.16
2.62%
1.86
6.8
0.49
113.34
PLANNER
61.768
0.28/7.352
0.12
1.67%
2.04
7.7
0.51
111.89
36.77
Human
59.783
0.44/7.381
0.12
1.12%
107.56
Table 2: PLANNER enhances the diversity of text generation and minimizes the occurrence of repetition in open-ended text completion tasks.
reducing repetition through holistic iterative refinement throughout the inference steps the diffusion process. Comparing with Diff-LM and other baselines, PLANNER achieved better diversity scores while maintaining comparable relevance scores. We also observe higher AuBLEU of PLANNER comparing with Diff-LM, indicating potentially higher overall denoising strength of PLANNER (See App. D for more details). Some examples of the generated text are available the App. We also observed PLANNER exhibits robustness towards prompts that are either repetitive or ill-composed, where FT failed (Fig.
D App. G
We further performed pairwise human evalua- tion on 300 examples hotel review generation from each system our internal crowd-source annotation platform. Each pair of text being pre- sented to judges in random order. The judges ranked the pairs for relevance, informativeness and human-like properties using a 3-point Likert- like scale, Overall judge preferences are shown in Table 3 A moderate preference can be ob- served for PLANNER over FT and VIB, except for human-like between PLANNER and VIB. We also observe that judges still prefer human responses over system generations in this task. Further details, including the human evaluation template used and interrater agreement analysis, are provided in the App. H
Metric
Methods
Win
Tie
Loss
Ours vs. FT
48.2%
9.2%
42.6%
Rel.
Ours VIB
50.7%
10.0%
39.3% #*
Ours vs. Human
39.3%
9.3%
51.3% 4*
Ours vs. FT
55.1%
5.7%
39.2% w*
Inf.
Ours vs. VIB
48.7%
8.0%
43.3%
Ours vs. Human
37.7%
8.7%
53.7% #58
Ours vs. FT 51.5%
8,4%
40.1 **
Hum.
Ours vs. VIB
40.0%
19.3%
40.7%
Ours vs. Human
34.3%
17.0%
48.7% 4*8
Table 3: Human evaluation on Relevance (Rel.). In-
formativeness (Inf.), and Human-likeness (Hum.). Statistical significant results: ** p 0.001, * 0.01.
6.4 Summarization
We further conducted evaluation summarization and present the results in Tab. Summarization is less open-ended than the text completion task, thus deterministic decoding approach like T5-search can produce high-quality text. Our evaluation shows that in comparison with T5-sample and Genie (Lin et al. 2022), PLANNER exhibits comparable Rouge-L scores, while improves other metrics. PLANNER achieves higher AuBLEU than Genie (See App. for more details).
diffusion model, PLANNER and Genie yielded lower Rouge-L scores in comparison with T5-search, with single summary. To align with Genie's evaluation, we provide the results with random runs in Tab, Ei where for each document generates summaries and the with the highest Rouge score is used. However, note that these summaries with
Owing to the sampling nature

---

best Rouge-L cannot be predetermined without an oracle summary. Comparing with se-search, PLANNER generates more diverse and less repetitive summaries. However, the improvement is less conspicuous comparing with the results observed in open-ended text completion tasks. We show some generations in the App.
+
(Tab. T2.
Notably, the generated content may occasionally include hallucinations or errors, especially for name entities and digits (App. Tab. Such occurrences can be attributed either the conversion errors in D or errors during the generation of paragraph embeddings, and requires further investigation.
Arch.
PPL
DIST/ENT
S-BL
Rep-41
BL1
R-L
Scoret
Len
AuBL
CNN Dailymail dataset
T5-search
58.12
0.11/7.726
0.24
6.69%
7.66
34.48
0.66
45.51
Ta-sample
67.58
0.11/7.790
0.20
3.50 %
5,05
30.15
0,64
48.51
Genie
179.9
0.09/7.293
0.24
4.16%
3.22
30.47
0.58
40.94
27.21
Genie (10)
170.6
0.10/7.355
0.24
4.32%
6.48
37.09
0.62
40.81
PLANNER
49.21
0.10/8.037
0.15
5.25%
6.92
30.43
0.62
52.33
43.91
PLANNER (10)
49.07
0.10/8.019
0.15
4.96%
11.42
36.81
0.66
53.14
Human
49.477
0.12/8.226
0.16
5.63%
51.15
XSum datasct
T5-scarch
29.41
0.12/7.200
0.31
14.83%
6.11
36.08
0.74
18.97
T5-sample
36.17
0.13/7.449
0.24
6.47%
3.62
31.18
0.71
20.78
Genie
186.7
0.09/6.935
0.28
8.56%
2.38
34.85
0.66
20.44
30.85
178.2
0.09/6.924
0.30
9.66%
5.06
41.59
0.68
19.97
PLANNER
67.94
0.11/7.553
0.21
5.38 %
4.84
33.97
0.69
20.04
57.88
PLANNER (10)
67.46
0.1
0.23
5.82%
11.61
41.23
0.72
19.89
Human
37.8
0.13/7.656
0.21
5.56%
21.19
Table 4: For summarization task, PLANNER outperform Genie (Lin et al., 2022 in generation diversity and fluency while maintaining comparable Rouge-L scores. (10) indicates the maximum results after 10 runs, following (Lin et al. 2022)
6.5 Analysis
Running time We conducted inference time benchmarks of each method single Nvidia A100. For the sentiment-guided generation task, the autoregressive baseline is 5x faster than our method as the generation for all methods can be batched. For all other tasks, the varying input lengths make direct batchification for the FT baseline not straightforward. In these scenarios, the latent diffusion over fixed number of latent codes offers computational advantages over a naive decoding of the FT baseline as the latent codes in our method can be conveniently batched.
For the hotel review completion task, the generation of 256 samples took 378 seconds to complete, including 83 seconds for decoding and 295 seconds for diffusion generation with 30 generation steps. The unbatched FT baseline took 1,693 seconds to complete 256 generations. Sorting input text by length and maximally batchifying them as possibley reduce the (batched) FT inference time to 338 seconds. The Diff-LM algorithm required 397 seconds to produce 256 samples using 200 generation steps, which is comparable to ours. Although our method is slower than the autoregressive ones, PLANNER enjoys the convenience of arranging input into the same length vectors without further length bucketing. On the CNN-DM summarization tasks, our method took 8.4 GPU hours to generate 11392 summaries. Genie's 's generation took 47.2 GPU hours. XSum gives similar inference running time benchmark to the results on CNN-DM.
Generations over diffusion steps In App. B we provide generation examples for both summariza- tion and sentiment-guiie -ugidd generation over different diffusion steps, which progress in coarse-to- fine manner. The generation from early time step tends to be less fluent and generic. As the time ap- proaches 0, the generation becomes more detailed. We presented quantitative results characterizing the evolution of the metrics over generation steps App. B improvement in the majority of the metrics the generation proceeds. Notably, most hallucinations
Fig. 5
It revealed clear trend of

---

occur during the late phase when more details are being incorporated. The model may excessively emphasize certain aspects, resulting in the correct generation being altered to an erroneous one (App. Tab. T0
7 Conclusion
We present a two-stage latent text diffusion model that uses an autoencoder to condense lengthy texts into limited number of paragraph embeddings, and continous time diffusion model that learns the distribution of these embeddings. Our proposed model alleviates the issue of repetition and advances generation diversity across different tasks. Compared to text diffusion models that perform diffusion solely on token or token embedding space, our method generates fluent text with improved diversity and reduced repetition. There may be toxicity fairness issues in the dateset used that we have not been able to identify. There are several limitations that warrant further investigation. Our work relies on an autoregressive decoder for converting latent representation into coherent text. is possible to explore the feasibility of non-autoregressive decoders to bolster efficiency while minimizing conversion errors and hallucination in the generation. Furthermore, the classifier-free guidance approach results in discrepancy between training and inference data distribution when feeding to the diffusion backbone. It would be interesting to investigate "calibration" strategy for the latent code to better fit the data distribution during training.
Acknowledgement
We thank Yinfci Yang, Barry Theobald, Zhe Gan, Edouard Grave, David Grangier, Tatiana Likhoma- nenko, Richard Bai and Ronan Collobert for their critical suggestions and helpful feedback throughout this project.
10

---

References
Alexander A Alemi, Ian Fischer, Joshua V Dillon, and Kevin Murphy. Deep variational information bottleneck. In ICLR, 2016.
Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg. Structured denoising diffusion models in discrete state-spaces. NewrIPS, 34: 34:17981-17993, 2021.
Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazcer. Scheduled sampling for sequence prediction with recurrent neural networks. In NewrlPS, volume 28, 2015.
Samuel Bowman, Luke Vilnis, Oriol Vinyals, Andrew Dai, Rafal Jozefowicz, and Samy Bengio. Generating sentences from a continuous space. In CONLL, pp. 10-21, 2016.
Ting Chen, Ruixiang Zhang, and Geoffrey Hinton. Analog bits: Generating discrete data using diffusion models with self-conditioning. arXiv preprint wr.iv:2208.042000 2022.
Pim de Haam, Dinesh Jayaraman, and Sergey I Levine. Causal confusion imitation learning. In H. Wallach,
H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32, Curran Associates, Inc., 2019. URL https: //ppoceedinge..... neurips
cc/paper_fiie www.ffeeeappppaeeee 2019/f1lee9////// http:///ww.//.////////////////00
as8pee333355888999922 Paper pdf
Shansan Gong. Mukai Li. Jiangtao Feng, Zhiyong Wu, and LingPeng Kong. Diffuseq: Sequence sequence text generation with diffusion models. In ICLR, 2023.
Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman. and Phil Blunsom. Teaching machines to read and comprehend. Advances neural information processing systems, 28, 2015.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurlPS 2021 Workshop on Deep Generative Models and Downstream Applications,
Downstreammppplicatinnn,, 2021.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurlPS, volume 33, 2020.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In ICLR, 2019.
Emiel Hoogeboom, Alexey Gritsenko, Jasmijn Bastings, Ben Poole, Rianne van den Berg, and Tim Salimans. Autoregressive diffusion models. In ICLR, 2021.
Emiel Hoogeboom, Jonathan Heek, and Tim Salimans. simple diffusion: End-to-end diffusion for high resolution images. arXiv preprint arXtv:230). 1103, 2023.
Zhiting Hu, Zichao Yang, Xiaodan Liang, Ruslan Salakhutdinov, and Eric P Xing. Toward controlled generation of text. In International conference machine learning, pp. 1587- 1596. PMLR, 2017.
Bowen Jing, Gabriele Corso, Renato Berlinghieri, and Tommi Jaakkola. Subspace diffusion generative models. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXIII, pp. 274 289. Springer, 2022.
Nitish Shirish Keskar, Bryan McCann, Lay R Varshney, Caiming Xiong, and Richard Socher. Ctrl: A conditional transformer language model for controllable generation. arXiv preprint arXiv:190.005856 2019.
Yoon Kim, Sam Wiseman, and Alexander M Rush. tutorial on deep latent variable models of natural language. arXiv preprint arXiv: wtt.::./22.68.. 2.6334,, 2018.
Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusson models. NeurIPS, 34: 21696-21707, 2021.
Alex M Lamb, Anirudh Goyal ALIAS PARTH GOYAL, Ying Zhang, Saizheng Zhang, Aaron C Courville, and Yoshua Bengio. Professor forcing: A new algorithm for training recurrent networks. Advances in neural information processing systems. 29, 2016.
Chunyuan Li, Xiang Gao, Yuan Li, Baolin Peng, Xiujun Li, Yizhe Zhang, and Jianfeng Gao. Optimus: Organizing sentences via pre-trained modeling of latent space. In EMNLP, 2020.
Jiwei Li, Myle Ott, Claire Cardie, and Eduard Hovy. Towards a general rule for identifying deceptive opinion spam. In ACL, pp. 1566 1566-117,2220 2014.

---

diversity-promoting Jiwei Li, Michel Galley, Chris Brockett. Jianfeng Gao, and Bill Dolan. A diversity promoting objective function for neural conversation models. In NAACL, 2016a.
Jiwei Li. Will Monroe, Alan Ritter. Dan Jurafsky, Michel Galley, and Jianfeng Gao. Deep reinforcement learning for dialogue generation. In EMNLP, pp. 1 192--1202, 2016b.
Xiang Li, John Thickstun, Ishaan Guirajani, Percy Liang, and Tatsunori B Hashimoto. Diffusion-lm improves controllable text generation. NeurIPS, 35:4328-43444 2022.
Chin-Yeau Lin, Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pp. 74-81, 2004.
Zhenghao Lin, Yeyun Gong, Yelong Shen, Tong Wu, Zhihao Fan, Chen Lin, Weizhu Chen, and Nan Duan. Genie: Large scale pre-training for text generation with diffusion model. arXiv preprint arXiv:2212.11666 2022.
Justin Lovelace, Varsha Kishore, Chao Wan, Eliot Shekhtman, and Kilian Weinberger. Latent diffusion for language generation. arXiv preprint arXiv:2212.09462 2022.
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. Don't give me the details, just the summary! topic- aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Con- ference on Empirical Methods in Natural Language Processing, pp. 1797-1807, Brussels, Belgium, October-November 2018. Association for Computational Linguistics. doi: 10. 10.18853/11/D1--120 URL https://eccaatthooooy..r///// 1206
Myle Ott, Michael Auli, David Grangier, and Mare' Aurelio Ranzato. Analyzing uncertainty in neural machine translation. In ICML, pp. 3956-3965. PMLR, 2018.
William Peebles and Saining Xie. Scalable diffusion models with transformers. arXiv preprint 201:: 2022.
Matt Post. A call for clarity in reporting BLBU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, October 2018.
Konpat Preechakul, Nattanat Chatthee, Suttisak Wizadwongsa, and Supasorn Suwajanakorn Diffusion autocn- coders: Toward a meaningful and decodable representation. In CVPR, pp. 10619-10629, 2022.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang. Michael Matena, Yanqi Zhou, Wei Li. and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-texi transformer. JMLR, 2020. URL http::/jmlr org/paapess/v11/200 html
Robin Rombach, Andreas Blatumann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, pp. 10684-10695, 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language undcrstanding. In NeurIPS, volume 35, pp. 36479 36494, 2022.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In ICLR, 2021a.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients the data distribution. NeurlPS, 32, 2019.
Yang Song, Jascha Sohl-Dickstein Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic diffcrential equations. In ICLR, 2021b.
Scan Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. Neural text generation with unlikelihood training. arXiv preprint arXiv: 1908. 04319, 2019.
Ronald Williams and David Zipser. learning algorithm for continually running fully recurrent neural networks. Neural computation, 1(2):270225) 1989.
Jin Xu, Xiaojiang Liu, Jianhao Yan. Deng Cai, Huayang Li, and Jian Li. Learning break the loop: Analyzing and mitigating repetitions for neural text generation. NeurIPS, 2022.
Lantao Yu, Weinan Zhang, Jun Wang, and Yong Yu. Seggan: Sequence generative adversarial nets with policy gradient. In AAAI, volume 31, 2017.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. In ICLR, 2019.
12

---

Yizhe Zhang. Zhe Gan, Kai Fan, Zhi Chen, Ricardo Henao, Dinghan Shen, and Lawrence Carin. Adversarial feature matching for text generation. In ICML, pp. 4006-4015. PMLR, 2017.
Yizhe Zhang, Michel Galley, Jianfeng Gao, Zhe Gan, Xiujun Li, Chris Brockett, and Bill Dolan. Generating informative and diverse conversational responses via adversarial information maximization. In Neur!PS, 2018.
Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. Texygen: A benchmarking platform for text generation models. In 41st international ACM SIGIR conference on research development in information retrieval, pp. 1097 1100, 2018.
Zixin Zhu, Yixuan Wei, Jianfeng Wang. Zhe Gan, Zheng Zhang, Wang. Gang Hua, Lijuan Wang. Zicheng Liu, and Han Hu. Exploring discrete diffusson models for image captioning. arXiv preprini arXiv:2211. 2022.
13


Reviews:
[{"review_id": "zSSq7yRKVs", "reviewer": "Reviewer_MZNE", "summary": "This paper proposes to combine latent semantic diffusion with autoregressive generation to alleviate the issues of exposure bias in training / inference of text based language models, and computational and performance cost of purely diffusion approaches. They improve the diffusion process by applying it to the latent semantic space instead of the token / embedding space. For this, they learn some \"semantic tokens\" for encoding paragraph level information and then use a decoder to map these to the raw text space.", "questions": "Can you talk a bit more about the design choices and how much impact they had on the results? For eg: how much impact does fixing \"distributional smoothness\" have?\nHave you performed other ablation experiments to justify claims?\nWhy did you subsample the test sets?", "limitations": "Addressed in appendix.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "Working with the diffusion model in the semantic spaces opens up the door for controllable generation.\nThey also provide an extensive study of the requirements for a good latent space for paragraph diffusion models.\nThe paper is well written and easy to follow.\nThe ideas employed to fix / ensure local smoothness (by perturbing data) and distributional smoothness (by using VAE) are simple and useful.\nThey propose a novel (?) metric called AuBLEU to evaluate the denoising capabilities of the model. I believe is this generally suitable for other works and could be impactful. However, it does not feel to be properly justified / grounded.\nHparams are provided.\n Human eval results are significant. The other results are sound and robust evaluation is performed. The paper is just lacking ablations on the design choices.\nThe analysis is complete and justifies the main claims made by the authors.", "weaknesses": "The changes employed by the authors, especially during the training stage are not properly ablated.\nIt is not clear if the proposed fixes provide benefits.\nSome of the design choices are not experimentally justified (eg: line 182)\nEvaluation is performed on just a sample of the test set (this is the first time I have seen something like this in a paper and I'm not sure how to take it - I'm not super comfortable as this makes your technique essentially un-comparable). This also might not be robust. (line 196).\nNo intention of providing code and it might be very hard to reproduce because of the many changes especially in the training setup.\n\n\nNitpicks:\ntypo line 181\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their helpful feedback. We will fix the typo pointed out. We address the questions in below:\n\n*1. Evaluate on subset datasets*\n\nPlease refer to general questions.\n\n*2. Ablation regarding rescaling*\n\nWe omit the rescaling step in this study due to the absence of the \"rescaling invariant\" property in the latent text code. Specifically, we have not imposed any constraints to ensure that the generated output remains the same after rescaling the latent code. In contrast, for Imagen, where the generation takes place in the raw pixel space, rescaling will predominantly retain the shape information while altering only the contrast and brightness. Initially, we conducted experiments involving rescaling, but results demonstrated poorer performance compared to the non-rescaled version, as evidenced by a **-10.6** ROUGE score drop on CNNDM. Consequently, we opted for the dynamic thresholding without rescaling. We will make this clearer.\n\n*3. Code Release*\n\nWe have a python+pytorch implementation that reproduces experiment results, and are finalizing legal approvals to open-source the codebase. We will release the code upon publication.\n\n*4. Ablation on design choice*\n\nWe indeed performed an ablation for \"distribution smoothness\" on the sentiment-guided generation task. The model trained without the variational objective underperforms the full model in all metrics by **-10.1** AuBLEU, **+18.8** PPL, and **-15.2%** ACC. We will incorporate the results in the next revision.\n    The evaluation pipeline might get a bit expensive if the diffusion model training and evaluation is also involved. Instead, we mostly use a surrogate metric (line 255) to monitor the overall quality of the learned representation. Detailed evidence presented in the appendix demonstrates a reasonably strong correlation between this empirical metric of the representation quality and the subsequent performance of diffusion generation. The models trained with the variational objective consistently improve the performance across the board.\n"}, {"review_id": "u8pOFUaSFw", "reviewer": "Reviewer_r7NC", "summary": "In this paper, the authors propose PLANNER, a model that combines latent semantic diffusion with autoregressive generation to generate fluent text while exercising global control over paragraphs. The proposed method is evaluated on various conditional generation tasks, and the results show its effectiveness in generating high-quality text.", "questions": "1. Unlike autoregressive models based on conditional probabilities, diffusion models are unable to ascertain the optimal sentence from the generated set. How did you address this problem in your paper?\n2. As the reverse diffusion goes on in latent space, does the latent representation z get closer to the representation of the ground-truth one in cosine distance or other distances? \n3. What is the influence of paragraph embeddings number, k in line 119? ", "limitations": " ", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper addresses the issue of repetitive and low-quality output generated by autoregressive models and proposes a novel approach using latent semantic diffusion.\n2. The combination of autoregressive decoding and latent diffusion allows for efficient generation over paragraph generation.\n3. The proposed method is evaluated on various tasks and shows improved generation quality compared to autoregressive and text diffusion baselines.", "weaknesses": "1. I understand that revisiting and revising the generated sentences can alleviate exposure bias, as errors can be reduced through further editing at the token level. However, revisiting and revising the latent space does not seem reasonable to me. From my perspective, the exposure bias occurs in the process of picking words out, but the authors merely employ GPT-2 without making any changes in the phase.\n2. The paper claims that they can generate longer text and paragraphs, but there is no further analysis about relationship between length of generated sentence and the performance. \n3. The evaluation of CNN/Daily Mail and XSum is performed on 256 subsampled examples from the test set. This manner is not convenient for the subsequent works to follow, and this also hinders reviewers from making fair comparisons between this work and others.\n4. I have reservations about the ability of the variational paragraph embedder to learn effective representations of sentences with different lengths. What would happen if there is a significant difference in sentence lengths, such as one being very short (e.g., 5 tokens) and the other being much longer (e.g., 512 tokens or more)?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their constructive feedback. We address the comments in below.\n\n*1. Evaluate on subset datasets*\n\nPlease refer to general questions. \n   \n*2. Exposure bias*\n\nOur hypothesis is that exposure bias occurs when there is a discrepancy between the training and inference stages, specifically during teacher forcing. In our diffusion model, we predict the latent semantics code in an non-autoregressive manner. No partial ground truth latent code has been fed into the diffusion model during training time. Consequently, the diffusion model is less affected by exposure bias issue. \nAs for the decoder, it is trained using an autoencoder with teacher forcing, which means that exposure bias can still exists in this stage. However, the impact of exposure bias is limited to the final translation of the semantics into text. We demonstrate that the error in this translation is minimal (reconstruction BLEU > 80\\%) due to the simplicity of the autoencoding task and the strong influence of the input latent code on the decoder, resulting in less error compounding effects. Hence, we experience much less exposure bias compared to the autoregressive approach.\n\n\n*3. Length Ablation*\n\nWe conducted tests on a total of five datasets using our model. The target generation length varied within a range of 15.2 to 181.29. This range was selected to encompass diverse generation lengths. If comparing across tasks, our method received more diversity/repetition metrics improvement in lengthier generation tasks (review generation). \nIn the hotel reviews dataset, the length of the target sentences can vary from 23 to 512. In Appendix Table 6-7, we provide examples of sentences reconstructed from the latent codes. Generally, shorter sentences are easier to represent and reconstruct, evidenced by the fact that tasks with shorter target sentences (XSum, CNN-DM) achieved higher $BLEU_{clean}$ scores. We will make these clearer.\n\n*4. No optimal solution*\n\nOur method, inheriting from the latent diffusion model, is a sampling technique. It is worth noting that even for tasks such as translation and summarization, the default generation methods for autoregressive models like popular LLMs rely on sampling. Previous work [1,2] has demonstrated that the notion of an \"optimal sentence\" may be a misleading \"red herring\", as optimizing the likelihood can result in low-quality outputs such as repetition and generation artifacts. This is particularly evident in open-ended generation scenarios, where the distribution of text is inherently multimodal.\n\n*5. Representation Distance*\nYes, as the reverse diffusion process progress, the representation will be closer to the ground truth representation. This is evidenced by the Figure 6 in the Appendix, which shows the BLEU score between the ground truth text and the text translated from the latent code at time $t$. The graph reveals a consistent pattern of progressive improvement in the BLEU score as $t$ decreases from 1 to 0.\n\n*6. Impact of $K$*\n\nPlease refer to general questions. \n\n[[1]](https://arxiv.org/abs/1904.09751) Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. *The curious case of neural text degeneration.* In ICLR, 2019 \n\n[[2]](https://arxiv.org/abs/2206.02369) Jin Xu, Xiaojiang Liu, Jianhao Yan, Deng Cai, Huayang Li, and Jian Li. *Learning to break the loop: Analyzing and mitigating repetitions for neural text generation.* In NeurIPS, 2022"}, {"review_id": "lPX3gohpGe", "reviewer": "Reviewer_6a1h", "summary": "This paper presents a 2-stage generative model for text. The first stage involves training a VAE over the data to obtain an effective encoder and decoder modules. The second stage uses the output of the frozen VAE encoder as the hidden state of the input text and learns a regular continuous diffusion model over the encoded hidden states of training data instances. For generation, the diffusion model produces a hidden state from noise and this hidden state is fed into the VAE decoder to generate text. This model is evaluated on sentiment-guided generation, text completion, and summarization and compared against baselines that include token-based diffusion models (instead of hidden state text diffusion models in this work), VAE/Enc-dec models, and standard autoregressive models (like fine tuned GPT-2).", "questions": "See above.", "limitations": "See above.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "– Although the approach seems straightforward, to the best of my knowledge, surprisingly I am not aware of other latent sequence/paragraph embedding based diffusion models for text generation. Hence, this fills a gap in the research on diffusion models for text.\n\n– The experimental design is reasonable and informative. \n\n– This approach seems to be more fluent, diverse, and effective than popular token-based diffusion models following the results.\n\n– Although typically less fluent than the autoregressive models, it still performs competitively when considering other metrics like diversity, reference overlap, and control.\n", "weaknesses": "– I have some concerns about evaluation, particularly related to the quality of baseline models. For example, on CNN-DM summarization task, publicly available results (https://paperswithcode.com/sota/abstractive-text-summarization-on-cnn-daily) show that T5 achieve R-L that is ~7-10 points better than reported in this paper (and is hence better than the R-L proposed model achieves). Therefore, I am doubtful about the quality of training/tuning of the baseline models reported in the paper.\n\n– Choosing “first k hidden state vectors” from the encoder in the VAE seems arbitrary. What is the motivation behind this hyperparameter? What is the effect of this k?\n\n– The paper alludes to learning “paragraph embeddings” which capture high level semantic properties which enables “coarse-to-fine” generation and allows for “planning”. However, no evidence is presented in this paper about compositional/planning capabilities of the model. While it is true that paragrpahs/tokens are generated from a lower dimensional manifold, it doesn’t automatically imply that these embeddings are linguistically interesting or allow for controlled planning-oriented generation. At an abstract conceptual level, how different are these representations from VAE based representation for example?\n\n– Although this is a generative model, the training objective is arbitrary and doesn’t seem linked to maximizing the likelihood/ learning the distribution of the training data. This is because the VAE and the diffusion components are separately trained and hence preclude a clean interpretation of the training objective.\n\n– I am not convinced that the “Distributional smoothness metric” reflect the nature of the manifold as is claimed in the paper.\n\n– Runtime analysis seems to compare unbatched autoregressive models with batched diffusion models. Sorting by length and then padding for batching is a standard practice for autoregressive models.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their insightful feedback. Next, we address the comments:\n\n*1. Weak baseline*\n\nThank you for the pointer. We have examined the link. Please correct us if we are making mistake, but it seems to us that the T5 model that achieved 7-10 points better than our T5 baseline might be much larger (11B, [https://paperswithcode.com/paper/exploring-the-limits-of-transfer-learning](https://paperswithcode.com/paper/exploring-the-limits-of-transfer-learning)) than our baseline (770M). We also found some publicly accessible model ([https://huggingface.co/sysresearch101/t5-large-finetuned-xsum-cnn](https://huggingface.co/sysresearch101/t5-large-finetuned-xsum-cnn)) that have finetuned T5-large using XSum + CNNDM, and they have reported lower ROUGE scores than ours.\n\n*2. Motivation of $k$*\n    \nPlease refer to general questions. \n\n*3. Planning*\n\nThank you for the insightful comment. According to our understanding, the diffusion model engages by utilizing the previous step output and the controlling signal to refine the representation and incorporate more intricate details. For an example, please refer to Appendix Table 8. Initially, the decoded text from early diffusion steps lacks specificity and coherence. However, as the diffusion inference progresses, the model gradually incorporates additional syntactic and semantic details, and also determines when to remove certain information. This process bears resemblance to the image diffusion process. The external controlling signal provides guidance at each stage of the text generation process. While the representation is learned by VAE, our approach differs in that the diffusion model generate samples close to the ones from the posterior distribution of the latent code, which potentially possesses a more sophisticated structure than a simple Gaussian distribution. \n\n*4. Arbitrary objective*\n\nOur understanding is that Diffusion is the process of optimizing the evidence lower bound (ELBO) as VAE, which is a valid objective. Our diffusion model is essentially a Latent diffusion model (LDM) which share the same objective thus remains meaningful. \n    Our two stage training setup resembles that of VQVAE/VQVAE2 (which actually reports likelihoods). In VQVAE, they state \"Whilst training the VQ-VAE, the prior is kept constant and uniform. After training, we fit an autoregressive distribution over z, p(z), so that we can generate x via ancestral sampling.\" We share similar rationale with VQVAE. In the first phase, we train a regular VAE with a simple prior and posterior; In the second phase, we freeze the decoder and posterior, but train a diffusion-model-based prior, which only optimizes the $KL(q(z|x) || p(z))$ term but freezes the $p(x|z)$ part. As a result, the ELBO should continue to improve.\n\n*5. Distribution smoothness metric*\n\nWe acknowledge that the PPL of the linear-interpolated sample might not fully reflect the nature of the manifold. Nevertheless, we would hope it has the potential to provide some insight into whether the posterior distribution is highly multimodal, with spikes and numerous density \"holes\". Given our knowledge we might not be aware of other better alternatives to evaluate the smoothness of a distribution. Therefore, we would greatly appreciate any suggestions that could expand our understanding.\n\n*6. Running time analysis*\n\nWe are aware that the unbatched autoregressive model is not the best choice (see line 343). When we evaluate the full test set of CNNDM and XSum we performed the batched version by sorting input text by length and maximally batchifying them as possible. The total running time reduced more than 4x from 11 hours (unbatched version) to 2.5 hours (batched version). We admit there might be more room to accelerate the autoregressive baseline. In fact, we do not intend to indicate our method can be faster than the autoregressive ones, but only to show the convenience of arranging input into the same length vectors. We will make this clearer. "}, {"review_id": "FgmgC5gUbt", "reviewer": "Reviewer_H5ac", "summary": "The paper presents a two-stage latent text diffusion model that uses an autoencoder to condense lengthy texts into a limited number of paragraph embeddings, and a continuous time diffusion model that359 learns the distribution of these embeddings. The paper presents detailed experiments on open-ended generation tasks and shows that the proposed model alleviates the issue of repetition and advances generation diversity across different tasks.", "questions": "Please see the weakness.", "limitations": "Please see the weakness.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper is well-motivated. The examples in Figure 1 with the self-reinforcing effect are impressive. The proposed method first summarizes the paragraph information with an autoencoder and then applies the diffusion process to control the token generation process, which is novel from my point. The paper conducts extensive experiments on open-ended generation tasks and provides some analyses of the running time.", "weaknesses": "1. Could you provide more experimental analyses on the learned paragraph embedding? I think that paragraph embedding is used to learn a high-level concept for planning but ignores the details of sentence/phrase structure to avoid copying similar phrases from previous prompts and repeating them.  Could you provide more experiments to show the quality of learned paragraph embedding? Directly extracting the first K tokens from the encoder is not the best choice for me.\n\n2. In experiments such as Table 1, the proposed method with a greedy decoding mode is comparable to baselines with top-p sampling in terms of diversity. A question is whether PLANNER combined with top-p can lead to better results. Are the proposed methods compatible with stochastic decoding methods?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their encouraging feedback. We address the questions in below:\n\n*1. Paragraph Embedding*\n\nPlease refer to the general question titled \"Motivation and Impact of $k$.\" Additionally, we provide an analysis of how the acquired embedding influences the eventual generation, along with examples illustrating the abilities of reconstruction, denoising, and interpolation in Appendix A.\n\n*2. PLANNER with stochastic decoding*\n    \nIt is possible and straightforward to implement stochastic decoding for the PLANNER. In fact we experimented with this option. In our experiments, we utilized nucleus sampling with a value of p=0.92 and K=50 (see below for an experiment on hotel review generation). By incorporating stochastic decoding, the diversity and repetition metrics can be improved, although at the expense of relevance and accuracy scores. It is important to mention that the decoder's role in PLANNER is to faithfully translate the latent code into the desired target text, rather than performing compositional/planning job. Stochastic decoding may disrupt this role and can lead to undesirable generation, as we observed an increase in hallucinations when combining PLANNER with stochastic decoding.\n\n| Method          | PPL  | DIST/ENT   | S-BL | Rep-4    | BLEU  | ROUGE | Len   |\n|-----------------|------|------------|------|----------|-------|-------|-------|\n| PLANNER greedy  | 47.3 | 0.17/6.60  | 0.52 | 1.55%    | 0.77  | 7.9   | 168.1 |\n| PLANNER top-p   | 72.0 | 0.20/6.80  | 0.38 | 0.94%    | 0.58  | 6.1   | 173.2 |"}]

Author rebuttal:
**Common Questions**:

*1. Evaluate on subset datasets*

In order to expedite the iterations of the experiment, we opted for a partial evaluation of our method as the full evaluation of our method / Genie takes 7h / 2d to complete on CNN-DM or XSum test set. Nevertheless, we agree with the reviewers' concern that this approach could potentially compromise the comparability of our results. Consequently, we have conducted a full evaluation on the entire test set of CNNDM and XSum, which is presented in the table below. The main conclusion remains the same. We will include all the updates in our forthcoming version. 


| **Arch.** | **PPL** | **DIST/ENT**↑ | **S-BL**↓ | **Rep-4**↓ | **BL**↑ | **R-L**↑ | **Score**↑ | **Len** | **AuBL**↑ |
|:---------:|:-------:|:-------------:|:---------:|:----------:|:-------:|:-------:|:---------:|:------:|:---------:|
|           |         |               |           |            |         |         |           |        |           |
| **CNN Dailymail dataset** | | | | | | | | | |
| **T5-search** | 58.12 | 0.11/7.726 | 0.24 | 6.69% | 7.66 | 34.48 | 0.66 | 45.51 | - |
| **T5-sample** | 67.58 | 0.11/7.790 | 0.20 | 3.50% | 5.05 | 30.15 | 0.64 | 48.51 | - |
| **Genie** | 179.9 | 0.09/7.293 | 0.24 | **4.16%** | 3.22 | 30.47 | 0.58 | 40.94 | 27.21 |
| **Genie$^{(10)}$** | 170.6 | 0.10/7.355 | 0.24 | 4.32% | 6.48 | **37.09** | 0.62 | 40.81 | - |
| **PLANNER** | 49.21 | **0.10/8.037** | 0.15 | 5.25% | 6.92 | 30.43 | 0.62 | 52.33 | **43.91** |
| **PLANNER$^{(10)}$** | 49.07 | 0.10/8.019 | **0.15** | 4.96% | **11.42** | 36.81 | **0.66** | 53.14 | - |
| **Human** | 49.477 | 0.12/8.226 | 0.16 | 5.63% | - | - | - | 51.15 | - |
|           |         |               |           |            |         |         |           |        |           |
| **XSum dataset** | | | | | | | | | |
| **T5-search** | 29.41 | 0.12/7.200 | 0.31 | 14.83% | 6.11 | 36.08 | 0.74 | 18.97 | - |
| **T5-sample** | 36.17 | 0.13/7.449 | 0.24 | 6.47% | 3.62 | 31.18 | 0.71 | 20.78 | - |
| **Genie** | 186.7 | 0.09/6.935 | 0.28 | 8.56% | 2.38 | 34.85 | 0.66 | 20.44 | 30.85 |
| **Genie$^{(10)}$** | 178.2 | 0.09/6.924 | 0.30 | 9.66% | 5.06 | **41.59** | 0.68 | 19.97 | - |
| **PLANNER** | 67.94 | **0.11/7.553** | **0.21** | **5.38%** | 4.84 | 33.97 | 0.69 | 20.04 | **57.88** |
| **PLANNER$^{(10)}$** | 67.46 | 0.11/7.529 | 0.23 | 5.82% | **11.61** | 41.23 | **0.72** | 19.89 | - |
| **Human** | 37.8 | 0.13/7.656 | 0.21 | 5.56% | - | - | - | 21.19 | - |

*2. Motivation and Impact of $k$*

The parameter $k$ determines the number of latent codes used to represent a paragraph and therefore controls the compression level. Latent codes with smaller values of $k$ are easier to model using the diffusion model, but may struggle to accurately preserve all the information in the original text. Additionally, smaller values of $k$ offer computational efficiency. as the sequence length for the diffusion model is $k$. 

To determine the best set of latent codes, we indeed conducted experiments using three different methods: 1) selecting the first $k$ hidden vectors, 2) selecting the last $k$ hidden vectors, and 3) selecting interleaving hidden vectors, one for every $L/k$ hidden vectors. The results of the ablation study are presented below. Based on our findings, we observed no significant difference among the different choices, so we opted for option 1.

Furthermore, we discovered that increasing the value of $k$ does not lead to a dramatic improvement in performance (as stated on line 218, see the ablation study below). To tradeoff between efficiency and performance, in most of our study we focus on using $k=16$

| Experiment (on hotel review) | BLEU_clean | BLEU_robust |
|-----------------------------|------------|-------------|
| First k (k=16)              | 79.59      | 43.17       |
| Last k (k=16)               | 78.96      | 42.85       |
| Interleaving k (k=16)       | 79.81      | 43.35       |
| k=8                         | 57.90      | 30.68       |
| k=32                        | 82.31      | 45.14       |

Meta-review: This paper introduces PLANNER, an innovative model that combines latent semantic diffusion with autoregressive generation to produce fluent text while maintaining global control over paragraphs. By integrating a "decoding" module with a "planning" module, the authors address issues of repetitive and low-quality output found in autoregressive models. The proposed method is evaluated on various conditional generation tasks, and the results demonstrate its effectiveness in generating high-quality long-form text efficiently. The reviewers are overall positive towards the contribution of this paper. I would recommend to accept this paper.
Decision: Accept (poster)