SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Automatic Evaluation Metrics Following previous work (Gong et al., 2023), we assess the fluency of the generation by computing the perplexity (PPL) under a GPT-2 large model. We use Ent-n (Zhang et al., 2018) and DIST-n (Li et al., 2016a) and self-BLEU (S-BL) (Zhu et al., 2018) to evaluate lexical diversity. We present DIST-n and Ent-n metrics solely at  $n = 1$  owing to their strong correlation of the varying  $n$  values. We use REP-n to assess the extent of repetition in the generation following previous work (Welleck et al., 2019; Xu et al., 2022). For relevancy we use standard metrics following (Gong et al., 2023), including SacreBLEU (BL) (Post, 2018), ROUGE-L (R-L) (Lin, 2004) and BERTScore (Score) (Zhang et al., 2019). Details are provided in App. F
AuBLEU: Evaluating Denoising Capability Our proposed model is a latent diffusion model, which differs from text diffusion models that operate directly on text or text embedding space. To comparing the denoising ability across different text diffusion models, we introduce a novel metric, named AuBLEU (AuBL). To compute the AuBLEU score, we first add varying levels of noise to each input text  $\mathbf{x}$  by performing diffusion at  $T$  different time steps  $t_0 < t_1 < \dots < t_T$ , corresponding to a series of SNR  $\omega_{t_0} > \dots > \omega_{t_T}$ . Next, we pass each corrupted input under different  $\omega$  to the denoising backbone model and obtain the predicted output  $\hat{\boldsymbol{x}}_i = \boldsymbol{F}(\boldsymbol{x}_{t_i})$ . We then compute the BLEU score between each  $(\hat{\boldsymbol{x}}_i, \boldsymbol{x})$  pairs and plot a curve with the x-axis representing  $\alpha^2 = \frac{\omega}{1 + \omega}$ , where  $\alpha^2$  is monotonically increasing with  $\omega$  and ranges from  $(0, 1)$ , and the y-axis indicating the corresponding BLEU score. Finally, we compute the area under curve to obtain the AuBLEU score (see App. D for more details).
Model Setups We used the BERT-large and GPT-medium models as initialization for the encoder  $\mathcal{E}$  and decoder  $\mathcal{D}$  respectively. The embedding dimension  $h$  was 1024, and the number of paragraph embeddings  $k$  was set to 16, as increasing the number did not result in significant improvement in performance. We provide more analysis on the impact of  $k$  in App. [A.2] The learning rate was set to  $2e - 4$ , and  $\beta$  was set to  $5e - 6$ . For the latent diffusion model, the channel size was set to 1024 to match the embedding dimension  $h$ , and the number of heads was set to 16 with 28 transformer layers. The total size of the latent diffusion model was 533M. The feature encoder  $\tau$  was also jointly learned, and was initialized with a T5-large encoder. We use DDIM throughout our experiments as it shows better performance than DDPM. In all our experiments, we use 30 diffusion steps to generate the final  $z$ , which strikes a good balance among the efficiency, diversity and relevance. In comparison, Diff-LM (Li et al., 2022) and Genie (Lin et al., 2022) report to use 200 steps and 2000 steps respectively to generate high-quality text. We set the CFG weights to be 2 and 5 for text completion and summarization tasks, respectively, based on generation performance on validation set. For summarization task, we also incorporate a shift noise scheduler based on (Hoogeboom et al., 2023). More details, including ablations on DDPM, number of diffusion steps and noise scheduler, are provided in App.
Baselines We compare our method with several baseline methods trained under Teacher Forcing scheme, including decoder-only Autoregressive LM finetuned on GPT-2 (FT), encoder-decoder (Enc-Dec) transformer model, and Varitional Information Bottleneck (VIB) (Alemi et al., 2016). We initialized the FT model using GPT-2 large (774M), whereas encoder and decoder in the Enc-Dec/VIB models (695M/697M) are initialized with bert-large and GPT-medium, respectively. All the considered models are finetuned on the target datasets. We follow Li et al. (2022) to report the FT baselines with two decoding strategies, top-p sampling  $(\mathrm{K} = 50$ $p = 0.92)$  and beam search (beam width 4), denoted as FT-sample and FT-search. We use top-p sampling for Enc-Dec/VIB generation. For summarization tasks, we finetune a T5-large model (770M) on the target datasets as baselines. We also compared two text diffusion models Diff-LM and Genie using their suggested configuration from the official repository. More details are in App.
Figure 3: Impact of the proportion of injected noise for learning Paragraph Embeddings on XSum dataset. Large substitution noise results in worse  $\mathbf{BLEU}_{\mathrm{clean}}$  but better  $\mathbf{BLEU}_{\mathrm{robust}}$  and  $\mathbf{PPL}_{\mathrm{int}}$ .
6.1 Paragraph Representation Learning
It is essential to learn a paragraph embedding space that is both accurate and smooth. To this end, we examined the effect of various substitution probabilities  $p$  to the input tokens  $x$ . Our findings, presented in Fig. 3 reveal that a smaller  $p$  results in a lower conversion error, as indicated by a higher reconstruction BLEU  $(\mathbf{BLEU}_{\mathrm{clean}})$ , albeit at the expense of local smoothness  $(\mathbf{BLEU}_{\mathrm{robust}})$  and distributional smoothness  $(\mathbf{PPL}_{\mathrm{int}})$ . We performed a grid search of  $p$  with 0.1 increment based on 512 samples. Empirically, we observed that a weighted score  $S_{\mathrm{overall}} = 0.5\mathbf{BLEU}_{\mathrm{clean}} + 0.8\mathbf{BLEU}_{\mathrm{robust}} - 0.3\mathbf{PPL}_{\mathrm{int}}$  correlate well with downstream diffusion performance, leading to fluent and accurate generation for PLANNER (see App. A for more details). We finally opted for  $p = 0.3$  for most datasets which strike a balance between conversion error and smoothness.
It is worth noting that there is an inevitable conversion loss, indicated by the fact that the  $\mathbf{BLEU}_{\mathrm{clean}}$  is between  $77\sim 87$  when generating hundreds of words (App. D). We observe that most of the lexical mismatch still maintain the similar semantics, with the exception of some name entity swapping. We show some paragraph reconstruction and denoising examples from our paragraph embedder in the App. A. We also include examples of interpolated generation from random paragraph pairs in the App. A. In general the transition of semantics is natural, indicating reasonable distributional smoothness of the paragraph embedding space.
6.2 Sentiment-Guided Generation
For sentiment-guided generation experiments, following previous works (Li et al., 2022; Hu et al., 2017; Keskar et al., 2019), we use a trained classifier to assess if the given sentiment is well-controlled in the generation. The trained classifier is initialized with BERT-
Arch.	PPL	ACC↑	DIST/ENT↑	S-BL↓	Rep-4↓	Len
FT-sample	20.86	70.2%	0.13/6.154	0.96	5.86%	113
Diff-LM	101.97	83.6%	0.15/5.115	4.05	6.23%	66.2
Ours	51.12	94.9%	0.16/6.360	0.77	2.37%	161
Human	47.94	96.7%	0.17/6.737	0.48	2.17%	157
large and finetuned on the training set, which yields an accuracy of  $96.75\%$  on the held-out test set. The results are provided in in Tab. PLANNER outperforms the baseline approaches in generating long reviews at higher levels of accuracy. Although PLANNER using a greedy decoding mode is at least comparable with FT with top-p sampling in terms of diversity, and has lower repetition as assessed by Rep-4 in generation.
The perplexity of the text generated by PLANNER is close to that of human-written text. We provide examples of the generated text in App. E. Interestingly, as shown in App. E with the same random seed but different controlling sentiment, PLANNER generates text with similar contents but different sentiments, suggesting the diffusion model may be able to disentangle the semantic space to certain extent. Unlike the autoregressive generation, the nature of the diffusion model allows the model to "regret" and iteratively refine on the current generations. In App. B we demonstrate how the generation evolves over multiple time steps in a coarse-to-fine manner in PLANNER.
6.3 Long-form Text Completion
We further evaluate our model on the long-form text completion tasks. For text diffusion baseline, we compared our method with Diff-LM (Li et al., 2022) on hotel review dataset. We could not perform a comparison on the C4 dataset with Diff-LM due to the significant amount (thousands) of GPU hours required to train Diff-LM adequately. A Diff-LM running time estimation is available in App. The results are provided in Tab. 2. FT-search performed poorly in this open-ended generation task as its generation exhibited high repetition levels, consistent with findings in previous research (Holtzman et al., 2019; Xu et al., 2022). Although our approach also employs a deterministic decoding method, we observed that it produces text with low Rep-4 metric, signifying that PLANNER is effective in
2except for CNNDM dataset where we use  $p = 0.5$
Arch.	PPL	DIST/ENT↑	S-BL↓	Rep-4↓	BL↑	R-L↑	Score↑	Len	AuBL↑
Hotel Review dataset
FT-search	1.87	0.03/4.865	3.50	86.60%	0.62	5.2	0.39	179.51	-
FT-sample	15.51	0.14/6.455	0.88	4.49%	0.78	6.8	0.53	164.50	-
Enc-Dec	33.82	0.18/6.379	0.57	3.25%	0.47	7.3	0.54	94.03
VIB	36.89	0.19/6.481	0.54	3.15%	0.45	7.1	0.54	86.11	-
Diff-LM	178.30	0.13/5.560	3.57	4.54%	0.84	8.8	0.43	175.10	26.16
PLANNER	47.36	0.17/6.602	0.52	1.55%	0.77	7.9	0.55	168.08	38.55
Human	47.60	0.20/7.023	0.60	1.46%	-	-	-	181.29	-
C4 subset dataset
FT-search	1.927	0.07/6.245	0.14	79.54%	0.77	5.2	0.37	154.88	-
FT-sample	12.244	0.25/7.136	0.44	7.01%	1.59	5.9	0.47	122.55	-
Enc-Dec	23.095	0.24/7.077	0.16	2.27%	1.92	7.5	0.5	118.07	-
VIB	19.701	0.24/7.003	0.16	2.62%	1.86	6.8	0.49	113.34	-
PLANNER	61.768	0.28/7.352	0.12	1.67%	2.04	7.7	0.51	111.89	36.77
Human	59.783	0.44/7.381	0.12	1.12%	-	-	-	107.56	-
reducing repetition through holistic iterative refinement throughout the inference steps in the diffusion process. Comparing with Diff-LM and other baselines, PLANNER achieved better diversity scores while maintaining comparable relevance scores. We also observe higher AuBLEU of PLANNER comparing with Diff-LM, indicating a potentially higher overall denoising strength of PLANNER (See App. D for more details). Some examples of the generated text are available in the App. E We also observed PLANNER exhibits robustness towards prompts that are either repetitive or ill-composed, where FT failed (Fig. 1 App. G).
We further performed pairwise human evaluation on 300 examples of hotel review generation from each system on our internal crowd-source annotation platform. Each pair of text being presented to 3 judges in random order. The judges ranked the pairs for relevance, informativeness and human-like properties using a 3-point Likertlike scale. Overall judge preferences are shown in Table 3. A moderate preference can be observed for PLANNER over FT and VIB, except for human-like between PLANNER and VIB. We also observe that judges still prefer human responses over system generations in this task. Further details, including the human evaluation template used and interrater agreement analysis, are provided in the App. H
Table 2: PLANNER enhances the diversity of text generation and minimizes the occurrence of repetition in open-ended text completion tasks.
Metric	Methods	Win	Tie	Loss
Rel.	Ours vs. FT	48.2%	9.2%	42.6% *
Ours vs. VIB	50.7%	10.0%	39.3% **
Ours vs. Human	39.3%	9.3%	51.3% **
Inf.	Ours vs. FT	55.1%	5.7%	39.2% **
Ours vs. VIB	48.7%	8.0%	43.3% *
Ours vs. Human	37.7%	8.7%	53.7% **
Hum.	Ours vs. FT	51.5%	8.4%	40.1% **
Ours vs. VIB	40.0%	19.3%	40.7%
Ours vs. Human	34.3%	17.0%	48.7% **
Table 3: Human evaluation on Relevance (Rel.), Informativeness (Inf.), and Human-likeness (Hum.). Statistical significant results:  $**p < 0.001$ ,  $* p < 0.01$ .
6.4 Summarization
We further conducted evaluation on summarization and present the results in Tab. 4. Summarization is less open-ended than the text completion task, thus a deterministic decoding approach like T5-search can produce high-quality text. Our evaluation shows that in comparison with T5-sample and Genie (Lin et al., 2022), PLANNER exhibits comparable Rouge-L scores, while improves other metrics. PLANNER achieves higher AuBLEU than Genie (See App. D for more details).
Owing to the sampling nature of the diffusion model, PLANNER and Genie yielded lower Rouge-L scores in comparison with T5-search, with single summary. To align with Genie's evaluation, we provide the results with 10 random runs in Tab.4, where for each document it generates 10 summaries and the one with the highest Rouge score is used. However, we note that these summaries with
best Rouge-L cannot be predetermined without an oracle summary. Comparing with T5-search, PLANNER generates more diverse and less repetitive summaries. However, the improvement is less conspicuous comparing with the results observed in open-ended text completion tasks. We show some generations in the App. E (Tab. 12).
Notably, the generated content may occasionally include hallucinations or errors, especially for name entities and digits (App. E, Tab. 13). Such occurrences can be attributed to either the conversion errors in  $\mathcal{D}$  or errors during the generation of paragraph embeddings, and requires further investigation.
Arch.	PPL	DIST/ENT↑	S-BL↓	Rep-4↓	BL↑	R-L↑	Score↑	Len	AuBL↑
CNN Dailymail dataset
T5-search	58.12	0.11/7.726	0.24	6.69%	7.66	34.48	0.66	45.51	-
T5-sample	67.58	0.11/7.790	0.20	3.50%	5.05	30.15	0.64	48.51	-
Genie	179.9	0.09/7.293	0.24	4.16%	3.22	30.47	0.58	40.94	27.21
\( \text{Genie}^{(10)} \)	170.6	0.10/7.355	0.24	4.32%	6.48	37.09	0.62	40.81	-
PLANNER	49.21	0.10/8.037	0.15	5.25%	6.92	30.43	0.62	52.33	43.91
\( \text{PLANNER}^{(10)} \)	49.07	0.10/8.019	0.15	4.96%	11.42	36.81	0.66	53.14	-
Human	49.477	0.12/8.226	0.16	5.63%	-	-	-	51.15	-
XSum dataset
T5-search	29.41	0.12/7.200	0.31	14.83%	6.11	36.08	0.74	18.97	-
T5-sample	36.17	0.13/7.449	0.24	6.47%	3.62	31.18	0.71	20.78	-
Genie	186.7	0.09/6.935	0.28	8.56%	2.38	34.85	0.66	20.44	30.85
\( \text{Genie}^{(10)} \)	178.2	0.09/6.924	0.30	9.66%	5.06	41.59	0.68	19.97	-
PLANNER	67.94	0.11/7.553	0.21	5.38%	4.84	33.97	0.69	20.04	57.88
\( \text{PLANNER}^{(10)} \)	67.46	0.11/7.529	0.23	5.82%	11.61	41.23	0.72	19.89	-
Human	37.8	0.13/7.656	0.21	5.56%	-	-	-	21.19	-
Table 4: For summarization task, PLANNER outperform Genie (Lin et al., 2022) in generation diversity and fluency while maintaining comparable Rouge-L scores.  ${ }^{(10)}$  indicates the maximum results after 10 runs, following (Lin et al., 2022).
6.5 Analysis
Running time We conducted inference time benchmarks of each method on a single Nvidia A100. For the sentiment-guided generation task, the autoregressive baseline is  $5\mathrm{x}$  faster than our method as the generation for all methods can be batched. For all other tasks, the varying input lengths make direct batchification for the FT baseline not straightforward. In these scenarios, the latent diffusion over a fixed number of latent codes offers computational advantages over a naive decoding of the FT baseline as the latent codes in our method can be conveniently batched.
For the hotel review completion task, the generation of 256 samples took 378 seconds to complete, including 83 seconds for decoding and 295 seconds for diffusion generation with 30 generation steps. The unbatched FT baseline took 1,693 seconds to complete 256 generations. Sorting input text by length and maximally batchifying them as possibly reduce the (batched) FT inference time to 338 seconds. The Diff-LM algorithm required 397 seconds to produce 256 samples using 200 generation steps, which is comparable to ours. Although our method is slower than the autoregressive ones, PLANNER enjoys the convenience of arranging input into the same length vectors without further length bucketing. On the CNN-DM summarization tasks, our method took 8.4 GPU hours to generate 11392 summaries. Genie's generation took 47.2 GPU hours. XSum gives similar inference running time benchmark to the results on CNN-DM.
Generations over diffusion steps In App. we provide generation examples for both summarization and sentiment-guided generation over different diffusion steps, which progress in a coarse-to-fine manner. The generation from early time step tends to be less fluent and generic. As the time approaches 0, the generation becomes more detailed. We presented quantitative results characterizing the evolution of the metrics over generation steps in App. Fig. It revealed a clear trend of improvement in the majority of the metrics as the generation proceeds. Notably, most hallucinations
occur during the late phase when more details are being incorporated. The model may excessively emphasize certain aspects, resulting in the correct generation being altered to an erroneous one (App. B, Tab. 10).
We present a two-stage latent text diffusion model that uses an autoencoder to condense lengthy texts into a limited number of paragraph embeddings, and a continuous time diffusion model that learns the distribution of these embeddings. Our proposed model alleviates the issue of repetition and advances generation diversity across different tasks. Compared to text diffusion models that perform diffusion solely on token or token embedding space, our method generates fluent text with improved diversity and reduced repetition. There may be toxicity or fairness issues in the dataset we used that we have not been able to identify. There are several limitations that warrant further investigation. Our work relies on an autoregressive decoder for converting latent representation into coherent text. It is possible to explore the feasibility of non-autoregressive decoders to bolster efficiency while minimizing conversion errors and hallucination in the generation. Furthermore, the classifier-free guidance approach results in a discrepancy between training and inference data distribution when feeding to the diffusion backbone. It would be interesting to investigate a "calibration" strategy for the latent code to better fit the data distribution during training.
We thank Yinfei Yang, Barry Theobald, Zhe Gan, Edouard Grave, David Grangier, Tatiana Likhomanenko, Richard Bai and Ronan Collobert for their critical suggestions and helpful feedback throughout this project.
Yizhe Zhang, Zhe Gan, Kai Fan, Zhi Chen, Ricardo Henao, Dinghan Shen, and Lawrence Carin. Adversarial feature matching for text generation. In ICML, pp. 4006-4015. PMLR, 2017.
Yizhe Zhang, Michel Galley, Jianfeng Gao, Zhe Gan, Xiujun Li, Chris Brockett, and Bill Dolan. Generating informative and diverse conversational responses via adversarial information maximization. In NeurIPS, 2018.
Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. Texygen: A benchmarking platform for text generation models. In The 41st international ACM SIGIR conference on research & development in information retrieval, pp. 1097-1100, 2018.
Zixin Zhu, Yixuan Wei, Jianfeng Wang, Zhe Gan, Zheng Zhang, Le Wang, Gang Hua, Lijuan Wang, Zicheng Liu, and Han Hu. Exploring discrete diffusion models for image captioning. arXiv preprint arXiv:2211.11694, 2022.