SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We also want to quantify the Diversity (Div) of generations. We define diversity as  $\mathbf{Div} = \prod_{n=2}^{4} \frac{\left|\text{unique n-grams}\left(\left\{\mathbf{w}_i\right\}\right)\right|}{\left|\text{total n-grams}\left(\left\{\mathbf{w}_i\right\}\right)\right|}$  where  $\{\mathbf{w}_i\}$  is a set of generated samples [68]. The metrics discussed so far can be optimized by generating samples from the training set. We measure the proportion of generated 4-grams that are found in the training set to quantify the degree of Memorization (Mem).
To evaluate the performance for monolingual seq2seq language generation tasks, we utilize Rouge [37] and BERTScore [75]. Rouge-1/2 measures the number of unigrams/bigrams in the reference that appear in the generated text and Rouge-L measures the longest common sequence between the texts. BERTScore uses contextual embeddings from a pretrained language model to measure the similarity between texts. We follow prior work and use the microsoft/deberta-xlarge-mnli model [18] to extract contextual embeddings. For our machine translation experiments, we report SacreBLEU scores [5] to ensure fair comparison with prior work.
For our unconditional and class-conditional language generation experiments, we sample 1000 instances from the diffusion model. For the MAUVE reference text, we sample 1000 instances from the test set. We repeat this 5 times and report the mean and standard deviation as mean $_{\text{stdev}}$ . We also compute reference values for our metrics with natural samples from the test set. The reference MAUVE, for instance, is computed between 1000 train and 1000 test samples. Qualitative samples from our models are in the supplemental materials.
5.1 Language Autoencoder
We evaluate the effectiveness of our proposed language autoencoder using heldout examples from our datasets. As a point of comparison, we also evaluate the default behavior of the language models that we use to develop the language autoencoders. A consequence of BART's particular denoising objective is that the pretrained model already generates a copy of the input language, although this is not true of other models such as T5 or FLAN-T5.
We present results for our two most complex datasets, ROCStories and AG News, in Table 1 and present the results for XSum, QQP, and WMT14-En-De, which show similar trends, in the appendix. We observe that our BART-base autoencoder is able to compress the feature space by a factor of  $24 \times$  while improving the fidelity of the reconstructions. Our autoencoding modules are also effective at converting the pretrained FLAN-T5 into a language autoencoder, even though that is different from the model's default behavior. Across both models and all datasets, our language autoencoders are able to achieve near-perfect reconstruction with a low-dimensional latent space.
5.2 Unconditional Language Generation
Baselines. We evaluate our approach's capacity for unconditional language generation with the ROCStories and AG News datasets. We compare against the recently proposed Diffusion-LM model
Table 1: Effectiveness of Language Autoencoder
Method	Latent Dimensions	Hidden Units	RocStories	AG News
Rouge-1/2/L	BLEU	Rouge-1/2/L	BLEU
BART-Base	L × 768	≤ 49,152	98.9/98.2/98.8	97.5	99.6/99.4/99.6	98.6
BART-Base Autoencoder	32 × 64	2048	99.2/98.5/99.2	97.6	99.7/99.4/99.7	98.8
FLAN-T5-Base	L × 768	≤ 49,152	21.5/11.8/19.4	0.7	63.6/53.0/59.6	42.3
FLAN-T5-Base Autoencoder	32 × 64	2048	98.4/96.9/98.4	95.8	99.1/98.3/99.1	96.8
[36]. We also fine-tune the pretrained GPT-2-Medium model, which is roughly  $1.6 \times$  larger than our denoising network, as a strong autoregressive baseline [49]. For sampling from GPT-2, we prompt it with a BOS token and utilize nucleus sampling ( $p = 0.95$ ) [24]. We explore different sampling configurations in the appendix and find that they lead to similar conclusions.
Results. We present this comparison in Table 2. We observe that our approach is significantly more effective than Diffusion-LM at modeling language distributions, as demonstrated by the higher MAUVE scores, while requiring fewer sampling steps. Diffusion-LM is unable to model diverse language distributions and exhibits poor diversity. Utilizing high quality latent spaces from pretrained language models improves the effectiveness of our diffusion model. We observe that both language models are highly effective for the AG News dataset, but using BART-base leads to a stronger MAUVE score for the ROCStories dataset. Across both datasets, FLAN-T5-base produces more diverse generations and exhibits less memorization.
While GPT-2 generally achieves strong language generation metrics, it is more susceptible to memorization than LD4LG. For the AG News dataset, GPT-2 exhibits significant memorization and a lower MAUVE score. We do find that GPT-2 samples have lower perplexity. However, measuring perplexity with a pretrained GPT-2 model likely biases the metric towards the fine-tuned GPT-2 model. Moreover, MAUVE scores have a stronger correlation with human judgments of quality [47].
Table 2: Unconditional Language Generation Evaluation. The fine-tuned language model is presented in gray.
Timesteps	ROCStories	AG News
MAUVE ↑	Ppl ↓	Div ↑	Mem ↓	MAUVE ↑	Ppl ↓	Div ↑	Mem ↓
Reference	-	.951.007	21.1.3	.414.003	.362.003	.951.014	43.61.2	.658.002	.385.005
Diffusion-LM [36]	2000	.043.006	47.3.6	.128.002	.434.002	.012.001	67.11.2	.043.002	.086.006
LD4LG (BART-Base)	250	.716.019	30.6.5	.331.005	.441.004	.866.016	100.62.9	.540.006	.293.001
LD4LG (FLAN-T5-base)	250	.481.007	37.5.4	.389.002	.387.002	.859.020	122.03.9	.624.008	.221.003
GPT-2-Medium	-	.788.025	20.0.2	.372.002	.688.006	.820.012	37.31.1	.532.017	.829.005
Benefits of Compression. Because the pretrained BART model already copies the input text, we can ablate the impact of learning a compact latent space by learning a diffusion model directly in the encoder feature space. One complication of this setting is that the sequence length of the BART features vary. During training, the sequence length is simply determined by the sample. During generation, however, we must specify the length. To determine the sequence length for generation, we opt to sample a length from the empirical distribution of lengths in the training set. We refer to this baseline as BART-Diffusion and outline full implementation details in the appendix.
We compare BART-Diffusion with our proposed approach in Table 3. We quantify the speedup by measuring how long it takes each approach to match the peak validation MAUVE of BART-Diffusion. We observe that learning a compact latent space is beneficial both in terms of absolute performance and wall-clock time, reaching the peak MAUVE of BART-diffusion in a quarter of the time. Compressing the latent space along the sequence dimension significantly reduces the overhead per iteration due to the quadratic cost of self-attention, and we also observe faster convergence.
Self-conditioning. We ablate the impact of self-conditioning in Table 4. We find that it significantly improves the MAUVE score and the perplexity of the generated text, but sacrifices some diversity.
Table 3: Benefits of Compression (ROCStories)
Hidden Units	Relative Speedup	MAUVE ↑	Ppl ↓	Div ↑	Mem ↓
BART-Diffusion	≤ 49,152	1.0×	.605 .024	46.8 .7	.424 .004	.304 .003
LD4LG (BART-base)	2048	3.86×	.716 .019	30.6 .5	.331 .005	.441 .004
Table 4: Impact of Self-Conditioning (ROCStories)
MAUVE ↑	Ppl ↓	Div ↑	Mem ↓
LD4LG (BART-base)	.716 .019	30.65	.331 .005	.441 .004
- Self-cond.	.480 .018	79.31 .0	.427 .004	.299 .003
Table 5: Metrics for class-conditional generation.
Conditioning	LD4LG (BART-base)	LD4LG (FLAN-T5-base)
MAUVE ↑	Mem ↓	MAUVE ↑	Mem ↓
World	Sports	Business	Sci/Tech	World	Sports	Business	Sci/Tech
Diffusion	World	.842 .017	.015 .002	.026 .002	.020 .002	.296 .002	.809 .024	.013 .001	.025 .002	.022 .002	.233 .005
Sports	.013 .001	.845 .024	.011 .001	.010 .000	.305 .003	.011 .001	.836 .020	.009 .000	.009 .000	.249 .004
Business	.024 .002	.011 .001	.752 .030	.068 .005	.363 .009	.025 .003	.011 .001	.765 .016	.076 .008	.244 .004
Sci/Tech	.023 .002	.012 .001	.082 .008	.813 .028	.225 .004	.024 .001	.011 .001	.082 .010	.843 .033	.169 .004
Comparisons	Conditional GPT-2	Reference
MAUVE ↑	Mem ↓	MAUVE ↑	Mem ↓
World	Sports	Business	Sci/Tech	World	Sports	Business	Sci/Tech
Diffusion	World	.805 .022	.012 .000	.025 .002	.021 .002	.402 .002	.963 .009	.018 .001	.034 .002	.032 .003	.388 .007
Sports	.017 .001	.840 .019	.012 .001	.013 .001	.369 .004	.018 .001	.958 .007	.014 .001	.014 .002	.346 .002
Business	.037 .003	.012 .001	.629 .029	.069 .007	.479 .007	.040 .005	.014 .001	.968 .009	.125 .009	.441 .003
Sci/Tech	.033 .002	.013 .001	.102 .015	.697 .027	.434 .004	.036 .003	.016 .001	.133 .013	.955 .011	.366 .003
5.3 Class-Conditional Language Generation
Baselines. Conditional training with control tokens is one of the most widely used methods for controlling autoregressive models [14, 29, 40, 33]. We prepend the class label to each sample as a control token and fine-tune GPT-2-medium for class-conditional generation. Because memorizing the training instances associated with each class is a trivial solution, we terminate training when the model's memorization exceeds the reference values.
Results. We evaluate the effectiveness of class-conditioning with the AG News topic classification dataset. We sample instances for each class and compute the MAUVE scores between natural instances from each class. We report these metrics in Table 5. We observe that the MAUVE scores are highest when the conditioning and ground-truth labels are aligned across all methods, demonstrating that the label guides the generation effectively. We observe that our approach is more consistently effective at class-conditional generation, particularly for the two most similar classes, business and sci/tech. The GPT-2 baseline is again more susceptible to memorization than our approach.
5.4 Sequence-to-Sequence Language Generation
Baselines. We compare against directly fine-tuning BART-base and FLAN-T5-base on the XSum summarization and QQP paraphrasing datasets. For diffusion baselines, we compare against the following continuous diffusion models learned in the space of word embeddings: DiffuSeq [16], CDCD [13], DINOISER [73], and GENIE [38]. We also compare against the following discrete diffusion models which learn to invert discrete corruption processes (e.g. masking): Reparameterized Discrete Diffusion (RDM) [76] and DiffusionBERT [19]. We compare directly against the metrics reported in prior work on our datasets. For XSum, we additionally train a DiffuSeq model using the official implementation. We note that Gong et al. [16] typically train their models much longer than ours. The XSum DiffuSeq model, for instance, is trained for over  $3 \times$  more epochs than our approach.
For machine translation, we compare directly against the prior work that reported SacreBLEU scores to ensure a fair comparison [48].
Table 6: Seq2Seq Evaluation on QQP. Results from fine-tuned language models are in gray.
Method	Sampling	Rouge-1/2/L ↑	BERTScore ↑	from fine-tuned language models are in gray.
DiffuSeq [16]	Random	55.2/29.2/52.7	82.4	Method	Sampling	Rouge-1/2/L ↑	BERTScore ↑
RDM-absorbing [76]	Random	—/—/57.9	83.7	DiffuSeq [16]	Random	18.9/1.3/13.6	46.8
RDM-multinomial [76]	Random	—/—/57.3	83.7	LD4LG (BART-base)	Random	37.6/15.5/30.8	74.1
LD4LG (BART-base)	Random	62.6/39.0/60.3	85.8	LD4LG (FLAN-T5-Base)	Random	38.1/15.9/31.2	74.8
LD4LG (FLAN-T5-Base)	Random	62.1/38.4/59.7	85.8	DiffuSeq [16]	MBR-5	19.3/1.7/14.1	46.9
DiffuSeq [16]	MBR-10	—/—/58.8	83.7	LD4LG (BART-base)	MBR-5	38.2/16.2/31.5	74.5
RDM-absorbing [76]	MBR-10	—/—/59.5	84.7	LD4LG (FLAN-T5-Base)	MBR-5	38.7/16.6/31.9	75.2
RDM-multinomial [76]	MBR-10	—/—/58.5	84.7	DiffuSeq [16]	Oracle-5	23.5/2.3/18.6	47.9
DiffusionBERT [19]	MBR-10	—/—/58.9	—	GENIE [38]	Oracle-5	37.3/15.3/29.4	—
LD4LG (BART-base)	MBR-5	63.3/40.3/61.1	86.2	GENIE w/pre-training [38]	Oracle-5	41.2/19.1/33.4	—
LD4LG (FLAN-T5-Base)	MBR-5	63.0/39.7/60.7	86.1	LD4LG (BART-base)	Oracle-5	42.4/19.4/36.4	75.3
DiffuSeq [16]	Oracle-5	67.4/43.9/65.8	83.7	LD4LG (FLAN-T5-Base)	Oracle-5	43.0/20.0/37.2	76.1
DiffuSeq [16]	Oracle-5	68.0/46.6/66.0	87.2	BART-Base	Nucleus	35.1/13.3/27.7	73.1
LD4LG (BART-base)	Oracle-5	67.8/46.0/65.7	87.2	FLAN-T5-Base	Nucleus	34.6/12.9/27.2	72.7
BART-Base	Nucleus	51.5/28.1/48.3	79.9	BART-Base	Beam	39.9/18.0/32.6	75.6
FLAN-T5-Base	Nucleus	55.0/30.1/52.3	83.2	FLAN-T5-Base	Beam	39.7/17.7/32.3	75.3
BART-Base	Beam	61.9/39.0/59.5	85.5
FLAN-T5-Base	Beam	63.0/40.1/60.5	86.2
Results. We present our comparison on QQP and XSum in Table 6 and Table 7. Our approach significantly outperforms recent diffusion language models across both datasets, especially for the more challenging XSum dataset. For instance, DiffuSeq is reasonably effective for QQP, but it struggles with XSum and fails to generate coherent text (see samples in appendix). Our method, on the other hand, is competitive with fine-tuning. LD4LG narrowly outperforms fine-tuning on QQP with MBR decoding, but the fine-tuned models are slightly more effective on the XSum dataset. Across both datasets, LD4LG with oracle sampling outperforms all approaches (including direct fine-tuning methods) with just 5 random samples. This demonstrates that LD4LG has good coverage,
but MBR decoding does not consistently identify the best candidate. In our experiments, we use classifier-free guidance with guidance strength  $w = 2.0$ . We ablate this choice with validation samples in Figure 3 and observe that such guidance meaningfully improves performance.
We report our machine translation results in Table 8. We observe that LD4LG outperforms the Diffusion-LM and CDCD baselines although it lags behind the DINOISER baseline. This demonstrates that our method can effectively take advantage of strong pre-trained multilingual language models for effective multilingual generation.
6 Future Work
Our experiments demonstrate that latent language diffusion models can generate high-quality natural language in a variety of settings. In continuous domains, diffusion models are remarkably effective for applications ranging from image editing [41] to solving inverse problems [64]. We are excited to explore the potential applications enabled by effective language diffusion models. We expect that LD4LG is a natural fit for applications such as language editing (e.g. style transfer) and controllable generat
Table 7: Seq2Seq Evaluation on XSum. Results from fine-tuned language models are in gray.
Table 8: Machine translation results on WMT14-En-De. Baseline results are from [13, 73].
Method	Sampling	SacreBLEU
En→De	De→En
CDCD [13]	Random	19.3	24.9
LD4LG (MT5-base)	Random	21.4	26.2
Diffusion-LM [36]	MBR-5	15.3	17.3
CDCD [13]	MBR-10	19.7	25.4
DINOISER [73]	MBR-5	24.3	28.8
LD4LG (MT5-base)	MBR-5	22.4	27.0
Figure 3: Ablation of classifier-free guidance on the XSum summarization benchmark.
Despite achieving good performance, LD4LG has important limitations. Sampling from diffusion models is slow due to the iterative generative process. LD4LG improves upon some prior continuous text diffusion models (that use 2000 steps) and only uses 250 sampling steps. However, speeding up the inference process of diffusion models is an active area of research and techniques developed for image diffusion can likely be adapted for LD4LG [65, 57]. Song et al. [65], for instance, distilled a trained image diffusion model to produce high-quality samples in a single step. We leave the extension of such techniques to language generation as future work. In subsection 5.4, we observe that diffusion models have excellent coverage but MBR decoding fails to identify the best candidate; developing improved sampling procedures or candidate re-ranking methods would likely improve performance for tasks such as summarization and machine translation.
Diffusion models. Diffusion models [61, 22] are a class of generative models that have led to impressive results in image synthesis, recently surpassing Generative Adversarial Networks [17, 12]. These models typically operate directly in pixel-space, learning a distribution over images. Rombach et al. [52] introduced latent diffusion for image synthesis and demonstrated that they can be learned in the latent space of a pretrained autoencoder. Latent diffusion has since been successful in other domains such as audio synthesis [32], symbolic music generation [58], and molecule generation [71].
Diffusion for Language. Prior work has focused on directly modeling discrete data by designing diffusion processes for discrete state spaces [25, 3, 26]. Li et al. [36] train a continuous diffusion model in the space of token embeddings that are learned jointly with the denoising objective and decode generations with a rounding step. Strudel et al. [67] scaled up this approach and instead learn the diffusion model in the space of pretrained word embeddings and find that low-dimensional embeddings are better suited for diffusion. Gong et al. [16] extend Diffusion-LM [36] to sequence-to-sequence tasks by concatenating the source and target sequence and only performing diffusion for the target sequence. Chen et al. [8] map words to arbitrary binary strings, represented as a sequence of real numbers. They then train a continuous diffusion model and round the generated sequences to produce binary strings. The authors also introduce self-conditioning, which we adopt for our method.
In this work, we demonstrate that latent diffusion is an effective paradigm for language generation. To achieve this, we introduce a method for compressing the high-dimensional, variable-length language representations from pre-trained language models into a compact, fixed-size latent representation that can be decoded into natural language. This compact latent representation is, by design, well-suited for learning continuous latent diffusion models. Our latent language diffusion models are effective for unconditional, class-conditional, and sequence-to-sequence language generation. They offer some benefits over fine-tuned auto-regressive language models and significantly outperform recent diffusion language models across a variety of datasets.
This research is supported by grants from the National Science Foundation NSF (IIS-2107161, and IIS-1724282, HDR-2118310). The Cornell Center for Materials Research with funding from the NSF MRSEC program (DMR-1719875), DARPA, arXiv, LinkedIn, and the New York Presbyterian Hospital.
[1] Chi-square Distribution, pages 70-72. Springer New York, New York, NY, 2008. ISBN 978-0-387-32833-1. doi: 10.1007/978-0-387-32833-1_54. URL https://doi.org/10.1007/978-0-387-32833-1_54.
[2] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual
[68] Yixuan Su, Tian Lan, Yan Wang, Dani Yogatama, Lingpeng Kong, and Nigel Collier. A contrastive framework for neural text generation. arXiv preprint arXiv:2202.06417, 2022.
[69] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS'17, page 6000-6010, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 9781510860964.
[70] Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tieyan Liu. On layer normalization in the transformer architecture. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 10524-10533. PMLR, 13-18 Jul 2020. URL https://proceedings.mlr.press/v119/xiong20b.html.
[71] Minkai Xu, Alexander Powers, Ron Dror, Stefano Ermon, and Jure Leskovec. Geometric latent diffusion models for 3d molecule generation. arXiv preprint arXiv:2305.01140, 2023.
[72] Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mT5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 483-498, Online, June 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.naacl-main.41. URL https://aclanthology.org/2021.nacl-main.41.
[73] Jiasheng Ye, Zaixiang Zheng, Yu Bao, Lihua Qian, and Mingxuan Wang. Dinoiser: Diffused conditional sequence learning by manipulating noises. arXiv preprint arXiv:2302.10025, 2023.
[74] Biao Zhang and Rico Sennrich. Root mean square layer normalization. Advances in Neural Information Processing Systems, 32, 2019.
[75] Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675, 2019.
[76] Lin Zheng, Jianbo Yuan, Lei Yu, and Lingpeng Kong. A reparameterized discrete diffusion model for text generation. arXiv preprint arXiv:2302.05737, 2023.
A Diffusion Models
We present a formal description of diffusion [22, 62, 30]. Diffusion models are latent variable models with latents  $\mathbf{z} = \{\mathbf{z}_t|t\in [0,1]\}$  that are given by the forward diffusion process  $q(\mathbf{z}|\mathbf{x})$ , with the data,  $\mathbf{x}\sim p(\mathbf{x})$ , being drawn from an unknown distribution.
The forward process is a Markovian process that iteratively adds Gaussian noise to the data over time
$$
q (\mathbf {z} _ {t} | \mathbf {x}) = \mathcal {N} (\mathbf {z} _ {t}; \sqrt {\alpha_ {t}} \mathbf {x}, (1 - \alpha_ {t}) \mathbf {I}), q (\mathbf {z} _ {t} | \mathbf {z} _ {s}) = \mathcal {N} (\mathbf {z} _ {t}; \sqrt {\alpha_ {t | s}} \mathbf {z} _ {s}, (1 - \alpha_ {t | s}) \mathbf {I})
$$
where  $\alpha_{t|s} = \alpha_t / \alpha_s$  and  $0\leq s < t\leq 1$ . The noise schedule, specified by  $\alpha_{t}\in [0,1]$ , decreases with  $t$  until the final latent becomes approximately Gaussian,  $q(\mathbf{z}_1)\approx \mathcal{N}(\mathbf{z}_1;\mathbf{0},\mathbf{I})$  — independent of the original data. The forward process therefore defines a transition from the data distribution to a Gaussian distribution.
Given access to the original data  $\mathbf{x}$ , the forward process can be inverted analytically. For  $t > s$ , we have
$$
q (\mathbf {z} _ {s} | \mathbf {z} _ {t}, \mathbf {x}) = \mathcal {N} (\boldsymbol {\mu} _ {Q} (\mathbf {z} _ {t}, \mathbf {x}, s, t), \sigma_ {Q} ^ {2} (s, t) \mathbf {I})
$$
$$
\pmb {\mu} _ {Q} (\mathbf {z} _ {t}, \mathbf {x}, s, t) = \frac {\sqrt {\alpha_ {s}} (1 - \alpha_ {t | s})}{1 - \alpha_ {t}} \mathbf {x} + \frac {\sqrt {\alpha_ {t | s}} (1 - \alpha_ {s})}{1 - \alpha_ {t}} \mathbf {z} _ {t}, \quad \sigma_ {Q} ^ {2} (s, t) = \frac {(1 - \alpha_ {s}) (1 - \alpha_ {t | s})}{1 - \alpha_ {t}}.
$$
We utilize this to define our generative process. Because  $\mathbf{x}$  is unavailable during generation, we train a neural network to approximate the original data given some noisy latent and the timestep,  $\hat{\mathbf{x}}_{\theta}(\mathbf{z}_t,t)\approx \mathbf{x}$ . The denoising network is trained utilizing a regression loss
with some time-dependent weighting  $\lambda_{t}$ . This loss function can be motivated as the weighted variational lower bound of the log likelihood of the data under the forward diffusion process [22, 31]. In practice, the denoising network is often parameterized as an  $\epsilon$ -prediction network [22] or a v-prediction network [57] where the velocity,  $\mathbf{v}$ , is defined as  $\mathbf{v} = \sqrt{\alpha_t}\epsilon - \sqrt{1 - \alpha_t}\mathbf{x}$ . These parameterizations can be interpreted as different weighting functions,  $\lambda_{t}$ , for the regression objective [57]. We adopt the v-parameterization throughout this work.
With a trained denoising network, we define our generative process as
$$
p _ {\theta} (\mathbf {z} _ {s} | \mathbf {z} _ {t}) = \mathcal {N} (\mathbf {z} _ {s}; \boldsymbol {\mu} _ {\theta} (\mathbf {z} _ {t}, s, t), \sigma^ {2} (s, t) \mathbf {I})
$$