SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Implementing the sampler During sampling, we can grow the tail statistic  $G_{t}$  without any overhead, as described in Algorithm 2. However, during training, we need to sample the tail statistic for each object to estimate the loss function. For this we need to sample the full tail  $x_{t:T}$  from the forward process  $q^{\mathrm{SS}}(x_{t:T}|x_0)$ , and then compute the tail statistic  $G_{t}$ . In practice, this does not add a noticeable overhead and can be computed in parallel to the training process if needed.
Reducing the number of steps We can sample from DDPMs more efficiently by skipping some timestamps. This wouldn't work for SS-DDPM, because changing the number of steps would require changing the noising schedule and, consequently, retraining the model.
However, we can still use a similar trick to reduce the number of function evaluations. Instead of skipping some timestamps  $x_{t_1 + 1:t_2 - 1}$ , we can draw them from the forward process using the current prediction  $x_{\theta}(G_{t_2}, t_2)$ , and then use these samples to obtain the tail statistic  $G_{t_1}$ . For Gaussian SS-DDPM this is equivalent to skipping these timestamps in the corresponding DDPM. In general case, it amounts to approximating the reverse process with a different reverse process:
$$
p _ {\theta} ^ {\mathrm {s s}} \left(x _ {t _ {1}: t _ {2}} \mid G _ {t _ {2}}\right) = \prod_ {t = t _ {1}} ^ {t _ {2}} q ^ {\mathrm {s s}} \left(x _ {t} \mid x _ {0}\right) | _ {x _ {0} = x _ {\theta} \left(G _ {t}, t\right)} \approx \prod_ {t = t _ {1}} ^ {t _ {2}} q ^ {\mathrm {s s}} \left(x _ {t} \mid x _ {0}\right) | _ {x _ {0} = x _ {\theta} \left(G _ {t _ {2}}, t _ {2}\right)}. \tag {25}
$$
We observe a similar dependence on the number of function evaluations for SS-DDPMs and DDPMs.
Time-dependent tail normalization As defined in Theorem 1, the tail statistics can have vastly different scales for different timestamps. The values of coefficients  $a_{t}$  can range from thousandths when  $t$  approaches  $T$  to thousands when  $t$  approaches zero. To make the tail statistics suitable for use in neural networks, proper normalization is crucial. In most cases, we collect the time-dependent means and variances of the tail statistics across the training dataset and normalize the tail statistics to zero mean and unit variance. We further discuss this issue in Appendix H.
Architectural choices To make training the model easier, we make some minor adjustments to the neural network architecture and the loss function.
Our neural networks  $x_{\theta}(G_t, t)$  take the tail statistic  $G_t$  as an input and are expected to produce an estimate of  $x_0$  as an output. In SS-DDPM the data  $x_0$  might lie on some manifold, like the unit sphere or the space of positive definite matrices. Therefore, we need to map the neural network output to that manifold. We do that on a case-by-case basis, as described in Appendices I-L.
Different terms of the VLB can have drastically different scales. For this reason, it is common practice to train DDPMs with a modified loss function like  $L_{simple}$  rather than the VLB to improve the stability of training (Ho et al., 2020). Similarly, we can optimize a reweighted variational lower bound when training SS-DDPMs.
Our work builds upon Denoising Diffusion Probabilistic Models (Ho et al., 2020). Interest in diffusion models has increased recently due to their impressive results in image (Ho et al., 2020; Song et al., 2020b; Dhariwal & Nichol, 2021) and audio (Popov et al., 2021; Liu et al., 2022) generation.
SS-DDPM is most closely related to DDPM. Like DDPM, we only rely on variational inference when defining and working with our model. SS-DDPM can be seen as a direct generalization of DDPM and essentially is a recipe for defining DDPMs with non-Gaussian distributions. The underlying non-Gaussian DDPM is constructed implicitly and can be seen as dual to the star-shaped formulation that we use throughout the paper.
Other ways to construct non-Gaussian DDPMs include Binomial diffusion (Sohl-Dickstein et al., 2015), Multinomial diffusion (Hoogeboom et al., 2021) and Gamma diffusion (Nachmani et al., 2021). Each of these works presents a separate derivation of the resulting objective, and extending them to other distributions is not straightforward. On the other hand, SS-DDPM provides a single recipe for a wide range of distributions.
DDPMs have several important extensions. Song et al. (2020a) provide a family of non-Markovian diffusions that all result in the same training objective as DDPM. One of them, denoted Denoising Diffusion Implicit Model (DDIM), results in an efficient deterministic sampling algorithm that requires a much lower number of function evaluations than conventional stochastic sampling. Their derivations also admit a star-shaped forward process (at  $\sigma_t^2 = 1 - \alpha_{t-1}$ ), however, the model is not studied in the star-shaped regime. Their reverse process remains Markovian, which we show to not be sufficient to invert a star-shaped forward process. Denoising Diffusion Restoration Models (Kawar et al., 2022) provide a way to solve general linear inverse problems using a trained DDPM. They can be used for image restoration, inpainting, colorization and other conditional generation tasks. Both DDIMs and DDRMs rely on the explicit form of the underlying DDPM model and are derived for Gaussian diffusion. Extending these models to SS-DDPMs is a promising direction for future work.
Song et al. (2020b) established the connection between DDPMs and models based on score matching. This connection gives rise to continuous-time variants of the models, deterministic solutions and more precise density estimation using ODEs. We suspect that a similar connection might hold for SS-DDPMs as well, and it can be investigated further in future works.
Other works that present diffusion-like models with other types of noise or applied to manifold data, generally stem from score matching rather than variational inference. Flow Matching (Lipman et al., 2022) is an alternative probabilistic framework that works with any differentiable degradation process. De Bortoli et al. (2022) and Huang et al. (2022) extended score matching to Riemannian manifolds, and Chen & Lipman (2023) proposed Riemannian Flow Matching. Bansal et al. (2022) proposed Cold Diffusion, a non-probabilistic approach to reversing general degradation processes.
To the best of our knowledge, for the first time, an approach for constructing diffusion without a consecutive process was proposed by Rissanen et al. (2022) (IHDM) and further expanded on by Daras et al. (2022) and Hoogeboom & Salimans (2022). IHDM uses a similar star-shaped structure that results in a similar variational lower bound. Adding a deterministic process based on the heat equation allows the authors to keep the reverse process Markovian without having to introduce the tail statistics. As IHDM heavily relies on blurring rather than adding noise, the resulting diffusion dynamics become very different. Conceptually our work is much closer to DDPM than IHDM.
We evaluate SS-DDPM with different families of noising distributions. The experiment setup, training details and hyperparameters are listed in Appendices I-L.
Synthetic data We consider two examples of star-shaped diffusion processes with Dirichlet and Wishart noise to generate data from the probabilistic simplex and from the manifold of p.d. matrices respectively. We compare them to DDPM, where the predictive network  $x_{\theta}(x_{t}, t)$  is parameterized to always satisfy the manifold constraints. As seen in Table 1, using the appropriate distribution results in a better approximation. The data and
modeled distributions are illustrated in Table 3. This shows the ability of SS-DDPM to work with different distributions and generate data from exotic domains.
Table 1: KL divergence between the real data distribution and the model distribution  $D_{KL}\left(q(x_0)\parallel p_\theta (x_0)\right)$  for Gaussian DDPM and SS-DDPM.
Dirichlet	Wishart
DDPM	0.200	0.096
SS-DDPM	0.011	0.037
Geodesic data We apply SS-DDPM to a geodesic dataset of fires on the Earth's surface (EOSDIS, 2020) using a three-dimensional von Mises-Fisher distribution. The resulting samples and the source data are illustrated in Table 3. We find that SS-DDPM is not too sensitive to the distribution family and can fit data in different domains.
Discrete data Categorical SS-DDPM is similar to Multinomial Text Diffusion (MTD, (Hoogeboom et al., 2021)). However, unlike in the Gaussian case, these models are not strictly equivalent. We follow a similar setup to MTD and apply Categorical SS-DDPM to unconditional text generation on the text8 dataset (Mahoney, 2011). As shown in Table 2, SS-DDPM achieves similar results to MTD, allowing to use different distributions in a unified manner. While D3PM (Austin et al., 2021) provides some improvements, we follow a simpler setup from MTD. We expect the improvements from D3PM to directly apply to Categorical SS-DDPM.
Table 2: Comparison of Categorical SSDPM and Multinomial Text Diffusion on text8. NLL is estimated via ELBO.
Model	NLL (bits/char)
MTD	≤ 1.72
SS-DDPM	≤ 1.69
Image data Finally, we evaluate SS-DDPM on CIFAR-10. Since the training data is constrained to a  $[0, 1]$  segment, we use Beta distributions. We evaluate our model with various numbers of generation steps, as described in equation (25), and report the resulting Fréchet Inception Distance (FID, (Heusel et al., 2017)) in Figure 6. Beta SS-DDPM achieves comparable quality with the Improved DDPM (Nichol & Dhariwal, 2021) and is slightly better on lower numbers of steps. As expected, DDIM performs better when the number of diffusion steps is low, but both SS-DDPM and DDPM outperform DDIM on longer runs. The best FID score achieved by Beta SS-DDPM is 3.17. Although the FID curves for DDPM and DDIM do not achieve this score in Figure 6, Ho et al. (2020) reported an FID score of 3.17 for 1000 DDPM steps, meaning that SS-DDPM performs similarly to DDPM in this setting.
Figure 6: Quality of images, generated using Beta SS-DDPM, DDPM and DDIM with different numbers of sampling steps. Models are trained and evaluated on CIFAR-10. DDPM and DDIM results were taken from Nichol & Dhariwal (2021).
Table 3: Experiments results. The first row is real data and the second is generated samples. For the von Mises-Fisher and Dirichlet models, we show two-dimensional histograms of samples. For the Wishart model, we draw ellipses, corresponding to the p.d. matrices  $x_0$  and  $x_{\theta}$ . The darker the pixel, the more ellipses pass through that pixel.
von Mises-Fisher	Dirichlet	Wishart
Real
Generated
We propose an alternative view on diffusion-like probabilistic models. We reveal the duality between star-shaped and Markov diffusion processes that allows us to go beyond Gaussian noise by switching to a star-shaped formulation. It allows us to define diffusion-like models with arbitrary noising distributions and to establish diffusion processes on specific manifolds. We propose an efficient way to construct a reverse process for such models in the case when the noising process lies in a general subset of the exponential family and show that star-shaped diffusion models can be trained on a variety of domains with different noising distributions. On image data, star-shaped diffusion with Beta distributed noise attains comparable performance to Gaussian DDPM, challenging the optimality of Gaussian noise in this setting. The star-shaped formulation opens new applications of diffusion-like probabilistic models, especially for data from exotic domains where domain-specific non-Gaussian diffusion is more appropriate.
We are grateful to Sergey Kholkin for additional experiments with von Mises-Fisher SS-DDDPM on geodesic data and to Tingir Badmaev for helpful recommendations in experiments on image data. We'd also like to thank Viacheslav Meshchaninov for sharing his knowledge of diffusion models for text data. This research was supported in part through computational resources of HPC facilities at HSE University. The results on image data (Image data in Section 5; Section L) were obtained by Andrey Okhotin and Aibek Alanov with the support of the grant for research centers in the field of AI provided by the Analytical Center for the Government of the Russian Federation (ACRF) in accordance with the agreement on the provision of subsidies (identifier of the agreement 000000D730321P5Q0002) and the agreement with HSE University No. 70-2021-00139.
Arjovsky, M., Chintala, S., and Bottou, L. Wasserstein gan. arxiv 2017. arXiv preprint arXiv:1701.07875, 30:4, 2017.
Karras, T., Aittala, M., Laine, S., Harkonen, E., Hellsten, J., Lehtinen, J., and Aila, T. Alias-free generative adversarial networks. Advances in Neural Information Processing Systems, 34, 2021.
Kawar, B., Elad, M., Ermon, S., and Song, J. Denoising diffusion restoration models. In Advances in Neural Information Processing Systems, 2022.
Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015.
Kingma, D. P. and Welling, M. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.
Kingma, D. P., Salimans, T., Poole, B., and Ho, J. Variational diffusion models. arXiv preprint arXiv:2107.00630, 2, 2021.
Kriskov, A., Stogbauer, H., and Grassberger, P. Estimating mutual information. Physical review E, 69(6):066138, 2004.
Lipman, Y., Chen, R. T., Ben-Hamu, H., Nickel, M., and Le, M. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747, 2022.
Liu, S., Su, D., and Yu, D. Diffgan-tts: High-fidelity and efficient text-to-speech with denoising diffusion gans. arXiv preprint arXiv:2201.11972, 2022.
Luo, S. and Hu, W. Diffusion probabilistic models for 3d point cloud generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2837-2845, 2021.
Mahoney, M. Large text compression benchmark, 2011. URL http://www.mattmahoney.net/dc/text.html.
Molchanov, D., Kharitonov, V., Sobolev, A., and Vetrov, D. Doubly semi-implicit variational inference. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 2593-2602. PMLR, 2019.
Nachmani, E., Roman, R. S., and Wolf, L. Denoising diffusion gamma models. arXiv preprint arXiv:2110.05948, 2021.
Nichol, A. and Dhariwal, P. Improved denoising diffusion probabilistic models. arXiv preprint arXiv:2102.09672, 2021.
Pitman, E. J. G. Sufficient statistics and intrinsic accuracy. Mathematical Proceedings of the Cambridge Philosophical Society, 32(4):567-579, 1936. doi: 10.1017/S0305004100019307.
Popov, V., Vovk, I., Gogoryan, V., Sadekova, T., and Kudinov, M. Grad-tts: A diffusion probabilistic model for text-to-speech. In International Conference on Machine Learning, pp. 8599-8608. PMLR, 2021.
Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., and Sutskever, I. Zero-shot text-to-image generation. In International Conference on Machine Learning, pp. 8821-8831. PMLR, 2021.
Rezende, D. J., Mohamed, S., and Wierstra, D. Stochastic backpropagation and approximate inference in deep generative models. In International conference on machine learning, pp. 1278-1286. PMLR, 2014.
Rissanen, S., Heinonen, M., and Solin, A. Generative modelling with inverse heat dissipation. arXiv preprint arXiv:2206.13397, 2022.
Saharia, C., Ho, J., Chan, W., Salimans, T., Fleet, D. J., and Norouzi, M. Image super-resolution via iterative refinement. arXiv preprint arXiv:2104.07636, 2021.
Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., and Ganguli, S. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning, pp. 2256-2265. PMLR, 2015.
Song, J., Meng, C., and Ermon, S. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020a.
Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S., and Poole, B. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020b.
Thanh-Tung, H. and Tran, T. Catastrophic forgetting and mode collapse in gans. In 2020 International Joint Conference on Neural Networks (IJCNN), pp. 1-10. IEEE, 2020.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Xiao, Z., Kreis, K., Kautz, J., and Vahdat, A. Vaebm: A symbiosis between variational autoencoders and energy-based models. arXiv preprint arXiv:2010.00654, 2020.
Yin, M. and Zhou, M. Semi-implicit variational inference. In International Conference on Machine Learning, pp. 5660-5669. PMLR, 2018.
Zhang, L., Goldstein, M., and Ranganath, R. Understanding failures in out-of-distribution detection with deep generative models. In International Conference on Machine Learning, pp. 12427-12436. PMLR, 2021.
Zhao, S., Ren, H., Yuan, A., Song, J., Goodman, N., and Ermon, S. Bias and generalization in deep generative models: An empirical study. Advances in Neural Information Processing Systems, 31, 2018.
Zhou, L., Du, Y., and Wu, J. 3d shape generation and completion through point-voxel diffusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5826-5835, 2021.
A Variational lower bound for the SS-DDPM model
$$
\begin{array}{l} \mathcal {L} ^ {\mathrm {s s}} (\theta) = \mathbb {E} _ {q ^ {\mathrm {s s}} (x _ {0: T})} \log \frac {p _ {\theta} ^ {\mathrm {s s}} (x _ {0 : T})}{q ^ {\mathrm {s s}} (x _ {1 : T} | x _ {0})} = \mathbb {E} _ {q ^ {\mathrm {s s}} (x _ {0: T})} \log \frac {p _ {\theta} ^ {\mathrm {s s}} (x _ {0} | x _ {1 : T}) p _ {\theta} ^ {\mathrm {s s}} (x _ {T}) \prod_ {t = 2} ^ {T} p _ {\theta} ^ {\mathrm {s s}} (x _ {t - 1} | x _ {t : T})}{\prod_ {t = 1} ^ {T} q ^ {\mathrm {s s}} (x _ {t} | x _ {0})} = (26) \\ = \mathbb {E} _ {q ^ {\mathrm {S S}} (x _ {0: T})} \left[ \log p _ {\theta} ^ {\mathrm {S S}} \left(x _ {0} \mid x _ {1: T}\right) + \sum_ {t = 2} ^ {T} \log \frac {p _ {\theta} ^ {\mathrm {S S}} \left(x _ {t - 1} \mid x _ {t : T}\right)}{q ^ {\mathrm {S S}} \left(x _ {t - 1} \mid x _ {0}\right)} + \underbrace {\log \frac {p _ {\theta} ^ {\mathrm {S S}} \left(x _ {T}\right)}{q ^ {\mathrm {S S}} \left(x _ {T} \mid x _ {0}\right)}} \right] = (27) \\ = \mathbb {E} _ {q ^ {\mathrm {S S}} \left(x _ {0: T}\right)} \left[ \log p _ {\theta} ^ {\mathrm {S S}} \left(x _ {0} \mid x _ {1: T}\right) - \sum_ {t = 2} ^ {T} D _ {K L} \left(q ^ {\mathrm {S S}} \left(x _ {t - 1} \mid x _ {0}\right) \| p _ {\theta} ^ {\mathrm {S S}} \left(x _ {t - 1} \mid x _ {t: T}\right)\right) \right] (28) \\ \end{array}
$$
B True reverse process for Markovian and Star-Shaped DDPM
If the forward process is Markovian, the corresponding true reverse process is Markovian too:
$$
q ^ {\mathrm {D D P M}} \left(x _ {t - 1} \mid x _ {t: T}\right) = \frac {q ^ {\mathrm {D D P M}} \left(x _ {t - 1 : T}\right)}{q ^ {\mathrm {D D P M}} \left(x _ {t : T}\right)} = \frac {q ^ {\mathrm {D D P M}} \left(x _ {t - 1}\right) q ^ {\mathrm {D D P M}} \left(x _ {t} \mid x _ {t - 1}\right) \prod_ {s = t + 1} ^ {T} q ^ {\underline {{\mathrm {D D P M}}} \left(\overline {{x _ {s} \mid x _ {s - 1}}}\right)}}{q ^ {\mathrm {D D P M}} \left(x _ {t}\right) \prod_ {s = t + 1} ^ {T} q ^ {\underline {{\mathrm {D D P M}}} \left(\overline {{x _ {s} \mid x _ {s - 1}}}\right)}} = q ^ {\mathrm {D D P M}} \left(x _ {t - 1} \mid x _ {t}\right) \tag {29}
$$
$$
q ^ {\mathrm {D D P M}} \left(x _ {0: T}\right) = q \left(x _ {0}\right) \prod_ {t = 1} ^ {T} q ^ {\mathrm {D D P M}} \left(x _ {t} \mid x _ {t - 1}\right) = q ^ {\mathrm {D D P M}} \left(x _ {T}\right) \prod_ {t = 1} ^ {T} q ^ {\mathrm {D D P M}} \left(x _ {t - 1} \mid x _ {t: T}\right) = q ^ {\mathrm {D D P M}} \left(x _ {T}\right) \prod_ {t = 1} ^ {T} q ^ {\mathrm {D D P M}} \left(x _ {t - 1} \mid x _ {t}\right) \tag {30}
$$
For star-shaped models, however, the reverse process has a general structure that cannot be reduced further:
$$
q ^ {\mathrm {S S}} \left(x _ {0: T}\right) = q \left(x _ {0}\right) \prod_ {t = 1} ^ {T} q ^ {\mathrm {S S}} \left(x _ {t} \mid x _ {0}\right) = q ^ {\mathrm {S S}} \left(x _ {T}\right) \prod_ {t = 1} ^ {T} q ^ {\mathrm {S S}} \left(x _ {t - 1} \mid x _ {t: T}\right) \tag {31}
$$
In this case, a Markovian reverse process can be a very poor approximation to the true reverse process. Choosing such an approximation adds an irreducible gap to the variational lower bound:
$$
\begin{array}{l} \mathcal {L} _ {\text {M a r k o v}} ^ {\mathrm {S S}} (\theta) = \mathbb {E} _ {q ^ {\mathrm {S S}} \left(x _ {0: T}\right)} \log \frac {p _ {\theta} \left(x _ {T}\right) \prod_ {t = 1} ^ {T} p _ {\theta} \left(x _ {t - 1} \mid x _ {t}\right)}{q ^ {\mathrm {S S}} \left(x _ {1: T} \mid x _ {0}\right)} = (32) \\ = \mathbb {E} _ {q ^ {\mathrm {S S}} (x _ {0: T})} \log \frac {p _ {\theta} (x _ {T}) \prod_ {t = 1} ^ {T} p _ {\theta} \left(x _ {t - 1} \mid x _ {t}\right) q \left(x _ {0}\right) \prod_ {t = 1} ^ {T} q ^ {\mathrm {S S}} \left(x _ {t - 1} \mid x _ {t}\right)}{q ^ {\mathrm {S S}} (x _ {T}) \prod_ {t = 1} ^ {T} q ^ {\mathrm {S S}} \left(x _ {t - 1} \mid x _ {t : T}\right) \prod_ {t = 1} ^ {T}} = (33) \\ = \mathbb {E} _ {q ^ {\mathrm {S S}} \left(x _ {0: T}\right)} \left[ \log q \left(x _ {0}\right) - \underbrace {D _ {K L} \left(q ^ {\mathrm {S S}} \left(x _ {T}\right) \| p _ {\theta} \left(x _ {T}\right)\right) - \sum_ {t = 1} ^ {T} D _ {K L} \left(q ^ {\mathrm {S S}} \left(x _ {t - 1} \mid x _ {t}\right) \| p _ {\theta} \left(x _ {t - 1} \mid x _ {t}\right)\right)} _ {\text {R e d u c i b l e}} - \right. (34) \\ - \underbrace {\sum_ {t = 1} ^ {T} D _ {K L} \left(q ^ {\mathrm {S S}} \left(x _ {t - 1} \mid x _ {t : T}\right) \| q ^ {\mathrm {S S}} \left(x _ {t - 1} \mid x _ {t}\right)\right)} _ {\text {I r r e d u c i b l e}} (35) \\ \end{array}
$$
Intuitively, there is little information shared between  $x_{t-1}$  and  $x_t$ , as they are conditionally independent given  $x_0$ . Therefore, we would expect the distribution  $q^{\mathrm{ss}}(x_{t-1}|x_t)$  to have a much higher entropy than the distribution  $q^{\mathrm{ss}}(x_{t-1}|x_{t:T})$ , making the irreducible gap (35) large. The dramatic effect of this gap is illustrated in Figure 3.
This gap can also be computed analytically for Gaussian DDPMs when the data is coming from a standard Gaussian distribution  $q(x_0) = \mathcal{N}(x_0; 0, 1)$ . According to equations (34-35), the best Markovian reverse process in this case is  $p_{\theta}(x_{t - 1}|x_t) = q^{\mathrm{ss}}(x_{t - 1}|x_t)$ . It results in the following value of the variational lower bound:
$$
\mathcal {L} _ {\text {M a r k o v}} ^ {\mathrm {s s}} = - \mathcal {H} [ q (x _ {0}) ] + \mathcal {H} [ q ^ {\mathrm {s s}} (x _ {0: T}) ] - \mathcal {H} [ q ^ {\mathrm {s s}} (x _ {T}) ] - \frac {1}{2} \sum_ {t = 1} ^ {T} \left[ 1 + \log \left(2 \pi \left(1 - \bar {\alpha} _ {t - 1} ^ {\mathrm {s s}} \bar {\alpha} _ {t} ^ {\mathrm {s s}}\right)\right) \right] \tag {36}
$$
If the reverse process is matched exactly (and has a general structure), the variational lower bound reduces to the negative entropy of the data distribution:
$$
\begin{array}{l} \mathcal {L} _ {*} ^ {\mathrm {S S}} = \mathbb {E} _ {q ^ {\mathrm {S S}} (x _ {0: T})} \log \frac {p _ {*} ^ {\mathrm {S S}} (x _ {0 : T})}{q ^ {\mathrm {S S}} (x _ {1 : T} | x _ {0})} = (37) \\ = \mathbb {E} _ {q ^ {\mathrm {S S}} \left(x _ {0: T}\right)} \log \frac {q ^ {\mathrm {S S}} \left(x _ {0 : T}\right)}{q ^ {\mathrm {S S}} \left(x _ {1 : T} \mid x _ {0}\right)} = (38) \\ = \mathbb {E} _ {q \left(x _ {0}\right)} \log q \left(x _ {0}\right) = - \frac {1}{2} \log (2 \pi) - \frac {1}{2} (39) \\ \end{array}
$$
As shown in Figure 7, the Markovian approximation adds a substantial irreducible gap to the variational lower bound.
Figure 7: Variational lower bound for a Gaussian Star-Shaped DDPM model computed with the true reverse process and with the best Markovian approximation.
We start by establishing the following lemma. It would allow us to change the variables in the condition of a conditional distribution. Essentially, we would like to show that if a variable  $y$  is only dependent on  $x$  through some function  $h(x)$ , we can write the distribution  $p(y|x)$  as  $p(y|z)|_{z=h(x)}$ .