Star-Shaped Denoising Diffusion Probabilistic Models
Authors: Andrey Okhotin
Dmitry Molchanov
Arkhipkin Sergeevich Vladimir
Grigory Bartosh
Viktor Ohanesian
Aibek Alanov
Dmitry P. Vetrov
Venue: NeurIPS 2023 poster (2023)
Keywords: Generative models
Diffusion
Exponential Family
TL;DR: We provide a simple recipe for designing diffusion-like models with distributions from the exponential family.

Abstract: Denoising Diffusion Probabilistic Models (DDPMs) provide the foundation for the recent breakthroughs in generative modeling.
Their Markovian structure makes it difficult to define DDPMs with distributions other than Gaussian or discrete.
In this paper, we introduce Star-Shaped DDPM (SS-DDPM).
Its *star-shaped diffusion process* allows us to bypass the need to define the transition probabilities or compute posteriors.
We establish duality between star-shaped and specific Markovian diffusions for the exponential family of distributions and derive efficient algorithms for training and sampling from SS-DDPMs.
In the case of Gaussian distributions, SS-DDPM is equivalent to DDPM.
However, SS-DDPMs provide a simple recipe for designing diffusion models with distributions such as Beta, von Mises–Fisher, Dirichlet, Wishart and others, which can be especially useful when data lies on a constrained manifold.
We evaluate the model in different settings and find it competitive even on image data, where Beta SS-DDPM achieves results comparable to a Gaussian DDPM.
Our implementation is available at https://github.com/andrey-okhotin/star-shaped

Star-Shaped Denoising Diffusion Probabilistic Models
Andrey Okhotin'
Dmitry Molchanov' BAYESG Budva, Montenegro amolch1 1 @gmail......mmmm com
HSE University, MSU University
Moscow, Russia
Vladimir Arkhipkin
Grigory Bartosh
Viktor Ohanesian Independent Researcher
Sber AI
AMLab, Informatics Institute University of Amsterdam Amsterdam, Netherlands g bartosh@uva. nl
Moscow, Russia
v.v. www.ooesssnnmmmllll...mmmmmm com
ftttiiikin.v@@@gaal.ccom
Aibek Alanov AIRI, HSE University
Dmitry Vetrov Constructor University
Moscow, Russia alanov. aib.kgmaiibeogg
Bremen, Germany
dvetrov@constructor http://vwooosttuuooooouuvuttttts university
Abstract
Denoising Diffusion Probabilistic Models (DDPMs) provide the foundation for the recent breakthroughs in generative modeling. Their Markovian structure makes it difficult define DDPMs with distributions other than Gaussian or discrete. In this paper, we introduce Star-Shaped DDPM (SS-DDPM). Its star-shaped diffusion process allows us bypass the need define the transition probabilities or compute posteriors. We establish duality between star-shaped and specific Markovian diffusions for the exponential family of distributions and derive efficient algorithms for training and sampling from SS-DDPMs. In the case of Gaussian distributions, SS-DDPM is equivalent DDPM. However, SS-DDPMs provide a simple recipe for designing diffusion models with distributions such as Beta, von Mises-Fisher, Dirichlet, Wishart and others, which can be especially useful when data lies on constrained manifold. We evaluate the model in different settings and find it competitive even on image data, where Beta SS-DDPM achieves results comparable to a Gaussian DDPM. Our implementation is available at wwt.coom//cc//o
1 Introduction
Deep generative models have shown outstanding sample quality in a wide variety modalities. Gen- erative Adversarial Networks (GANs) (Goodfellow et al. 2014 Karras et al.) 2021), autoregressive models (Ramesh et al. 2021), Variational Autoencoders Kingma & Welling 2013 Rezende et al. 2014), Normalizing Flows (Grathwohl et al. 2018 Chen et al. 2019) and energy- based models (Xiao et al. 2020) show impressive abilities to synthesize objects. However, GANs are not robust to the choice of architecture and optimization method (Arjovsky al.) 2017 Gulrajani al.) 2017 Karras et al., 2019: Brock et al.] 2018), and they often fail to cover modes in data distribution ZZha
al. 2018 Thanh-TTum Tung & Tran 2020). Likelihood-basse models avoid mode collapse but may overestimate the probability low-density regions (Zhang et al. 2021).
"Equal contribution
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Zg
FT
20
2:-1
S1
Rr-1
20 (a (as)
Figure 1: Markovian forward processes of DDPM (left) and the star-shaped forward process of SS-DDPM (right).
Recently, diffusion probabilistic models (Sohl-Dickstein et al. 2015 Ho et al. 2020) have received lot of attention. These models generate samples using trained Markov process that starts with white noise and iteratively removes noise from the sample. Recent works have shown that diffusion models can generate samples comparable in quality or even better than GANs (Song et al., 2020b; Dhariwal & Nichol 2021), while they do not suffer from mode collapse by design, and also they have autoregressive models (Kingma al. (00TT). Moreover, diffusion
log-likelihood comparable
models show these results in various modalities such as images (Saharia et al. (021), sound (Popov
et al.
202T
Liu et al. 2022) and shapes (Luo & Hu 2021 Zhou al. (202T).
The main principle of diffusion models to destroy information during the forward process and then restore during the reverse process. In conventional diffusion models like denoising diffusion probabilistic models (DDPM) destruction of information occurs through the injection of Gaussian noise, which is reasonable for some types of data, such images. However, for data distributed on manifolds, bounded volumes, or with other features, the injection of Gaussian noise can be unnatural, breaking the data structure. Unfortunately. is not clear how replace the noise distribution within traditional diffusion models. The problem is that we have to maintain a connection between the distributions defining the Markov noising process that gradually destroys information and its marginal distributions. While some papers explore other distributions, such as delta functions (Bansal et al. 2022) or Gamma distribution (Nachmani et al. 2021), they provide ad hoc solutions for special cases that are not easily generalized.
In this paper, we present Star-Shaped Denoising Diffusion Probabilistic Models (SS-DDPM), new approach that generalizes Gaussian DDPM to an exponential family of noise distributions. In SS-DDPM, one only needs to define marginal distributions at each diffusion step (see Figure We provide a derivation of SS-DDPM, design efficient sampling and training algorithmss, and show its cquivalence DDPM (Ho al. 2020) in the case of Gaussian noise. Then, we outline number of practical considerations that aid in training and applying SS-DDPMs. In Section 5 we demonstrate the ability of SS-DDPM to work with distributions like von Mises- Dirichlet and Wishart. Finally, we evaluate SS-DDPM on image and text generation. Categorical SS-DDPM matches the performance of Multinomial Text Diffusion (Hoogeboom et (02T)) on the text8 dataset, while our Beta diffusion model achieves results, comparable to a Gaussian DDPM on CIFAR-10.
2 Theory
2.1 DDPMs
We start with a brief introduction of DDPMs. The Gaussian DDPM (Ho et al., 2020 is defined as a forward (diffusion) process q MDDM (Zor) and corresponding reverse (denoising) process The forward process is defined as Markov chain with Gaussian conditionals:
pg
T
g (
MDPM (xt (r
1=1
DDPM (0// =N (215 (I:vI- B,0(-1,
where g(vo) is the data distribution. Parameters Br are typically chosen in advance and fixed, defining the noise schedule of the diffusion process. The noise schedule chosen such a way that the final ar no longer depends on zo and follows a standard Gaussian distribution DDPM (xr) = - O,T)

---

x1
82
F1-1
FT
(eilee)))
20
1:-1
pol
(a) Denoising Diffusion Probabilistic Models
(b) Star Sar-Shhped Denoising Diffusion Probabilistic Models
Figure 2: Model structure of DDPM and SS-DDPM.
The reverse process p pRers (ZDTT follows similar structure and constitutes the generative part of the model:
pDPM ( )= = Dn (IT)
DDPM (x2-1114),
Pg
P8
P8
DDPM
(xt-1
N Ho t). Eo(re.t))
The forward process (DDPM((001) of DDPM typically fixed, and all the parameters the model are contained in the generative part of the model pe pDDPM (za:r)) These parameters are tuned to maximize the variational lower bound (VLB) on the likelihood of the training data:
CDDPM (0) DDPM
log po DDDPM (xolor)
DDEL (q (xt xo) P0 DDPM
(x+-1(20))
(5)
CDDPM (0) + max
(6)
One of the main challenges in defining DDPMs is the computation of the posterior ( [Zt, zo). Specifically, the transition probabilities DDPM (x,(2;111 have to be defined in such a way that this posterior is tractable. Specific DDPM-like models are available for Gaussian (Ho et al., 2020), Categorical (Hoogeboom et al. 2021) and Gamma (Kawar et al. 2022) distributions. Defining such models remains challenging in more general cases.
2.2 Star-Shaped DDPMs
As previously discussed, extending the DDPMs to other distributions poses significant challenges. In light of these difficulties, propose construct model that only relies marginal distributions q (2)) in its definition and the derivation of the loss function.
We define shaped diffusion non-Markovian forward process q (Zo:T) that has the following structure:
T
1et (To:) = (100) II d (x)))))
where ((20) the data distribution. note that in contrast to DDPM all noisy variables I; are conditionally independent given instead of constituting Markov chain. This structure of the forward process allows us to utilize other noise distributions, which we discuss in more detail later.
2.3 Defining the reverse model
In DDPMs the true reverse model pDPM (xooT) has a Markovian structure (Ho et al. 2020 allowing for efficient sequential generation algorithm:
(DPM( DDPM (TT)
DDPM (1)))
(8)
3

---

For the star-shaped diffusion, however, the Markovian assumption breaks:
(#O.T) 90 - (IT)
II
t=1
Consequently, we now need to approximate the true reverse process by a parametric model which is conditioned on the whole tail Tr.T.
T
88 Pg (IOTT) = Po
88 (xT)
II t=1
Pe
38
(x
(10)
It is crucial to use the whole tail T1 rather than just one variable 2, when predicting æ star-shaped model, As we show in Appendix B we try to approximate the true reverse process with Markov model, we introduce substantial irreducible gap into the variational lower bound. Such sampling procedure fails to generate re- alistic samples, as can be seen in Figure 3
x
Monoy
909
875
750
625
500
375
250
125
Figure: 3: Markov reverse process fails to recover realistic images from star-shaped diffusion, while a general reverse process produces realistic images. The top row is equivalent to DDIM of ar A similar effect was also observed by Bansal let al. (2022).
Intuitively, in DDPMs the information about To that is contained in Tt+1 nested into the information about 2o that is contained in I₁. That is why knowing 24 allows us to discard r++1. star-shaped diffusion, however, all variables contain independent pieces of information about 2o and should all be taken into account when making predictions.
We can write down the variational lower bound as follows:
_ (0)
Eyss logpo(ro/
Σ DKL. (1 (at-1
p0
1 (x. (SETT)
(11)
With this VLB, we only need the marginal distributions q( to define and train the model, which allows us to use wider variety of noising distributions. Since conditioning the predictive model po (= ( on the whole tail Fx. is typically impractical, we propose more efficient way to implement the reverse process next.
2.4 Efficient tail conditioning
Instead of using the full tail Tr.7, we would like to define some statistic G, (((Tr) that would extract all information about 2g from the tail Ter-- Formally speaking, we call Gj sufficient tail statistic if the following equality holds:
(II-I 9
(12)
One way to define G2 is to concatenate all the variables ZeT into single vector. This, however, is impractical, as its dimension would grow with the size of the tail t
The Homman-Koopmnn-Dmmae
(Pitman 1936) theorem (PKD) states that exponential families admit sufficient statistic with constant dimensionality. It also states that other distribution admits one: such statistic were to exist, the distribution has to be a member of the exponential family. Inspired the exponential family of distributions. In the case of star-shaped diffusion,
by the PKD, we cannot apply the PKD directly, as it was formulated for i.i.d, samples and our samples are not identically distributed. However, can still define sufficient tail statistic G, for a specific subset of the exponential family, which we call an exponential family with linear parameterization:
Theorem 1. Assume the forward process of star-shaped model takes the following form:
q nd (2, hx(2,) exp (2.) (T. TT(x)) - (x(x))) (x = Azf(zo) + by-
(13)
(14)

---

Let G2 be tail statistic, defined as follows:
G gc(xeT) = TTx))
(15)
Then, Ge is sufficient tail statistic:
88 (rt-1 (x.T) = (1
(16)
Here definition (13 is the standard definition of the exponential family, where he(x)) is the base measure, 7/, (200 is the vector of natural parameters with corresponding sufficient statistics T(xe). and (2o) is the tog-partition function. The key assumption added is the linear parameterization of the natural parameters (4). We provide the proof in Appendix C When A, is scalar, we denote it as
a, instead.
For the most part, the premise of Theorem D restricts the parameterization of the distributions rather than the family of the distributions involved. As we discuss in Appendix we found it easy to come up with linear parameterization for wide range of distributions in the exponential family. For example, we can obtain a linear parameterization for the Beta distribution q(2, (zo) = (ey az, B2) using To as the mode of the distribution and introducing
concentration parameter Mi:
a
(17)
8 = 1 + (1
(18)
In this case, ne(2)) = Mezo, T(x,) = log 1-x, and we can use equation 15
to define the sufficient tail statistic GN. We provide more examples in Appendix F We also provide an implementation-reann reference sheet for a wide range of distributions in the exponential family in Table
We suspect that, just like in PKD, this trick is only possible for a subset the exponential family. In the general case, the dimensionality of the sufficient tail statistic G, would have to grow with the size of the tail Tę.. It is still possible to apply SS-DDPM in this case, however, crafting the (now only approximately) sufficient statistic G, would require more careful consideration and we leave it for future work.
2.5 Final model definition
To maximize the VLB (II. each step of the reverse process should approximate the true reverse distribution:
(zt-1 ( x 200 (It (laer) / (x= (xo )dro-
Po 23
(19)
Similarly to DDPM (Ho et al. 2020), we choose to approximate 2 (x.T) with a delta function centered at the prediction of some model u (x(:))))) This results the following definition of the reverse process of SS-DDPM:
Pe (xt- (xe-1 = th (x (00
(20)
The distribution p (æo (x1:) ) can be fixed to some small- variance distribution Pø (zg (2)) centered the final prediction ž. z4 (G1 (II:T), 1), similar to the dequantization term, commonly used in DDPM. If this distribution has trainable parameters, the corresponding term can be removed from the training objective. This dequantization distribution would then only be used for likelikel estimation and, optionally, for sampling.
Together with the forward process and the VLB objective 11] this concludes the general definition of the SS-DDPM model. The model structure is illustrated in Figure 2 The corresponding training and sampling algorithms are provided in Algorithms T and 2
The resulting model is similar to DDPM in spirit. We follow the same principles when designing the forward process: starting from vorvanriance distribution, centered at zp at t 1, we gradually increase the entropy of the distribution 98 (x,[2)) until there is no information shared between To and 24 at T'
We provide concrete definitions for Beta, Gamma, Dirichlet, von Mises, von Mises-Fisher, Wishart, Gaussian and Categorical distributions in Appendix F
5

---

Algorithm SS-DDPM training
Algorithm 2 SS-DDPM sampling
repeat
90h (IT) GT = T
((3))
~ Cniform(1,......
for t T to 2 do
Ft.7 of (0)
To - x (Gr, 1)
G. G. a- (z.(2.)
3t-1 1 10 20 =00
Move along
G = G. + 1T
until Convergence
end for
To ~ Pe (ro
2.6 Duality between star-shaped and Markovian diffusion
While the variables 21. follow a star-shaped diffusion process, the corresponding tail statistics G1:r form a Markov chain:
T G. Σ (x. G441 (xx)
(21)
since v, is conditionally independent from G++2.T given G++1 (see Appendix B Bfor details). Moreover, we can rewrite the probabilistic model in terms of Gg and see that variables (Tog G1.1) form (not necessarily Gaussian) DDPM.
In the case of Gaussian distributions, this duality makes SS-DDPM and DDPM equivalent. This equivalence can be shown explicitly:
Theorem 2. Let aa define the noising schedule for a DDPM model 2 via Br = (a.-1
DDPM
/ a Let ( be Gaussian SS-DDPM forward process with the following noising schedule and sufficient tail statistic:
q (2te = N (Zti
ai
To,
(22)
DDPM
T
0xg Js
Ge(Fer)
where
(23)
V Va
PDPM
s=t
07
0i @DDPM
01++1
-DPM
(24)
1
DDPM
0,+1
DDDM
Then the tail statistic G. follows Gaussian DDPM noising process q DDMM (Io.T) (abrr)aarrre Grur defined by the schedule o o equivalent.
DDPM Moreover, the corresponding reverse processes and VLB objectives are also
We show this equivalence in Appendix D We make use of this connection when choosing the noising schedule for other distributions.
This equivalence means that SS-DDPM is a direct generalization of Gaussian DDPM. While admitting the Gaussian case, SS-DDPM can also be used implicitly define a non-Gaussian DDPM in the space of sufficient tail statistics for a wide range of distributions.
3 Practical considerations
While the model is properly defined, there are several practical considerations that are important for the efficiency of star-shaped diffusion.
Choosing the right schedule It is important to choose the right noising schedule for a SS-DDPM model. significantly depends on the number of diffusion steps and behaves differently given different noising schedules, typical to DDPMs. This is illustrated in Figure where we show the noising schedules for Gaussian SS-DDPMs that are equivalent to DDPMs with the same cosine schedule.

---

T-250
47.
07:
7- 100
T- 1000
3/00
X
10
x xesass
Gretess
08
125 250
375
500
625
750
875
999
Figure 5: Top: samples 2, from a Gaussian DDPM
02
forward process with cosine noise schedule. Bot- tom: samples G2 from a Beta SS-DDPM forward process with noise schedule obtained by match- ing the mutual information. Middle: correspond- ing samples H from that Beta SS-DDPM forward process. The tail statistics have the same level of noise zi wpDPM while the samples BetaS dif- fused much faster.
00
1.0
1/T
Figure 4: The noising schedule ot for Gaus- sian star-shaped diffusion, defined for differ- ent numbers of steps T using eq. (24). All the corresponding equivalent DDPMs have the same cosine schedule ap DDPM
Since the variables G2 follow a DDPM-like process, we would like to somehow reuse those DDPM noising schedules that are already known to work well. For Gaussian distributions, can transform DDPM noising schedule into the corresponding SS-DDPM noising schedule analytically by equating x [aro; G.) x x.DDX). In general case, we look for schedules that have approximately the same
level of mutual information [(rg;; G₂) as the corresponding mutual information / ( x:DPP)) for DDPM model for all timesteps We estimate the mutual information using Kraskov (Kraskov et al. 2004) and DSIVI (Molchanov et al. (2019) estimators and build look-up table to match the noising schedules. This procedure is described in more detail in Appendix G The resulting schedule for the Beta SS-DDPM is illustrated in Figure [ Note how with the right schedule appropriately normalized tail statistics Ge look and function similarly the samples 23 from the corresponding Gaussian DDPM. We further discuss this in Appendix
Implementing the sampler During sampling. we can grow the tail statistic G, without any over- head, as described in Algorithm 2 However, during training, we need to sample the tail statistic for each object to estimate the loss function. For this we need sample the full tail Fu. from the forward process _ (Ier (ao), and then compute the tail statistic GG. In practice, this does not add noticeable overhead and can be computed in parallel to the training process if needed.
Reducing the number of steps We can sample from DDPMs more efficiently by skipping some timestamps. This wouldn't work for SS-DDPM, because changing the number of steps would require changing the noising schedule and, consequently, retraining the model.
However, can still use similar trick reduce the number of function evaluations. Instead of skipping some timestamps It, +1:t2-1 we can draw them from the forward process using the current prediction (Gt, 12), and then these samples to obtain the tail statistic Ge,- For Gaussian SS-DDPM this is equivalent to skipping these timestamps in the corresponding DDPM. In general case, amounts to approximating the reverse process with different reverse process:
L2
38 (Di GG,)) = II t=t,
d0
= II t=
(25)
We observe similar dependence on the number of function evaluations for SS-DDPMs and DDPMs.
defined Theorem [ the tail statistics can have vastly different scales for different timestamps. The values of coefficients a can range from thousandths when approaches 7 to thousands when approaches zero. To make the tail statistics suitable for use in neural networks, proper normalization is crucial. In most cases, we collect the time-dependent means and variances of the tail statistics across the training dataset and normalize the tail statistics to zero mean and unit variance. We further discuss this issue in Appendix H
Time-dependent tail normalization
Architectural choices To make training the model easier, we make some minor adjustments the neural network architecture and the loss function.
7

---

Our neural networks Gr t) take the tail statistic G2 as an input and are expected to produce an estimate of z as an output. In SS-DDPM the data 2g might lie on some manifold, like the unit sphere or the space of positive definite matrices. Therefore, we need to map the neural network output to that manifold. We do that on a case-by-case basis, as described Appendices [
Different terms of the VLB can have drastically different scales. For this reason, it is common practice to train DDPMs with a modified loss function like Liimpl rather than the VLB improve the stability of training (Ho et al. 2020). Similarly, we can optimize a reweighted variational lower bound when training SS-DDPMs.
4 Related works
Our work builds upon Denoising Diffusion Probabilistic Models (Ho et al. 2020). Interest in diffusion
models has increased recently due to their impressive results in image (Ho et al. 2020 Song et al.)
20206 Dhariwal & Nichol 2021) and audio (Popov et al. 2021 Liu et al. 2022) generation.
SS-DDPM is most closely related to DDPM. Like DDPM, we only rely on variational inference when defining and working with our model. SS-DDPM can be seen as a direct generalization of DDPM and essentially is a recipe for defining DDPMs with non-Gaussian distributions. The underlying non-Gaussian DDPM is constructed implicitly and can be seen as dual to the star-shaped formulation that we use throughout the paper.
Other ways to construct non- Gaussian DDPMs include Binomial diffusson (Sohl-Dickstein et al. 2015). Multinomial diffusion (Hoogeboom et al. 202I) and Gamma diffusion (Nachmani et al.
2021 Each of these works presents a separate derivation of the resulting objective, and extending them to other distributions is not straightforward. On the other hand, SS-DDPM provides a single recipe for wide range of distributions.
DDPMs have several important extensions. Song et al. (2020a) provide a family of non-Markovian diffusions that all result in the same training objective as DDPM. One of them, denoted Denoising Diffusion Implicit Model (DDIM), results in an efficient deterministic sampling algorithm that requires much lower number of function evaluations than conventional stochastic sampling. Their derivations also admit a star-shaped forward process (at of = 1- a,-1), however, the model is not studied in the star-shaped regime. Their reverse process remains Markovian, which we show to not be sufficient to invert star-shaped forward process. Denoising Diffusion Restoration Models (Kawar al., 2022) provide way to solve general linear inverse problems using a trained DDPM. They can be used for image restoration, inpainting, colorization and other conditional generation tasks. Both DDIMs and DDRMs rely on the explicit form of the underlying DDPM model and are derived for Gaussian diffusion. Extending these models to SS-DDPMs is a promising direction for future work.
Song et al. 20206) established the connection between DDPMs and models based on score matching. This connection gives rise to continuous-tin time variants of the models, deterministic solutions and more precise density estimation using ODEs. We suspect that a similar connection might hold for SS-DDPMs as well, and it can be investigated further in future works.
Other works that present diffusion-like like models with other types of noise applied manifold data, generally stem from score matching rather than variational inference. Flow Matching (Lipman et al.) 2022) is an alternative probabilistic framework that works with any differentiable degradation process. De Bortoli et al. (2022) and Huang et al. (002)) extended score matching to Riemannian manifolds, and Chen & Lipman (2023) proposed Riemannian Flow Matching. Bansal al. (2022) proposed Cold Diffusionn a non- non- probabilisiic approach to reversing general degradation processes.
To the best of our knowledge, for the first time, an approach for constructing diffusion without consccutive process was proposed by Rissanen et al. (022) (IHDM) and further expanded on by Daras et al. (2022) and Hoogeboom & Salimans (2022). IHDM uses similar star-shaped structure that results in a similar variational lower bound. Adding a deterministic process based on the heat equation allows the authors to keep the reverse process Markovian without having to introduce the tail statistics. As IHDM heavily relies on blurring rather than adding noise, the resulting diffusion dynamics become very different. Conceptually our work is much closer to DDPM than IHDM.
8

---

5 Experiments
We evaluate SS-DDPM with different families of noising distributions. The experiment setup, training details and hyperparameters are listed in Appendices L
Table 1: KI. divergence between the real data distribution and the model distribu- tion D & (g(2n) pg (zo)) for Gaussian DDPM and SS-DDPM.
Synthetic data We consider two examples of star- shaped diffusion processes with Dirichlet and Wishart noise to generate data from the probabilistic simplex and from the manifold of p.d. matrices respectively. We compare them to DDPM, where the predictive network xo(x;; is parameterized constraints. As seen in Table I using the appropriate dis- tribution results in a better approximation. The data and modeled distributions are illustrated in T This shows the ability of SS-DDPM to work with different distributions and generate data from exotic domains.
Dirichlet
Wishart
DDPM SS-DDPM
0.200
0.096
0.011
0.037
always satisfy the manifold
Geodesic data We apply SS-DDPM to geodesic dataset of fires on the Earth's 's surface (EOSDIS) 2020) using a three-dimensional von Mises-Fisher distribution. The resulting samples and the source data are illustrated in T We find that SS-DDPM is not too sensitive to the distribution family and can data in different domains.
Discrete data Categorical SS- SS-DDPM is similar to Multi- nomial Text Diffusion (MTD, (Hoogeboom al., 202I)). However, unlike in the Gaussian case, these models are strictly equivalent. We follow a similar setup to MTD and apply Categorical SS-DDPM to unconditional text generation on the text8 dataset (Mahoney (01T). As shown in Table 2 SS-DDPM achieves similar results to MTD, allowing to use different distributions a unified manner. While D3PM (Austin et al. 2021) provides some improvements, we follow simpler setup from MTD, We expect the improvements from D3PM to directly apply to Categorical SS-DDPM.
Table 2: Comparison of Categorical SS-
DDPM and Multinomial Text Diffusion on text8. NLL is estimated via ELBO.
Model
NLL (bits/char)
MTD
1.72
SS-DDPM
S 1.69
Image data Finally, evaluate SS-DDPM
on CIFAR-10. Since the training data is con- strained to [0, 1] segment, we use Beta distri- butions. We evaluate our model with various numbers of generation steps, described in equation 25 and report the resulting Fréchet
16
DDPM
14
DDIM
12
Beta SS-DDPM
10
Inception Distance (FID, (Heusel et al. (017)) in Figure 6 Beta SS-DDPM achieves compara- ble quality with the Improved DDPM (Nichol & Dhariwal 202T) and is slightly better on lower numbers steps. As expected, DDIM performs better when the number of difffusion steps is low, but both SS-DDPM and DDPM outper- form DDIM on longer runs. The best FID score achieved by Beta SS-DDPM 3. Although the FID curves for DDPM and DDIM do not achieve this score in Figure 6 Ho al. (2020) reported an FID score of 3.17 for 1000 DDPM steps, meaning that SS-DDPM performs simi- larly to DDPM in this setting.
2-
10a sampling steps
10a
Figure 6: Quality of images, generated using Beta SS-DDPM, DDPM and DDIM with different num- bers of sampling steps. Models are trained and evaluated on CIFAR- 10. DDPM and DDIM results were taken from Nichol & Dhariwal (20T

---

Table 3: Experiments results. The first row is real data and the second is generated samples. For the von and Dirichlet models, we show two-dimensional histograms of samples. For the Wishart model, draw ellipses, corresponding the p.d. matrices zy and zg. The darker the pixel, the more ellipses pass through that pixel.
von Mises-Fisher
Dirichlet
Wishart
Real
Generated
Conclusion
We propose an alternative view on diffusion-like probabilistic models. We reveal the duality between star-shaped and Markov diffusion processes that allows to go beyond Gaussian noise by switching to a star-shaped formulation. It allows us to define diffusion-like models with arbitrary noising distributions and establish diffusion processes on specific manifolds. We propose an efficient way to construct a reverse process for such models in the case when the noising process lies in general subset of the exponential family and show that star-shaped diffusion models can be trained on variety of domains with different noising distributions. On image data, star-shaped diffusion with Beta distributed noise attains comparable performance Gaussian DDPM, challenging the optimality of Gaussian noise in this setting. The star-shaped formulation opens new applications of diffusion-like probabilistic models, especially data from exotic domains where domain-specine non-Gaussian diffusion is more appropriate.
Acknowledgements
We are grateful to Sergey Kholkin for additional experiments with von Mises-Fishee SS-DDPM geodesic data and to Tingir Badmaev for helpful recommendations in experiments image data. We'd also like to thank Viacheslav Meshchaninov for sharing his knowledge of diffusion models for text data. This research was supported in part through computational resources of HPC facilities at HSE University. The results on image data (Image data in Section 5 Section L were obtained by Andrey Okhotin and Aibek Alanov with the support of the grant for research centers in the field of Al provided by the Analytical Center for the Government of the Russian Federation (ACRF) in accordance with the agreement the provision of subsidies (identifier of the agreement 1100000773032155000000 and the agreement with HSE University No. 70-2021-00139
References
Arjovsky, M., Chintala, S., and Bottou, arXiv:1701.07875. 30:4, 2017.
Wasserstein gan. arxiv 2017. arXiv preprint
10

---

Austin, J., Johnson, D. D., Ho, J., Tarlow, D., and van den Berg, R. Structured denoising diffusion models in discrete spaces. Advances in Neural Information Processing Systems, 34:17981- 17993, 2021.
Bansal, A., Borgnia, E., Chu, H.-M., L. Kazemi, H., Huang, F., Goldblum, M., Geiping, and Goldstein, Т. Cold diffusion: Inverting arbitrary image transforms without noise. arXiv preprint 220.::22289999., 2022.
Brock, A., Donahue, J., and Simonyan, K. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv: 1809. 1096, 2018.
Chen, R. T. and Lipman, Y. Riemannian flow matching on general geometries. arXiv preprint arXiv:2302.03660, 2023.
Chen, R. T., Behrmann, J., Duvenaud, D. K., and Jacobsen, J.-H. Residual flows for invertible generative modeling. Advances in Neural Information Processing Systems, 32, 2019.
Daras, G., Delbracio, M., Talebi, H., Dimakis, A. G., and Milanfar, P. Soft diffusion: Score matching for general corruptions. arXiv preprint ar 2TXvv:220.005442, 2022.
De Bortoli, V., Mathieu, Hutchinson, M., Thornton, J., Teh, Y. W., and Doucet, A. Riemannian score-based generative modeling. arXiv preprint arXiv:2202.02760 2022.
Dhariwal, P. and Nichol, A. Diffusion models beat gans image synthesis. arXiv preprint arXiv:2105.05233, 2021.
EOSDIS. Land, atmosphere near real-time capability for (lance) system operated by nasa's 's earth science data and information system (esdis). rall-ine//iimm/aaiiv----eee-- 2020.
Goodfellow, I., Pouget-Abadie, Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
Grathwohl, W., Chen, R. T., Bettencourt, J., Sutskever, I., and Duvenaud, D. Ffjord: Free-form continuous dynamics for scalable reversible generative models. arXiv preprint 18000:7810.01167, 2018.
Gulrajani, I,, Ahmed, F., Arjovsky, M., Dumoulin, V., and Courville, A. C. Improved training of wasserstein gans. Advances in neural information processing systems, 30, 2017.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. corr abs/1512.03385 (2015), 2015.
Heusel, M.. Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. Gans trained by two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Ho, J., Jain, A., and Abbeel, Denoising diffusion probabilistic models. arXiv preprint arXiv:2006. 1239, 2020.
Hoogeboom, E. and Salimans, T. Blurring diffusion models. arXiv preprint arXlv:2209.05557, 2022.
Hoogeboom, E., Nielsen, D., Jaini, P., Forré, P., and Welling, M. Argmax flows and multinomial diffusion: Learning categorical distributions. Advances in Neural Information Processing Systems, 14:14454-22465, 2021.
Huang, C.-W., Aghajohai,,, M., Bose, J., Panangaden, P., and Courville, A. C. Riemannian diffusion models. Advances in Neural Information Processing Systems, 22-21, 2022.
Karras, T., Laine, S., and Aila, T. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEFF/CVE conference on computer vision and pattern recognition, pp. 4401-4410, 2019.

---

Karras, T., Aittala, M., Laine. S., Härkönen, E., Hellsten, J., Lehtinen, J., and Aila, T. Alias-free generative adversarial networks. Advances in Neural Information Processing Systems, 34, 2021.
Kawar, B., Elad, M., Ermon, S., and Song. J. Denoising diffusion restoration models. In Advances in
Neural Information Processing Systems, 2022.
Kingma, D. and Ba, Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015.
Kingma, D. P. and Welling, M. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.
Kingma, D. P., Salimans, T., Poole, B., and Ho. J. Variational diffusion models. arXiv preprint 1xXiv:207.000630. 2, 2021.
Kraskov, A., Stögbauer, H., and Grassberger, Estimating mutual information. Physical review E. 69(6):066138, 2004.
Lipman, Y., Chen, R. T., Ben-Hamu, H., Nickel, M., and Le, M. Flow matching for generative modeling. arXiv preprint arXiv:2210.0274,, 2022.
Liu, S., Su, D., and Yu, D. Diffgan-tts: High-fidelity and efficient text-to-speech with denoising diffusion gans, arXiv preprint 11X1 2022.
Luo, S. and Hu, W. Diffusion probabilistic models for 3d point cloud generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2837-2845, 2021.
Mahoney, M. Large text compression benchmark, 2011. URL httt:://ww.mmattmahorze .nt// dc/text... html
Molchanov, D., Kharitonov, V., Sobolev, A., and Vetrov, D. Doubly semi-implicit variational inference. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 2593-2602, PMLR, 2019.
Nachmani, E., Roman, S., and Wolf, L.. Denoising diffusion gamma models. arXiv preprint arXiv:2110.05948, 2021.
Nichol, A. and Dhariwal, Improved denoising diffusion probabilistic models. arXiv preprint arXiv:2102.00967. 2021.
Pitman, E. J. G. Sufficient statistics and intrinsic accuracy. Mathematical Proceedings of the Cambridge Philosophical Society, 32(4):567-579 1936. doi: 10.10177500050041000099
Popov, V., Vovk, I., Gogoryan, V., Sadekova, T., and Kudinov, M. Grad-tts: A diffusion probabilistic model for text-to-speech. In International Conference on Machine Learning, pp. 8599-8608. PMLR, 2021.
Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C.,, Radford, A., Chen, M., and Sutskever, I. Zero- shot text-to-image generation. International Conference on Machine Learning, pp. 8821-8831, PMLR, 2021.
Rezende, J. Mohamed, S., and Wierstra, D. Stochastie backpropagation and approximate inference in deep generative models. In International conference on machine learning, pp. 1278-1286. PMLR, 2014.
Rissanen, S., Heinonen, M., and Solin, A. Generative modelling with inverse heat dissipation. arXiv preprint arXiv:2206.13397, 2022.
Saharia, C., Ho, Chan, W., Salimans, T., Fleet, J., and Norouzi, M. Image super-resolution via iterative refinement, arXiv preprint arXiv:2 2021.
Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., and Ganguli, S. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning, pp- 2256-2265. PMLR. 2015.
12

---

Song. J. Meng. C., and Ermon, S. Denoising diffusion implicit models. arXiv preprint 2XXi::0100.22022 2020a.
Song, Y. Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S., and Poole, B. Score-based generative modeling through stochastic differential equations. arXiv preprint 2rX1v:2011.13456, 2020b.
Thanh-Tung, H. and Tran, T. Catastrophic forgetting and mode collapse in gans. In 2020 International Joint Conference Neural Networks (IJCNN), pp. 1-10. IEEE, 2020.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Xiao, Z., Kreis, K., Kautz, J., and Vahdat, A. Vaebm: symbiosis hetween variational autoencoders and energy-hased models. arXiv preprint arXiv:2010.00654 2020.
Yin, M. and Zhou, M. Semi-implicit variational inference. In International Conference on Machine Learning, pp. 5660-5669, PMLR, 2018.
Zhang, L., Goldstein, M., and Ranganath, R. Understanding failures in out-of-distribution detection with deep generative models. In International Conference on Machine Learning, pp. 12427-12436. PMLR, 2021.
Zhao, S., Ren, H., Yuan, A., Song. J., Goodman, N., and Ermon, S. Bias and generalization in deep generative models: An empirical study. Advances in Neural Information Processing Systems, 31, 2018.
Zhou, L., Du, Y., and Wu, J. 3d shape generation and completion through point-voxel diffusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5826-5835, 2021.
13

---

A Variational lower bound for the SS-DDPM model
Pg (To:T) 9 (I1-T xо)
Pa (To 21:T Po (IT) IT Pá (Tt- (I1-1/22T))
88 (6)
E. ges (zorr) log
E (mo.r)
log
(26)
I gesd (1))
T
23 F4 (1:TT)
Po (cr) + log q (21r (200)
log 23 (2o|21:7)
+
log
(27)
9 (It-1 (0)
R (
log 20 (Iol (oolar)))
DEL (q (z=- 11 (+ ((TT)
(28)
B True reverse process for Markovian and Star-Shaped DDPM
If the forward process is Markovian, the corresponding true reverse process is Markovian too:
(I1-1:T) (
/DDPM QnnMM (ILTT
(x (3 /
II I P
(T. Rts =q -DDPN (xt-1 -DD (22))
DDPM (x1-1 (xrr)
(DDPM p (Tg xT
(29)
PDPM (ZOTT) q( (z))
ppPM (xa(21-1)
D (IT)
(I1-1lzer) DDPM (FTT)
II
DDPM (I1-1 (x)) (30)
[=1
For star-shaped models, however, the reverse process has general structure that cannot be reduced further.
(=or)) II 20 (0)(=0)= -x)
II 00 (xL-1L
(31)
In this case, a Markovian reverse process can be very poor approximation to the true reverse process. Choosing such an approximation adds an irreducible gap to the variational lower bound:
( I po (o
Cas Markow (0) ( log
(32)
(31:TT1 21)
25 (Ro.T) log po(ET) [T Po ( 2z (x 11= (2
(33)
(xT) I 88 (x+ [1:TT 11
R (xar)
(go) - DRE
Σ 1=1
DEL (4 (fe-1(z)) po (e(
(34)
Reducible
T
Σ DeL. (1 qee (xe- (2x)))
(35)
Inreducible
Intuitively, there is little information shared between Ft-1 and 2,, as they are conditionally independent given 2o. Therefore, we would expect the distribution (#t-1|az to have a much higher entropy than the distribution g ( making the irreducible gap (35) large. The dramatic effect of this gap is illustrated in Figure 3
This gap can also be computed analytically for Gaussian DDPMs when the data coming from a standard Gaussian distribution ((zu) = A/ (æo; 0, 1). According to equations (3), the best Markovian reverse process in this case is (o( = q (x4-1la))) [xe). results in the following value of the variational lower bound:
T
Markov = -[(g(20)) + (xarr) - H( (xT))
28 + log (2 0 , x ))]
(36)
2
14

---

If the reverse process is matched exactly (and has a general structure), the variational lower bound reduces to the negative entropy of the data distribution:
0
-10
P (2o:T)
4 (x0:7) log
(37)
q (II: (ro)
8 (3O:r)
-30
=Bss log
(38)
Markovian reverse
98th (01:2 (20)
True reverse
-40
(ooo)) log ((20)) log(2
(39)
200
400
As shown in Figure 7 the Markovian approximation adds substantial irreducible gap to the variational lower bound.
Figure 7: Variational lower bound for Gaussian Star-Shaped DDPM model computed with the true reverse process and with the best Markovian approximation.
C Proof of Theorem
We start by establishing the following lemma. It would allow us change the variables in the condition of conditional distribution. Essentially, we would like to show that if a variable y is only dependent on x through some function h(r), we can write the distribution plylæ)) p(yl =>)==)(())-
Lemma Assume random variables x and y follow a joint distribution p(x, v) = p())p(y)x)) where p(ulz) is given by [1, h. Then plylar) p(yl=)] where the variable is defined = h(z).
Proof. We can write down the joint distribution over all variables as follows:
p(z,y, 2) p()) h(z)))( h(z)).
(40)
By integrating x out, get
ply, 2) = /
/ - (lt))f(u.))(==MMM)) h(x))dr=
plz, U,,,8,2), z)xx
(41)
= =(y,)2) / p(x)6(2 - Mxx)d f(y,2)p(2)
(42)
Finally, obtain
p(ulz)1==h(m) = p(y.2) 2)
1==h(a)= = f(y, f(y,=))==n(() = f(y,h(x)) = plulz)
(43)
p(2)
Theorem 1. Assume the forward process of star-shaped model takes the following form:
(x)(a)) = hx())) exp (n TT(:) - 24 (2ro) (xo) = Azf(zz) + b..
(13)
(14)
Let Gt be a tail statistic, defined as follows:
T
G. = (((xeT) = >AATTT (T.))
(15)
5=t
Then, G. is a sufficient tail statistic:
th (x1-1(1TT) = - (I
(16)
Proof.
nd (01-1 (xxT) = (II-11 (20 )dro
(44)
g (2o) = ((xo) IT
s= 8 (x, (0)
g(ar)
T
[[no(ae)
Σ (mo( 2c(2x))
T
exp
(45)
9 (ITTT
(2x.T)
15

---

g(zo)
2'
Σ (00l((oo)bb)))) T(aa) - (o(coo)) -Qacoo)))) -
T
II Mo(xa)) exp
(46)
pea (INTT)
g(20)
II Hea (Hal exp { [(x0)
>
(1s(s)) + 10! - -2a(c0l)
(47)
qss (FAT))
smt
sm0
q(
T
( exp f(oo) + Σ (T.) (Po(oo))
a distribution must]
(48)
9 (Ter)
be normalized
g(zo) exp Gi 22(x0)
Lemma for G. 2] Loo as and Te.r as x
9 "8 (xul
(49)
[q(( exp [((o)) - E
/ dwo
q 88 (zx- (2)e)) = (100 = / (o00 = (
(50)
D Gaussian SS-DDPM is equivalent to Gaussian DDPM
Theorem 2. Let 0g pDPs define the noising schedule for a DDPM model De via B. = (a a, a.DraM)
//ppa)//aa DDPM Let
1/a-11
tt (Tur) be Gaussian SS-DDPM forward process with the following noising schedule and sufficient tail statistic:
q 88 (xel = N (z ai
Io,
1-
at
(22)
1 - oz
DDPM
T
(((TT)) =
where
(23)
V a
s=1
at DDMM
0x 1++1
DDPM
(24)
DDPM
01+1
DDDM
Then the tail statistic G. follows a Gaussian DDPM noising process q (lr.r)) defined by the schedule at apor Moreover, the corresponding reverse processes and VLB objectives are also equivalent.
Proof. We start by listing the necessary definitions and expressions used in the Gaussian DDPM model (Ho et al. 2020
T
DDPM = ((0))
II
DDPM (xt 11)-1)
(51)
i=1
DDPM (2o1 = N (125 /1 - 1
(52)
DDPM
@DDPM
(53)
at
DDPM
a PDPM
(54)
MDMM (are (20) = N ari V
o
DDMM
xo, (1 oz (
(55)
DUrM
1-11 Br
DDPM
1 0
DDPM 40
0
DDDM
(56)
DDPM
5DDPM
a.DP 3
(57)
pDPM (1 20) N (x--1; Phe (2t, vo), 20) 3DDPMY
(58)
16

---

pe P (IO.T) =gobpM (TT)
PDPM
DDPM [1 (1)-1
(59)
DDPM ( (2,)) = N (zz-1: (Tr (I, t)).
(60)
In this notation, the variational lower bound for DDPM can be written follows (here we omit the term corresponding to FT because it's either zero or a small constant):
CDDPM (0) = (DDMM(E
log PoggpoPM(an)) (xglZ1)) - Σ
Det (2,-
pb padps (x,
(x) (x2-1|zz))
(61)
t=
T
log pa
DDDMMMMa (20o(1)))
Σ DEL (D ro) (p 1 (2 2.PMMM)))))
(62)
y Vappa aa 3DDP
(xo 28 xpprr (Ie, (xcr)()))
1 1-pppM
DKL (FoP 2o)
/ (x)11(1 mo)l pmp
(63)
2MPpPr
For convenience, we additionally define a1++ O. Since all the involved distributions typically isotropic, consider a mne-dimensional case without the loss of generality.
First, we show that the forward processes oe (zo: G1:r) and q DDPM (Xorr) ) are equivalent.
1- 1-a az
DDPM
T
Es
G.
(64)
Since G. is a linear combination of independent (given mo) Gaussian random variables, its distribution (given xo) is also Gaussian. All the terms conveniently cancel out, and we recover the same form as the forward process of DDPM.
(G = - Gri 1
DDPM
T
a
DDPM
T
a. ) ))
(65)
DDPM
a
DDPM
amt
Gii aj
DDPM
DDPM
oi DDPM
a'
DDPM
= N
at
borst 400
V at
DDPM
(66)
=N (G1s
(Gs5 v aapPp
To (1 - a aDDem
DDPM (oe (co)
(a, GG
(67)
Since both processes share the same (( and have the same Markovian structure, matching these marginals sufficiently shows that the processes are equivalent:
(zo. G1:) = DDPM murr=Guer
(68)
Consequently, the posteriors are matching too:
(G,-1(G(, 20) a nDMM (21-1 200
(69)
Next, we show that the reverse processes p (G,- and pp pDuPM (I+- (21) are the same. Since G, in SS-DDPM is the same as 24 in DDPM, we can assume the predictive models v (G,, 1) and
1) to coincide at da G..
G
1 DDPM
Σ
T
DDPM
184-1
DDPM
DDPM
Gi =
(70)
DUPH
DUPST
38
DDPM
04 11
DDPM
DDPM 0f-1
a
DDPM
DDPM -
G
(71)
02
PDDP
==t-1
where -t-1 Pø
(It-1 (G.) = 88 (zz-1 (10) 100 (m0)) ().,)) Therefore, p (G.- is also a Gaussian distribution. Let's
take care the mean first:
1 1- DDPM
DIPM
DDPM
B 25 (G-11,)
0,1
30 (Gt,
G
(72)
DDPM
1
53 at-1
02
IDPM
17

---

DDPM
DDPM
DMPM
No (G + Vapars (1 a--1 mDPM
G
(73)
DDPM
1- 0f MaPM
f111
(1 01
DDPM)
a at
DDDP DDPM
ro(Gr,,)) + VapaPM(1 (1 - (DDP
DDPM
Gr
(74)
1
0z
DDPM
0 DDPM
DDPM
(1
1-11 Jay
DDPP MDPM
4 DDPM
(DPM)
C-- zo () +
DDPM
G =
(75)
1 0 DDPM
0
DDPM
DDPM
a.Dm + @
DeMMM t)
DDDM
0/-1
DDPM
G.
(76)
1
02
at
DDPM
DDPM B
DDPM
at DDPM 28 (G2
t) +
Gt
(77)
at
TDDPM
Now, let's derive the variance:
Dags g8 (G.- GG.)
a-1
DDPM
a1-1
(1 -
88
- =
(78)
Vapapa
0/-1
0t11
DDPM
(1 - D )
ar-
22
01
DDPM
at
DDPM
88
a1
DDPM
1:-0
88
1--1
DDPM
ap DDPM
=
(79)
(1 0
DDPM
)a
DDPM
(1 - 1-
α apops
(1
DDPM
)apDeM
= -1-
(80)
02 DDDPM
JDDPM
1- apppp
1 T
- DDPM at
PDrM
of
DDPM + 0t
DDPM)
DDPM 1t @DDPM
ADnM
(81)
Therefore,
v Va-11 3e
DorM To (Gt + Vap DDPM at-1
DDPM)
pe 2 (G.- =N
Gr-15
1 0f
DDPM
Ge, 30 5DDPM
= P8
DDPM
(31-1
(3+
(2x)
10
(82)
Finally, we show that the variational lower bounds are the same.
logre (x) - Σ DRL. (10- (r. (r-- (c))
8 (0) E.g (soor)
(83)
1=2
T
log as (nol (T) - > DxL (1 (1=
|| q0 (10m-1)))))mmm00000
= (
(84)
2=
Since G1 from the star-shaped model is the same as x from the DDPM model, the first term log v G1) coincides with
log ph DDPs (2gl2<)))))) 11
(Va.
a 88
(T0- - zo t)))²
DKL.
g (20-
(85)
2(1 a7-1)
1
DDPM
DDPM
(30 - Ia ())=
(50- - 300-IoGG,, (r,(,e,))
(86)
21
88
2
1
DDPM
at
DDPM
1
0
DDPM
0t
DDPM
(50 x00 (Go 1))2
1
a B gpnpx
DDPM
(50 - Io(
(87)
2 (1 0
DDPM
(1
a pDrM
2 (1
a PDPM
- o (DPM)
pDPM
pDPM)
a.-1
PDPM MDDpp
(xo - z
-DDD (2,,, t))²
(1 (1-
(x0 x poPp (z):
(88)
2(1- a.-PPN) ap.pM.
3 Maddg
/ ADDPM
2
1-apDPM
(xg-
xB
DDPM (x), t))
DKL
(D- 200
waPm (x+-1 [xt, xx) po..pp))))))) (89)
Therefore, we finally obtain (0) = CDDPM(O).
18

---

E Duality between star-shaped and Markovian diffusion: general case
We assume that (zx;) and all matrices A, (except possibly Ar) are invertible. Then, there is a bijection between the set of tail statisties GeT and the tail Zer.
First, we show that the variables Gier form a Markov chain.
= / (.G--//oo...rr))((o//errrrd.r
GGrr- (aT)
(90)
/ [[G=-1 q(Gr-11G1)
(91)
T
g( G.T) G II (G-IIG q( G1) -(G I ((G
g(zo)g
(92)
1=2
The last equation holds since the reverse of a Markov chain is also a Markov chain.
This means that a shar-shaped diffusion process on F1.T implicitly defines some Markovian diffusion process on the tail statistics GT. Due the definition of G G. A (x1-1), we can write down the following factorization of that process:
T
(( GET) = (( II
((G:-11Ge..0)) t=2
(93)
The posteriors g(Gr-1|Gr, mo) can then be computed by change of variables:
T-1 (AIT(G---Gi)
= g zo)
- det
(94)
dG,-1
This also allows us to define the reverse model like it was defined in DDPM:
Pe(Gi-1(G,)
(95)
This definition is consistent with the reverse process of SS-DDPM po po( = q ( (G Now that both the forward and reverse processes are defined, we can write down the corresponding variational lower bound. Because the model is structured exactly like a DDPM, the variational lower bound is going to look the same. However, we can show that it equivalent to the variational lower bound of SS-DDPM:
T'
Dunl ! Bg(
logro(zo|G)))
Σ
DKL. (q(Gr-
21) [Ipo DKL. (g(Gr/zo) po (GT))
(96)
Byc 20)) log g(G:-1|Ge,, DRL (a(xT(40) ((lo((TT))
logro(2o|G.)) - Gir))
(97)
po Gt)
2=
E By(a, 1120 log ((5:-1000) - DRi. (4(ε)(20) pa(er))) po Do(21_1(G)) G+)
= (
logpo(zo|G1)
(98)
2=2
= ))(
logPo(zo|G]) - - DKL (q(x=-1<=0) po - DRL (a(=)(20)
(IPo(ET))) (l = C (0)
(99)
As we can see, there are two equivalent ways to write down the model. One is SS-DDPM, a star-shaped diffusion model, where we only need to define the marginal transition probabilities g( (rp). Another way to rewrite the model terms of the tail statistics G.... This way we obtain non-Gaussian DDPM that is implicitly defined by the star-shaped model. Because of this equivalence, we see the star-shaped diffusion of æι.T and the Markovian diffusion of G1:r as pair of dual processes.
19

---

F SS-DDPM in different families
The main principles for designing SS-DDPM model are similar to designing DDPM model. When goes to zero, we wish to recover the Dirac delta, centered at .np:
(x) t-20
8(xx - m0)
(100)
When goes to T, we wish to obtain some standard distribution that doesn't 't depend on Zig:
(FT)
(101)
In exponential families with linear parameterization of the natural parameter 17: (an) a, (x)) b,, we can define the schedule by choosing the parameters and by that satisfy conditions [000 After that, we can use Theorem to define the tail statistic (=()))) using the sufficient statistic T2x)) of the corresponding family. However, as shown in the following sections, in some cases linear parameterization admits a simpler sufficient tail statistic.
The following sections contain examples of defining the SS-DDPM model for different families. These results are summarized in T
F.1
Gaussian
20 (c)lro) = N (15 V zo. (1 -0,
(102)
Since Gaussian SS-DDPM equivalent to a Markovian DDPM, is natural to directly reuse the schedule from a Markovian DDPM. As we show in Theorem 2 given Markovian DDPM defined by apDeM the following parameterization will produce the same process in the space of tail statistics:
DDPM
at+1
(103)
a
58
1
DDPM
DDPM /++1
1
DDPM
T
(((E))
(104)
a
DDPM
amat
The KI. divergence computed as follows:
DKL. (1 (x)(20) 11 (xr() 04 (I0 xz0)
(105)
2(1 - 0x:0))
F.2
Beta
(<ze|=o) = Beta( (t)
(106)
There are many ways to define a noising schedule. We choose to fix the mode of the distribution at xo and introduce a concentration parameter Ps, parameterizing a₂ 1 MTJo and 3r = 1+ v1 zo). By setting 1n to zero, we recover uniform distribution, and by setting it to infinity we obtain the Dirac delta, centered at zg. Generally, the Beta distribution log x has a two-dimensional sufficient statistic T(x) log ))) However, under this parameterization, we can derive a
one-dimensional tail statistic:
g( (Fo) x exp (ar - 1) log ve + (B: - 1) log(1 xz)) = exp {urto log + v1 - 2o. log(1 - xz)} = (107)
=
1xzolog
Ft 1
+ vz log(1 x1) exp (T h+()))))))
(108)
Therefore we can use ()() = Uz00 and T(x) = log a I-x to define the tail statistic:
(x(xer) Σ Us log I2
(109)
F2
The KL divergence can then be calculated as follows:
(at) (20)) (r + 14 14 - 20)(((a((zo)) -(33))))))))(33()))))))) ((3((z))))
DKL (0 (xelzo) 18 (x, = logg Beta(or Be (xo))
(110)
20

---

Xt
MA
t = 0
t= 7
t = 14
t = 21
t = 28
35
42
49
56
63
Figure 8: Visualization of the forward process Dirichlet SS-DDPM on a three-dimensional probabilistic simplex.
E3 Dirichlet
( (z0))00) = Dirichlet(xr;; "...
(111)
Similarly the Beta distribution, we choose fix the mode of the distribution zo and introduce a concentration parameter parameterizing af + v. This corresponds to using the natural parameter (xn) = 1/Tp. By setting 1/2 to zero,
we recover uniform distribution, and by setting it to infinity we obtain the Dirac delta, centered at To- We can use the sufficient statistic (xx) (log x² ..... logx*) )T to define the tail statistic:
G((=T)) Σ
log K)T
(112)
The KI. divergence then be calculated as follows:
K ((a)()))) + - =t)e(at log (xa))
DKL (a
(113)
F.4 Categorical
g(xx|20) Cat(zr;
(114)
In this case, mimic the definition of the categorical diffusion model, used in D3PM. The noising process is parameterized the probability vector Pt x.Q By setting Qo to identity, recover the Dirac delta, centered at Z0-
In this parameterization, the natural parameter admits linearization (after some notation abuse):
(xp) = 1,)( = xo log Q,, where
(115)
log is taken element-wise, and we assume log 0 = The sufficient statistic here is a vector 2.. Therefore, the tail statistic can be defined as follows:
( Σ (log Q. x!)
Σ log( log(Q_a?))
(116)
The KL divergence can then be calculated as follows:
(xo
DKL ( (er = Σ (xo log
(117)
(x)
21

---

Note that, unlike in the Gaussian case, the categorical star-shaped diffusion is not equivalent to the categorical DDPM. The main difference here is that the input G, to the predictive model Zo(G.) is now a continuous vector instead of one-hot vector.
As discussed in Section H. proper normalization is important for training SS-DDPMs. In the case of Categorical distributions, we can use the SoftMax(-) function to normalize the tail statistics without breaking sufficiency:
G. Soft Max (G((T))
(118)
To see this, we can retrace the steps of the proof of Theorem
(2o((ar)) x q(ro) exp { ((20) exp zo log G. + log Σ exp (GppG } x q(p))) { log 6.}
d
(119)
Therefore,
(xo)(t)))) q(00) exp { zo log Ci
88 (xo)(r))
(120)
Eao g(āo) exp To log [.]
Also, Categorical distributions admit convenient way compute fractions involving (GGe
((G//zo)
Σ
1 1 (((xer)]
(121)
Jur
#2t
Σ Ze-T
exp
2o log Q,)=!
1 [ = (((=))) = Σ exp (an [G. = (((r))
(122)
exp Goo
Σ 1 [G. (((xx))] = exp (znGr} #G.
(123)
This allows us to define the likelihood term p(an ( as follows:
g (m)))
exp Gao p(10)
(1) Σn, (G (()o) EE. exp (1 p(5))
(124)
where ((x)) can defined as the frequency of the token To the dataset. It also allows us to estimate the mutual information between zo and G₁: (GG() = (q(()))(C))(o))) log Σ 20 @Gee@o))(
(Gal(Fo)
[(x: G+) = (((2))((G))oog log ((G))
(125)
exp G, #a,
exp Gr)
(G. log Σn, cxp [ #G. ((0))
(G log En, exp Gi ((x))
(126)
It can be estimated using Monte Carlo. We use it when defining the noising schedule for Categorical SS-DDPM.
F55
von Mises
(<x:/20) = (on Ke)
(127)
The von Mises distribution has two parameters, the mode æ and the concentration k. It is natural to set the mode of the noising distribution zp and vary the concentration parameter Ke. When Ke goes infinity, the von Mises distribution approaches the Dirac delta, centered at xo. When K₁ goes to 0, it approaches a uniform distribution on a unit circle. The
sufficient statistic T(x:) sin x,
cos It
coos Z0 and the corresponding natural parameter is (xp) = Ket sin Zo
The tail statistic
(((T)) is therefore defined as follows:
(e( = Σ Kis
COS Zs sin .r,
(128)
The KL divergence term can be calculated as follows:
T, (Nee)
DEL ( (xg(20) |l p0 (z, = Kit (xee)
(1 cos(20 - x6)
(129)
22

---

t=0
t 11
22
33
44
t
t 66
t= 77
t= 88
t 99
Figure Visualization of the forward process the von Mises-Fisher SS-DDPM on the three-dimensional unit sphere.
F.6
von Mises-Fisher
(x22/20) = vMF( Ko)
(130)
Similar to the one-dimensional case, we set the mode of the distribution to zo, and define the schedule using the concentration parameter Ke. When K₂ goes to infinity, the von Mises-Fisher distribution approaches the Dirac delta, centered at æo- When Kg goes to 0, it approaches uniform distribution on unit sphere. The sufficient statistic is T(x) = x, and the corresponding natural parameter is no (Iu)
Kyžu- The tail statistic Gr (xer) is therefore defined follows:
Ge(Fer)
Σ Kg
(131)
s=t
The KL divergence term can be calculated as follows:
TK/2(KEt DEL (g (xe(20) |l 10 (xx(G)) Kt "x/22-1(ie)
10 x8)
(132)
E7 Gamma
( [(x; (1s ou Be)
(133)
There are many ways to define schedule. We choose to interpolate the mean of the distribution from 2o at = to 1 at = 7'. This can be achieved with the following parameterization:
(zo) (E + (1 (t)=
(134)
The mean of the distribution is Cu and the variance is Cg 3.
7 Therefore, we recover the Dirac delta, centered at To. when we set & to 0 and to infinity. To achieve some standard distribution that doesn't depend on To, can set & to 1 and o to some fixed value or.
In this parameterization, the natural parameters are a₂ 1 and - (zo), and the corresponding sufficient statistics are log Zt and he. Since the parameter an, doesn't depend on zo, we only need the sufficient statistic T(x:) 2, define the tail statistic:
((IET) ==
as(1 -,)
(135)
s=1
The KI. divergence can be computed as follows:
log 3c(xo) (((28) 3r(x
DKL (q (x, (x0) 11 (x, (G)) = at
+ Be
(136)
23

---

Xt
t=0
t= 7
t= 14
= 21
t= 28
t 42
49
Figure 10: Visualization of the forward process in Wishart SS-DDPM on positive definite matrices of size 2 2.
F88 Wishart
((x,/Xo) = Wi((,,; mz)
(137)
The natural parameters for the Wishart distribution are - V,, and -_p_ To achieve linear parameterization, we need to linearly parameterize the inverse of Vi rather than Vi directly. Similar the Gamma distribution, we interpolate the mean of the distribution from Xo at t = 0 to I a = T. This can be achieved with the following parameterization:
Pr E,1 + &
(138)
vx(X)) = n. - (Xo)
(139)
We recover the Dirac delta, centered at Xo. when we set & to 0 and n, to infinity. To achieve some standard distribution that doesn't 't depend on Xo. we set & to and n, some fixed value NT.
X, The sufficient statistic for this distribution T(X;) log |x,I) Since the parameter n, doesn't it depend on Xo. we don't
need the corresponding sufficient statistic log [Xe] and can use T(X,) Xx, to define the tail statistie:
T
C(Xer,t)
Σ ns(1 - E,)X.
(140)
8=t
The KL divergence can then be calculated as follows:
DKL. (qth (1 ||
(I, G.))
Th 2
log (X.)v(((() - -r
-v (X.) +2]
(141)
G Choosing the noise schedule
In DDPMs, we train neural network to predict given the current noisy sample z₁- In SS-DDPMs, we use the tail statistic G, as the neural network input instead. Therefore, is natural to search for schedule, where the "level of noise" in G₂ is similar to the "level of noise" in from some DDPM model. Similarly to D3PM, we formalize the "level of noise" the mutual information between the clean and noisy samples. For SS-DDPM it would s (xo; Gt), and for DDPM it would be [DOPS((30; (x). We would like to start from well-performing DDPM model and define similar schedule by matching (xp;; G.) R TDDPM (zo; 2x). Since Gaussian SS-DDPM is equivalent to DDPM, the desired schedule can be found in Theorem In other cases, however, it is difficult to find matching schedule.

---

In our experiments, we found the following heuristic to work well enough. First, we find a Gaussian SS-DDPM schedule that is equivalent to the DDPM with the desired schedule. We denote the corresponding mutual information as IN (zo: G,) TDDeM (zo: Then, we can match it in the original space of iy and hope that the resulting mutual information in the space of tail statistics is close too:
T (1oo:a) NN (z0; m4) & 1 (No: Gt)
N (To; Gi)
(142)
Assuming the schedule is parameterized by a single parameter v, we can build a look-up table | T.)) for a range of parameters v. Then we can use binary search build schedule to match the mutual information _ (2rg; wi) the mutual information schedule I (zo; 22). While this procedure doesn't t allow to match the target schedule exactly. it provides a good enough upproximation and allows obtain an adequate schedule. We used this procedure to find the schedule for the Beta diffusion in our experiments with image generation.
To build the look-up table, we need robust way estimate the mutual information. The target mutual information IN (mo; 24) can be computed analytically when the data ((x)) follows a Gaussian distribution. When the data follows an arbitrary distribution, it can be approximated with a Gaussian mixture and the mutual information can be calculated using numerical integration. Estimating the mutual information for arbitrary noising distributions is more difficult. We find that the Kraskov estimator (Kraskov et al., 2004) works well when the mutual information is high (I 2). When the mutual information is lower, we build a different estimator using DSIVI bounds (Molchanov et al. 2019).
y (x0o mn) = D (9 (zo, 2x) d (zo)a (z.)) = H [r.] (((zz/10)
(143)
This conditional entropy is available in closed form for many distributions in the exponential family. Since the marginal distribution 90h (xg. x44) = gas (Ia po (2r) is a semi-implicit distribution (Yin & Zhou 2018)) we can use the DSIVI
sandwich (Molchanov al. 2019 to obtain an upper and lower bound on the marginal entropy [[]a]]
-Egss log / (21 ((0
[[xre] Eggs log 4 (x+)
(144)
K
H[wx] 2 -Bag (aggi~g(zo) Bz, Nogeell00 log
1
Σ
(145)
K +
k=0
K
1
H[z;] < -B.g. -q(zo) Be Besellrl log
(x2l()))
(146)
K Amm
These bounds are asymptotically exact and be estimated using Monte Carlo. We use K 1000 when the mutual information is high (1 < 1 < 2), K = 100 when the mutual information is lower (0.002 < 1 < }), and estimate the expectations using M = 10 K- samples for each timestamp. For values I > 2 we use the Kraskov estimator with . 10º samples and = 10 neighbors. For values I < 0.002 we fit an exponential curve i(t) cat+ to interpolate between the noisy SIVI estimates, obtained with K = 50 and M = 10
For evaluating the mutual information [a G.) between the clean data and the tail statistics, use the Kraskov estimator with 1 = 10 and M = 10
The mutual information look-up table for the Beta star-shaped diffusion, as well as the used estimations of the mutual information, are presented in Figure I1 The resulting schedule for the beta diffusion is presented in Figure [2 and the comparison of the mutual information schedules for the tail statistics between Beta SS-DDPM and the referenced Gaussian SS-DDPM is presented in Figure 13
For Categorical SS SS-DDPM we estimate the mutual information using Monte Carlo. We then choose the noising schedule to match the cosine schedule used by HHooeeooom et al. (202I) using a similar technique.
H Normalizing the tail statisties
We illustrate different strategies for normalizing the tail statistics in Figure [4] Normalizing by the sum of coefficients is not enough, therefore we resort to matching the mean and the variance empirically. We refer to this trick as time-dependent tail normalization.
Also, proper normalization allows us to visualize the tail statistics by projecting them back into the original domain using (G.). The effect of normalization is illustrated in Figure 15
25

---

DSIVI upper bound
10
DSIVI lower bound
Kraskov
10
Combined
10
-2
[(xo Xu)
10
4.0
10
3.5
10
10
3.0
10
2000
4000
6000
8000
10000
10
10
-2
10
10
10"
10
10"
10"
Figure 11: The mutual information upok-up table for Beta star-shaped diffusion.
10
10
10
10
10
Vt
1(xo:
10
10
10
10
Beta SS-DDPM
Gaussian SS-DDPM
10
200
400
600
800
1000
200
600
800
Figure 12: The schedule for Beta star-shaped diffusion.
Figure 13: Mutual information between clean data and the tail statistics for beta star-shaped diffusion.
Synthetic data
We compare the performance of DDPM and SS-DDPM on synthetic tasks with exotic data domains. In these tasks, we train and sample from DDPM and SS-DDPM using T = 64 steps. We use an MIP with 3 hidden layers of size 512, swish activations and residual connections through hidden layers (He et al. 2015). We use sinusoidal positional time embeddings (Vaswani et al. 2017) of size 32 and concatenate them with the normalized tail statistics G. We add mapping to the use gradient clipping and EMA weights to improve
corresponding domain on top of the network. During training, stability. All models on synthetic data were trained for 350k iterations with batch size 128. time- tine-dependent tail normalization. We use DDPM with linear schedule and Lgimple
our experiments with
SS-DDPM synthetic data or Lyb as a loss function. We choose between gimmlle and Lew objective based on the KL divergence between the data distribution and the model distribution DxL (g(2g) Po (2o)). To make an honest comparison. statistics for DDPM in the same way that we do in dine-dependent tail normalization. In DDPM, a neural network makes predictions using 34 input, sampling stages.
precompute normalization
precompute normalization statistics for 2; and fix them during the training and
Probabilistic simplex We evaluate Dirichlet SS-DDPM on a synthetic problem of generating objects on three- dimensional probabilistic simplex. We use mixture of three Dirichlet distributions with different parameters training
26

---

6.1
0.5
0.5
0.5
0.6
12
12
161
-02
-04
-00
200
820
1000
200
302
000
(a) No normalization
(b) Normalized by the sum of coefficients
(c) Matched the mean and variance
C. Ce
C.
(G. EG,) Ttdl (2o)
std(C+)
Figure 14: Example trajectories of the normalized tail statistics with different normalization strategies. The trajectories are coming from a single dimension of Beta SS-DDPM.
Sum
Stats
100
500
700
Figure Visualizing the tail statistics for Beta SS-DDPM with different normalization by mapping the normalized tail statistics G. into the data domain using (Gx). Top row: normalized by the sum of coefficients, G. Σ7 Bottom
matched the mean and variance, G, (G, EG,) (wa(T( ET(x))
std(G.))
data. The Dirichlet SS-DDPM forward process is illustrated in Figure To map the predictions the domain, put the Softmax function on the top of the MLP. We optimize Dirichlet SS-DDPM on the VLB objective without any modifications using Adam with learning rate of 0.0004. The DDPM was trained on Luib using Adam with learning rate of 0.0002. An illustration of the samples is presented in Table 4
Table 4: Generating objects from three-dimensional probabilistic simplex.
Original
DDPM
Dirichlet SS-DDPM
Symmetric positive definite matrices We evaluate Wishart SS-DDPM on a synthetic problem of generating symmetric positive definite matrices of size 2 x 2. We mixture of three Wishart distributions with different parameters as training data. The Wishart SS-DDPM forward processes are illustrated in Figure 10 For the case of symmetric positive definite matrices V, MLP predicts lower triangular factor Lo from the Cholesky decomposition Ve Le For stability of sampling from the Wishart distribution and estimation of the loss function, we add scalar matrix 10 to the predicted symmetric positive definite matrix Vo. In Wishart SS-DDPM we use an analogue Tisimple as loss function. To improve the training stability, divide each KL term corresponding to timestamp by n,, the de-facto concentration parameter
27

---

from the used noising schedule (see Table 6). Wishart SS-DDPM was trained using Adam with learning rate of 0.0004.
The DDPM was trained on siiple using Adam with learning rate of 0.0004. Since positive definite 2 x matrices can be interpreted as ellipses, we visualize the samples by drawing the corresponding ellipses. An illustration of the samples is presented in S
Table 5: Generating symmetric positive definite 2 x 2 matrices.
Original
DDPM
Wishart SS-DDPM
J
Geodesic data
We evaluate von Mises- SS-DDPM on the problem of restoring the distribution on the sphere from empirical data of fires on the Earth's ' surface (EOSDIS) 2020). We illustrate points on the sphere using a 2D projection of the Earth's 's map. We train and sample from von Mises Fisher SS-DDPM using T 100 steps. The forward process is illustrated in Figure We use the same MLP architecture as described in Appendix [ and also use time- deeedeennddnt tail normalization. To map the prediction onto the unit sphere, we normalize the three-dimensiooal output of the MLP. We optimize Luth using the Adam W optimizer with a learning rate of 0.0002 and exponential decay with 7 = 0.999997. The model is trained for 2, 000, 000 iterations with batch size 100. For inference, we also use EMA weights with a decay of 0.9999.
K Discrete data
We evaluate Categorical SS-DDPM on the task of unconditional character-level text generation on the text& dataset. We evaluate model sequences of length 256. Using the property of Categorical SS-DDPM, that mutual information between Zg and G. (see Appendix F.4.. we match the noising schedule of G2 to the noising schedule of x, by Austin et al. (202T). We optimize Categorical SS-DDPM we use the default T5 encoder architecture with 12 layers. 12 heads, mip dim 3072 and qkv dim 768. We add positional embeddings to the sequence of tail statistics G. and also add the sinusoidal positional time embedding
directly compute
the VLB objective. Following BallowingAAustin Austin et al. (202T).
the beginning.
We use Adam (Kingma & Ba 2015) with learning rate 5 x 10-4 with a 10, 000-step learning rate warmup, but instead of inverse sqrt decay, we use exponential decay with ?- 0.999995. We use standard 90, 000, 000/5, 000, 000/5500, 000 train-test-validation split and train neural network for 512 epochs (time costs when using 3 NVIDIA A 110 GPUs: training took approx. 112 hours and estimating NLL on the test set took approx. 2.5 hours). Since the Sof tmax function does not break the sufficiency of tail statistic in Categorical SS-DDPMs (see Appendix [.4). we can use the Sof tmax function as an efficient input normalization for the neural network. Categorical SS-DDPM also provides us with a convenient way to define log p(zo| G1), and we make use of it to estimate the NLL score (see Appendix F.4
L Image data
In our experiments with Beta SS-DDPM, we use the NCSN+ neural network architecture and train strategy by [Song] et al. (2020b) (time costs when using 4 NVIDIA 1080 GPUs: training took approx. 96 hours, sampling of 50, 000 images took approx. 10 hours). We put a sigmoid function on the top of NCSN++ to map the predictions to the data domain. We use time-dependent tail normalization and train Beta SS-DDPM with T' = 1000. We matched the noise schedule in Beta SS-DDPM to the cosine noise schedule Gaussian DDPM (Nichol & Dhariwal 202T) in terms of mutual information, as described in Appendix G
28

---

M Changing discretization
When sampling from DDPMs, we can skip some timestamps to trade off computations for the quality of the generated samples. For example, we can generate Ft, from Ft,, in one step, without generating the intermediate variables +112----
p [x12) = goors((r, [2,, mo)l mo p (F12)t2)
(147)
DDPM abbा 1
aDDP AUieR
1
(1
DDDMPM
DDPM
DDPM
N
1 14
28
DDPM
(Tlz, 12) +
It2
1 12
apors dt
Eg
(148)
=W
DDPM
MDPM
DDPM
TT22 12)
+
Okgg
IDPM
-
DDPM) 021
Lt2
(1
DDDM
MDPM
12
DDPM
DDPM
012
DDPX)
DDDM
(149)
In the general case of SS-DDPM, we can't just skip the variables. If we skip the variables, the corresponding tail statistics will become atypical and the generative process will fail. To keep the tail statistics adequate, we can sample all the intermediate variables x, but do it in way that doesn't 't use additional function evaluations:
G Σ T(xs) + Gea where
(150)
2, (z (o | 2
(151)
In case of Gaussian SS-DDPM this trick is equivalent to skipping the variables in DDPM:
G₁ = 1-an aq
DDPM
12-1
I
DDPM
/aDPM
/
G =
(152)
DDPM
aPDPM
8=1
1
DDPM
02
49 (Gtz) (2)
+
an
DDPM t
88
cx
abbra (1 1/
aDDPM)
G12
(153)
0
38
@DDPM
111
11
DDD
DDPM
DDPM
a
DDPXI (1 - ieaaib
EG.,
88 (G12s 12) +
Gta
(154)
01g
DiPM
C12
DOPM
(1 a
DDPM
(appeM (122
DDPS
T6 (G1g) (2)
G12
(155)
DDPM
(1 0i1
DDDM) (1
0
DDPM)
DDPM (1
(aada
(apDeM (422)
(1-
(Giz (2) +
Gta
(156)
v /ah /apDrM( (1
0:22
@DDDMM
DDPM (1
0/2 DDDPM)
(1 at
2
tg
0
at
DDPM 2
0
DDPM
012
DDPM
DG1,
(157)
DDPM
1 0
0
DDPM
DDPM)( (1 0
DDPM
a.
(a
(add
(158)
In general case, this trick can be formalized as the following approximation to the reverse process:
28 (Iz (Inn)t(Grra)= = II
qed (100) R
II
q 2
(159)
t=
29

---

Table 6: Examples of SS-DDPM in different families. There are many different ways to parameterize the distributions and the corresponding schedules. Further details are discussed in Appendix F.I
F.8
DISTRIBUTION (n//2o)
NOISING SCHEDULE
TAIL STATISTIC (((TT))
KI. DIVERGENCE
DRs. (f" ll p (a(lec4):))
GAUSSIAN
7
Viaata T-m,
N (as: vārao me)
Tela--o0 2(1-aa)
Rt E R
[-0
L-T
where di
142
BETA Beta (Fo), E [0
ae((a) + Pez7a
(xoo + v1 xo)
(sl) log Beta(ar( Be(m)))
El Uo log 1-
+00
Ut
0
+ n2 - (0)(((( ()(((()))))
6-0
1-T
DIRICHLET
a3(20) 1 + v.a
Σ R-1
I(a (mo)) + log r(at (mo))
Dir (my; of (mo)
of (zw))
E Me log
Re E [0. 1 EK =1
+00
V/
+ ve(a a =))
CATEGORICAL
(z(ou) = z0Q,
Cat (14; Pr (2a)) J. [0
(pu(no))s log (pi(2o))s (pi(ze))
>1 log (Q,x!)
>
Q
Qr
DP %1
6-0
vM (2) (ar du, Rs) 2t. E [- m]
VON MISES
+00
/ous Σn No
Ailee) (1 - cos(uro ax)) Rk To(me)
[-10
2-1T
sin
VON MISES-FISERE
vMF (2re 2o, Ks)
4:00
0
DI No
Facyaliss) No ( 20 x0)
J. E [] ||
6-0
(xro) =cc(Ee (1 -())rr
GAMMA
T ((0o) (20)) dt E (0, +00)
400
21,0000 as(1 (0(1-6.)am -6,Jas
ai [og dr(so) + (atro) Bilog)
-
4 1-7
0 040
2-32"
(Xo) + 61)
WISHART W (Xama Vi(Xo))
V((Xa) = 14 Hi (Xo)
[log VV ((X)) tr ((X)) +p)
ΣV EL no(1 8,)X.
2
X, gpxp
ni 1-1 DT
X, 0
0 E ビ-+
t-10
30


Reviews:
[{"review_id": "PvRqMRJdQS", "reviewer": "Reviewer_EZ37", "summary": "This paper proposed a star-shaped denoising diffusion probabilistic model (DDPM), which extends DDPM to non-Gaussian noises. As a result, the backward/generative process requires conditioning on tails. The authors then propose an efficient tail conditioning strategy which works when the forward process follows an exponential family with linear parameterization. Duality between star-shaped and Markov diffusion processes are also established which provides theoretical support for star-shaped DDPM for its ability to go beyond Gaussian noises. The effectiveness of star-shaped DDPM is demonstrated on several experiments.", "questions": "1. How does DDPM perform on the Geodesic data in the experiments?\n2. How much improvement can SS-DDPM have over DDPM on real data? The advantage over DDPM seems a bit tiny based on current evaluation. Also, in which case would non-Gaussian noise be useful for DDPMs?\n3. Rissanen et al. (2022) also proposed a similar star-shaped DDPM, where the reverse process is Markovian based on the heat equation. How does your method compare to their's?", "limitations": "Yes, they do.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. This paper is written very clearly and well organized. \n2. It also provides a general framework for DDPM that can incoporate no-Gaussian noises, where previous efforts are designed for specific noises and not generalizable to other distributions.\n3. Duality between star-shaped and Markovian diffusion is established, which shows SS-DDPM and vanilla DDPM are equivalent in the Gaussian case. ", "weaknesses": "1. While experiments on synthetic data demonstrate the ability of SS-DDPM to deal with non-Gaussian noises, the effectiveness of SS-DDPM on real data needs further verification. The current experiment on real data seems a bit inadequate.\n\n2. Star-shaped denoising diffusion models have been proposed before (Rissanen et al. (2022)), and there is no comparison to this previous method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the comments! We would like to address your questions below.\n\n**W1 and Q2:**\nDDPM is a special case of SS-DDPM, so it is safe to expect SS-DDPM to perform at least as good as DDPM. At this stage our goal was to demonstrate that the model can be successfully applied with a variety of noising distributions rather than perfecting the model for each individual task or finding the best task for the model. Careful tuning of the noising schedule and other improvements to the training procedure (e.g. resampling timestamps using importance sampling, more flexible reverse distributions, more architectural improvements, etc) could be used to improve the performance further, but the added hyperparameters would require a careful ablation and make the experiments a lot more expensive. In general, we would expect non-Gaussian SS-DDPM to outperform Gaussian DDPM when data (or parts of data) lies on manifolds that naturally support distributions from exponential families, e.g. hyperspheres, p.s.d. matrices, simplexes, etc. For example, data like molecular graphs could combine categorical data (atoms), spherical data (orientations) and positive data (distances).\n\n**W2:**\nIHDM heavily relies on blurring rather than adding noise, resulting in very different diffusion dynamics. The small amount of noise (std=0.01) is there for regularization and to prevent the probabilistic model from being degenerate. Due to blurring and low noise there is a lot of information shared between consecutive timestamps, making both the forward and the reverse processes essentially Markovian. While the ELBO looks similar due to a similar factorization of the forward process, the underlying model is very different: our model is much closer to DDPM than to IHDM. We will expand on these differences in the next revision.\n\n**Q1:**\nWe have evaluated the NLL on the fire dataset. Our result is $-1.26 \\pm 0.14$. For comparison, the results reported in Riemannian Diffusion Models (Huang et al 2022) are $-1.38 \\pm 0.05$ for RDM, $-1.24 \\pm 0.07$ for RSDE (Riemannian Score-Based) and $0.28 \\pm 0.2$ for DDPM (Stereographic SDE). Our model performs similar to RSDE and RDM and outperforms DDPM.\n\n**Q3:**\nIHDM achieves 18.96 FID on CIFAR-10, while our Beta SS-DDPM achieves 3.17 (IHDM uses 200 steps and we use 1000; at 200 steps our FID is 3.45). Also, while it looks like IHDM can use any noise distribution, it is not trivial to generalize it to other domains: one would need to come up with an appropriate notion of blurring.\n"}, {"review_id": "xr44i36dPz", "reviewer": "Reviewer_UBua", "summary": "The paper presents the Star-Shaped DDPM (SS-DDPM), a general recipe for designing a diffusion model with a noising process lying in a general subset of the exponential family. With a Gaussian noising process, SS-DDPM recovers the DDPM. Diverse experiments on synthetic and practical image and text datasets demonstrate the effectiveness of the proposed SS-DDPM.", "questions": "What are the limitations of the proposed SS-DDPM, when compared with other diffusion models?", "limitations": "N/A", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "(1) A general recipe for designing a diffusion model, termed Star-Shaped DDPM (SS-DDPM), is proposed.\n\n(2) The recipe is derived and analyzed in detail, with convincing statistical justifications.\n\n(3) Diverse experiments are conducted to demonstrate the effectiveness of the SS-DDPM.", "weaknesses": "(1) Many technical details are given in the supplementary material.\n\n(2) The advantages and disadvantages of the SS-DDPM over the DDPM are not explicitly discussed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the comments! We would like to address your questions below.\n\n**Weakness (1):**\nWe have tried to summarize the most crucial technical details in Section 3. Due to space limitations, we had to put more details in the appendix. Unfortunately, the NeurIPS template makes it more difficult to directly reference the appendix (supplementary material has to be uploaded as a separate file), so we hope that the upcoming arXiv version would be easier to read. Please let us know if particular sections stand out as being confusing, and we would do our best to clarify them in the next revision.\n\n**Weakness (2) and Question (1):**\nSS-DDPM is a direct generalization of DDPM, so many properties of the models are shared. The main advantage of SS-DDPM is being able to use other noising distributions which may be more appropriate to the particular task. The main disadvantage of non-Gaussian SS-DDPM is that it loses many interesting extensions of DDPM like DDIM, DDRM and the connection to SDEs. We feel like it should be possible to adapt these extensions to support SS-DDPM, so it is an interesting opportunity for future work. Also, we found it to be more difficult to come up with sensible noising schedules for SS-DDPM. SS-DDPM is rather sensitive to the noising schedule, so it would be nice to have a better way to choose (or learn) noising schedules. In this work we try to reuse the schedules used by existing models by matching the mutual information between the target clean data and the noisy input of the denoiser. Finally, in non-Gaussian SS-DDPM sampling the tail statistic $G_t$ requires sampling the whole tail $x_t, x_{t+1}, …, x_T$. This is not a big deal in most applications, as sampling these datapoints does not require evaluation of neural networks, but it is still worth noting. In practice it doesn’t increase the training time as DNN evaluation is still the bottleneck."}, {"review_id": "laGbKbj3Hd", "reviewer": "Reviewer_mgR2", "summary": "This paper proposes a non-Markovian diffusion model named the star-shaped diffusion model that generates a sequence of noised image from the original image in the forward process.\n\nThis paper studies the theoretical foundation of such new type of model showing that if the forward process is based on a subset of the exponential family of distribution, a tail statistics is sufficient statistics for the reversed process. Based on such a foundation, the reversed process can be conducted.\n\n", "questions": "Is there any ODE-version of this SDE-based model?", "limitations": "See above", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The overall structure of this new diffusion model is new, novel and interesting.\n\nThe application of PKD theory is sound.", "weaknesses": "Although the model is pretty novel and new, there is a lack of motivation from practical problems -- the experiment are weak and kind of toy. A showcase of the unique application of this model is appreciated.\n\nLack of comparison of some other non-markovian diffusion on constrained domains such as [1,2]\n\n[1] Learning Diffusion Bridges on Constrained Domains\n[2] First Hitting Diffusion Models for Generating Manifold, Graph and Categorical Data", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the comments! We would like to address your questions below.\n\n**Weakness 1:**\nAt this stage our goal was to demonstrate that the model can be successfully applied with a variety of noising distributions rather than perfecting the model for each individual task or finding the best task for the model. In general, we would expect non-Gaussian SS-DDPM to outperform Gaussian DDPM when data (or parts of data) lies on manifolds that naturally support distributions from exponential families, e.g. hyperspheres, p.s.d. matrices, simplexes, etc. For example, data like molecular graphs could combine categorical data (atoms), spherical data (orientations) and positive data (distances).\n\n**Weakness 2:**\nThanks for pointing out “Learning Diffusion Bridges on Constrained Domains”, we missed it! While we don’t have any experiments that overlap with that paper on hand, we would expect Categorical SS-DDPM to perform similar to D3PM. The results could probably be improved by tuning the noising schedule to the particular tasks. Same point holds for the categorical data experiments with FHDM. Regarding FHDM on geodesic data: we achieve a similar performance on the fire dataset (our result is $-1.26 \\pm 0.14$, FHDM reports $-1.24 \\pm 0.08$). We will conduct the experiments on other geodesic datasets and add a comparison.\n\n**Question 1:**\nStrictly speaking, for now there is no known direct connection between SS-DDPM and SDE-based models beyond the Gaussian case. However due to the stochasticity in the reverse process, SS-DDPM is indeed similar to SDE-based methods. Due to its close connection to DDPM, we suspect that it should be possible to adapt the DDIM model to SS-DDPM. DDIM is a deterministic version of DDPM that is close to ODE versions of SDE-based models."}, {"review_id": "QJcOghsmmi", "reviewer": "Reviewer_EaR3", "summary": "This paper proposes a star-shaped diffusion probabilistic model, which is non-Markovian, and more like an autoregressive way of predicting intermediate states xt. Specifically, the authors define a forward process q(xt|x0) that each intermediate state xt is directly related to the initial state x0. The reverse process is defined by p(xt|xt, xt+1, ..., xT). To efficiently solve p(xt|xt, xt+1, ..., xT), the authors propose a compressive representation Gt(xt, xt+1, ..., xT) to extract all the information of x0 contained in history states {xt+1, ..., xT}, so the modified distribution becomes p(xt|Gt). The training and inference go in a similar way as DDPM. Simple experiments show that the proposed method performs better than the baseline methods MTD and DDPM.", "questions": "I doubt the reverse process of q(xt|x0) should be p(x0|xt), can you explain why the reverse process is p(xt|xt, xt+1, ..., xT)?", "limitations": "The scope of this paper is interesting, but it needs more time to polish the theory, presentation, and experiments. Considering its current form, I think this paper does not meet the acceptance criteria of NeurIPS. However, if there is enough evidence to show the value of this paper, I may also change my score.", "rating": 5, "confidence": 5, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "1. This paper provides a novel view of the definition of diffusion models and is very inspirational. It is worth thinking about whether the Markov process is the best design choice considering the fact that the whole history {xt, xt+1, ..., xT} contains more information than a single state xt. \n\n2. The author proposed a novel diffusion model with basic evaluations, which may inspire the community for further exploration. ", "weaknesses": "1. The presentation needs to be improved: (1) T, Ωt, and ht in Eq.13 lack interpretation; (2) It is not well proved that Gt covers all the information of x0 in {xt, xt+1, ..., xT}. Although the authors mention that there are explanations in the supplementary material, you need to ensure the reader has a basic understanding by just reading the paper.\n\n2. The advantage and application value of the proposed method is not clear. For example, can you provide a practical case that DDPM can not address while SS-DDPM can?\n\n3. The experiments are too simple and not persuasive. To prove the advantage of SS-DDPM, it is encouraged to compare with state-of-the-art methods on diverse benchmarks, \n\n4. As the authors said in line 128, \"In general case the dimensionality of Gt would grow with the size of the tail\"\n\n4. There is no evidence that SS-DDPM can apply acceleration methods like DDIM.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the comments! We would like to address your concerns and questions below.\n\n**Weakness 1 (1):**\nEq. 13 is a standard definition of the exponential family of distributions with a standard notation. $\\mathcal{T}(x_t)$ is the sufficient statistic, $\\eta(x_0)$ is the natural parameter, $h(x_t)$ is the base measure and $\\Omega(x_0)$ is a normalization constant, sometimes called a (log)partition function. We will improve this section.\n\n**Weakness 1 (2):**\nWe formulate this property as Theorem 1 and prove it in Appendix 3. It is a technical proof, and it is common practice to only include the proofs in the Appendix. The intuition behind this theorem is the same as behind the original PKD theorem. We generalize it for the non-i.i.d. case, which is why we have to add an additional constraint (linearity of the parameterization). Strictly speaking, we prove the following equality: $q(x_0 | x_t, \\dots, x_T) = q(x_0 | G_t)$, which means that “$G_t$ covers all the information of $x_0$ in $x_t,\\dots, x_T$”.\n\n**Weakness 2:**\nWhen the data or its parts lie on a manifold, its density in the original space would be degenerate, and DDPM would fail to recover it. Even if DDPM would produce good-looking samples, its density estimation would be unreliable. On the contrary, SS-DDPM with an appropriate distribution would both produce good-looking samples and a reliable estimate of the density. For example, data like molecular graphs could combine categorical data (atoms), spherical data (orientations) and positive data (distances).\n\n**Weakness 3:**\nAt this stage our goal was to demonstrate that the model can be successfully applied with a variety of noising distributions rather than perfecting the model for each individual task or finding the best task for the model. Comparing with domain-specific state-of-the-art methods would require ad-hoc task-specific modifications and a very expensive hyperparameter search to make the comparison fair. These more challenging settings are something we would like to explore in the future.\n\n**Weakness 4:**\nIn this paper we only consider distributions from the exponential family. In that case we provide a simple recipe for constructing a sufficient tail statistic $G_t$ of fixed dimensionality. Although the general case is beyond the scope of our paper, we suspect that in practice adequate statistics of fixed size would be enough, even if they are not strictly sufficient statistics.\n\n**Weakness 5:**\nThis paper is focused on establishing the base SS-DDPM model. Since SS-DDPM and DDPM are closely connected, we suspect that it would be possible to adapt other modifications of DDPM to SS-DDPM as well (e.g. DDIM, DDRM, connection to SDE). However, these extensions are beyond the scope of this paper.\n\n**Question 1:**\nWe discuss and illustrate it in detail in Appendix 2. The short answer is that we need to look at the forward and reverse processes as a whole, not at inverting individual steps.\n\nLonger answer: Diffusion models model the whole set of variables $x_0, x_1, \\dots, x_T$ by learning a joint distribution $p_\\theta(x_0, x_1, \\dots, x_T)$, called the reverse process, to approximate a fixed joint distribution $q(x_0, x_1, \\dots, x_T)$, called the forward process. When the forward process is Markovian, i.e. it factorizes as $q(x_0)\\prod q(x_t|x_{t-1})$, we can also rewrite it in reverse as $q(x_T)\\prod q(x_{t-1}|x_t)$, and, therefore, it makes sense to approximate it with a Markovian reverse process $p_\\theta(x_T)\\prod p_\\theta(x_{t-1}|x_t)$ (by using the same factorization and sufficiently flexible distributions, we can in theory reduce the approximation gap to zero). However, when the forward process is star-shaped, i.e. it factorizes as $q(x_0)\\prod q(x_t|x_0)$, we can only write it in reverse as $q(x_T)\\prod q(x_{t-1}|x_t, x_{t+1}, \\dots, x_T)$. We could still try to approximate it with a Markovian reverse process $p_\\theta$. It was the first thing we tried, and it fails miserably: see Figure 3 for details. In Appendix 2 we show that such approximation introduces a huge irreducible approximation gap which grows with the number of diffusion steps. Because of that we need to use the reverse process with the full dependency structure $p_\\theta(x_T)\\prod p_\\theta(x_{t-1}|x_t, x_{t+1}, \\dots, x_T)$. Fortunately, when the noising distributions come from the exponential family, we can rewrite it as a Markov process over sufficient tail statistics $p_\\theta(x_T)\\prod p_\\theta(x_{t-1}|G_t)$, where $G_t=G(x_t, \\dots, x_{t+1})$, so the procedure is still computationally efficient.\n"}, {"review_id": "abQRr5LkKa", "reviewer": "Reviewer_P24X", "summary": "The paper introduces a new probabilistic structure for denoising models that does away with a Markov forward process. The authors derive the reverse process in terms of a sufficient statistic that allows for efficient reverse sampling. The form of the model is derived for a variety of noising distributions and the model is evaluated on a range of different data modalities.", "questions": "In the proof of Theorem 1, how do you go from line (23) to line (24), as in, how do you know what q(G_t | x_0) is, for the numerator in (24)?\n\nFigure 6 is confusing to me because the figure makes it seem like the model is better than DDPM at all NFE but in the text the best FID is stated to be 3.17 which is the same as the best FID from DDPM and so shouldn't Figure 6 show the method and DDPM at least overlapping at 10^3 sampling steps?\n\nIn DDPM, the Gaussian form of the reverse distribution is justified by considering the limit as the noise in each step is taken to 0 and the number of steps is taken to infinity. Is there any such justification for the Gaussian form that you assume in your model or is this a heuristic?", "limitations": "The limitations of the framework e.g. restriction of the distributions and ability to find the sufficient statistics was nicely discussed.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "This work can potentially help many people in the generative modelling community because there has been a lot of work trying to extend corruption process to different data types e.g. discrete, manifold data and this paper has a general recipe that appears to work for a large chunk of these and more which could be really helpful when encountering non-standard data types. The example of the Wishart distribution on positive definite matrices is very cool.\n\nI can foresee many extensions to this work, looking to bring more distributions into the framework, extending it to continuous time which could bring up a lot of interesting technical problems and broadening the class of possible distributions by introducing approximate sufficient statistics.\n\nThe paper has a good suite of experiments on a variety of real world practically relevant data such as text and images and the performance of the model seems good", "weaknesses": "I think the paper could benefit from more intuitions and visualizations for the role of G. Initially, the model seems strange as x_t can jump a lot between x_t and x_{t+1} due to going to x_0 and resampling q(x_t | x_0). However, it seems that G instead is the smoothly changing statistic that is more analogous to the state in previous diffusion models. For example, I think figure 9 in the supplementary is quite interesting and could be moved to the main as it shows that a correctly normalized G looks very similar to the state in norm diffusion models. It would be good to have more intuition as to the role of G and how it is more closely linked to the 'x_t' of normal diffusion models.\n\nFurther, there is not really a experiment that really motivates the use of these more interesting distributions in a scenario where normal diffusion models really don't work at all. I appreciate that this is somewhat an open question at the moment in the literature more generally and the community has already proven an interest in adapting the corruption process to the data modality of interest so this may not be a big requirement.\n\nEdit after rebuttal: I have read the authors rebuttal and appreciate the answers to my questions about the derivations and the FID scores. I intend to keep my score as it is.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the comments! We would like to address your questions below.\n\n**Weakness 1:**\nWe used Figure 5 to illustrate this point in the main text, however we should probably make this point more clear in the text. The close connection between $G_t$ in SS-DDPM and $x_t$ in DDPM is crucial to understanding the model. We would try to expand Figure 5, give more attention to Figure 9 or rewrite the text to make this more clear.\n\n**Question 1:**\nThe first transition ((23) to (24)) is just using the normalization constraint. Since we have started with a distribution on $x_0$, we can drop all multiplicative terms that do not depend on $x_0$ and then renormalize. We have started with a conditional distribution $q(x_0 | x_{t:T})$, and were able to rewrite the density as a function of $x_0$ and $G_t$. This immediately means that the resulting distribution is a conditional distribution $q(x_0 | G_t)$, hence the second transition on line (24). Because of that we do not need to compute these distributions explicitly. We have unintentionally skipped over this step as it has previously seemed to be obviously true. However, we were not able to recall or find an existing lemma that establishes this fact (although surely it must exist somewhere), so we will include a more detailed proof. Thanks for pointing it out! Something along these lines should work:\n\n**Lemma 1.** Assume $x$ and $y$ follow a joint distribution $p(x, y) = p(x)p(y | x)$, where $p(y | x)$ is defined as $f(y, h(x))$. We would like to show that $p(y | x)=p(y | z)|_{z=h(x)}$, where $z=h(x)$.\n\nWe can write down the joint distribution $p(x, y, z) = p(x)f(y, h(x))\\delta(z - h(x))$. Then $p(y, z)=\\int p(x, y, z) dx=$ $\\int p(x)f(y, h(x))\\delta(z - h(x)) dx =$ $\\int p(x)f(y, z)\\delta(z - h(x)) dx =$ $f(y, z)\\int p(x)\\delta(z - h(x)) dx=$ $f(y, z)p(z)$.\nFinally, the conditional probability is $p(y | z)=\\frac{p(y, z)}{p(z)}=f(y, z)$ — the same function.\n\nIn the case of line (24) $x_0$ is $y$, $x_{t..T}$ is $x$ and $G_t$ is $z$.\n\n**Question 2:**\nUnfortunately, the original DDPM paper did not include the results on shortened generation (FID vs NFE). Therefore, we used the results reported by Nichol&Dhariwal 2021 (Improved DDPM), where the final performance of DDPM was slightly worse. This discrepancy is important, which is why we only conclude that SS-DDPM achieves a comparable performance rather than strictly outperforms DDPM.\n\n**Question 3:**\nIn our model we use the same distribution for the reverse process as we do for the forward process (e.g. Beta distribution for Beta SS-DDPM, etc — we define it in eq. 20). This is a natural choice because different distributions from the exponential family have different supports, and choosing other distributions would likely make the involved KL divergences infinite. Also, when these distributions are kept the same, it is usually possible to compute the KL divergence analytically and simplify it (see Table 3 in the end of the Appendix). However, you are right that this is just an approximation, and in general other distributions could make sense too. For example, mixtures could be a good candidate to consider: the “true” reverse distribution in eq. (19) is essentially an infinite mixture of these distributions. The same argument could be made for Gaussian discrete-time DDPMs too."}, {"review_id": "kNYg4RdX4H", "reviewer": "Reviewer_7FuV", "summary": "This introduces the so-called Star-Shaped DDPM (SS-DDPM). Instead of using Gaussian forward diffusion step in a Markovian manner, the diffusion process is directly conditioned on the data, in order to construct the \"true\" reverse process (posterior), resulting in the star-shaped diffusion process. This opens the door for using other transition process as demonstrated in the paper.  ", "questions": "When using other transition distributions in the forward process, what is the strategy to actually diffuse the data (into noises)?", "limitations": "Not much in this perspective. No comments", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. This is a solid paper that I have reviewed so far.  All the conclusions are backed by the theory.\n2. The paper is well presented and easy to follow the main the idea with full materials in the supplementary\n3. The idea presented in the paper is novel and may have wide applications.", "weaknesses": "Frankly speaking, I like to see this paper to be accepted.  I cannot form any weakness.  The only point I may raise is it would be much much better for the authors to share their implementation source code.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the kind comments!\n\nWe will release the source code with the camera ready version of the paper.\n\nRegarding your question:\nIn general, we follow the same intuition as when designing the forward process in DDPM. We start with low noise ($x_1$, $x_2$, ... should be reasonably close to $x_0$), and end with high noise ($x_T$, $x_{T-1}$, … should be almost pure noise, almost independent from $x_0$). And then we need to choose the probabilities such that the inputs to the denoising neural network (in DDPM it is $x_t$, and in our case it is the sufficient tail statistic $G_t$) gradually becomes more and more noisy. See Figure 5 for reference. To choose the transition probabilities somewhat automatically, we can refer to the rate at which the mutual information $I(x_0; G_t)$ changes with $t$, see Appendix 7: Choosing the noise schedule for details."}]

Author rebuttal:

Meta-review: Reviewers agree that this paper makes solid conceptual contributions to the literature on diffusion generative models.  In future revisions, please be sure to include the additional conceptual and experimental comparisons that arose in the discussion, and to clarify the technical points about which reviewers had questions.
Decision: Accept (poster)