SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\operatorname {K L U B} \left(z _ {t}, x _ {0}\right) = \operatorname {K L} \left(q \left(z _ {t} ^ {\prime} \mid f _ {\theta} \left(z _ {t}, t\right)\right) \mid q \left(z _ {t} ^ {\prime} \mid x _ {0}\right)\right). \tag {16}
$$
The proof in Appendix B utilizes the fact that  $q(z_{t}^{\prime}) = \mathbb{E}_{(z_{t},x_{0})\sim q(z_{t}\mid x_{0})p_{data}(x_{0})}[q(z_{t}^{\prime}\mid x_{0})]$ .
Note that the mathematical definition of KLUB is reused throughout the paper and can refer to any of the equations (11), (12), (15), or (16) depending on the context. Similar to previous analysis, we have
$$
\operatorname {K L U B} \left(z _ {t}, x _ {0}\right) = D _ {\ln B (a, b)} \left\{\left[ \eta \alpha_ {t} x _ {0}, \eta \left(1 - \alpha_ {t} x _ {0}\right) \right], \left[ \eta \alpha_ {t} f _ {\theta} \left(z _ {t}, t\right), \eta \left(1 - \alpha_ {t} f _ {\theta} \left(z _ {t}, t\right)\right) \right] \right\}, \tag {17}
$$
and can conclude with the following Lemma.
Lemma 5.  $\mathrm{KLUB}_t$  in (15) is optimized when the same optimal solution given by (14) is met.
**Optimization via KLUBs.** With the two KLUBs for both time reversal and error accumulation control, whose optimal solutions are the same as in (14), we are ready to optimize the generator  $f_{\theta}$  via stochastic gradient descent (SGD). Specifically, denoting  $\omega, \pi \in [0,1]$  as two weight coefficients, the loss term for the  $i$ th data observation  $x_0^{(i)}$  in a mini-batch can be computed as
$$
\mathcal {L} _ {i} = \omega \operatorname {K L U B} \left(s _ {i}, z _ {t _ {i}}, x _ {0} ^ {(i)}\right) + (1 - \omega) \operatorname {K L U B} \left(z _ {t _ {i}}, x _ {0} ^ {(i)}\right), \tag {18}
$$
$$
z _ {t _ {i}} \sim q (z _ {t _ {i}} \mid x _ {0} ^ {(i)}) = \operatorname {B e t a} (\eta \alpha_ {t _ {i}} x _ {0} ^ {(i)}, \eta (1 - \alpha_ {t _ {i}} x _ {0} ^ {(i)})), s _ {i} = \pi t _ {i}, t _ {i} \sim \operatorname {U n i f} (0, 1).
$$
2.4 Discretized Beta Diffusion for Generation of Range-Bounded Data
For generating range-bounded data, we discretize the beta diffusion chain. Denote  $z_{t_0} = 1$  and let  $t_j$  increase with  $j$ . Repeating (5) for  $T$  times, we define a discretized forward beta diffusion chain:
$$
q \left(z _ {t _ {1: T}} \mid x _ {0}\right) = \prod_ {j = 1} ^ {T} q \left(z _ {t _ {j}} \mid z _ {t _ {j - 1}}, x _ {0}\right) = \prod_ {j = 1} ^ {T} \frac {1}{z _ {t _ {j - 1}}} \operatorname {B e t a} \left(\frac {z _ {t _ {j}}}{z _ {t _ {j - 1}}}; \eta \alpha_ {t _ {j}} x _ {0}, \eta \left(\alpha_ {t _ {j - 1}} - \alpha_ {t _ {j}}\right) x _ {0}\right). \tag {19}
$$
A notable feature of (19) is that the marginal at any discrete time step  $t_j$  follows a beta distribution, similarly defined as in (4). We also note while  $q(z_{t_{1:T}} \mid x_0)$  defines a Markov chain, the marginal
$$
q \left(z _ {t _ {1: T}}\right) = \mathbb {E} _ {x _ {0} \sim p _ {\text {d a t a}} \left(x _ {0}\right)} \left[ q \left(z _ {t _ {1: T}} \mid x _ {0}\right) \right] \tag {20}
$$
in general does not. Unlike in beta diffusion, where the transitions between  $z_{t}$  and  $z_{t-1}$  are applied multiplicatively, in Gaussian diffusion, the transitions between  $z_{t}$  and  $z_{t-1}$  are related to each other additively and  $z_{t_{1:T}}$  forms a Markov chain regardless of whether  $x_{0}$  is marginalized out.
The discretized forward beta diffusion chain given by (19) is reversible assuming knowing  $x_0$ . This means given  $x_0$ , it can be equivalently sampled in reverse order by first sampling  $z_{t_T} \sim q(z_{t_T} | x_0) = \mathrm{Beta}(\eta \alpha_{t_T} x_0, \eta (1 - \alpha_{t_T} x_0))$  and then repeating (8) for  $t_T, \ldots, t_2$ , with PDF  $q(z_{t_{1:T}} | x_0) = q(z_{t_T} | x_0) \prod_{j=2}^{T} \frac{1}{1-z_{t_j}} \mathrm{Beta}\left(\frac{z_{t_j-1}-z_{t_j}}{1-z_{t_j}}; \eta(\alpha_{t_j-1}-\alpha_{t_j})x_0, \eta(1-\alpha_{t_j-1}x_0)\right)$ . This non-causal chain, while not useful by itself, serves as a blueprint for approximate generation.
Specifically, we approximate the marginal given by (20) with a Markov chain in reverse order as
$$
p _ {\theta} \left(z _ {t _ {1: T}}\right) = p _ {p r i o r} \left(z _ {t _ {T}}\right) \prod_ {j = 2} ^ {T} p _ {\theta} \left(z _ {t _ {j - 1}} \mid z _ {t _ {j}}\right) = p _ {p r i o r} \left(z _ {t _ {T}}\right) \prod_ {j = 2} ^ {T} q \left(z _ {t _ {j - 1}} \mid z _ {t _ {j}}, f _ {\theta} \left(z _ {t _ {j}}, \alpha_ {t _ {j}}\right)\right). \tag {21}
$$
To start the reverse process, we choose to approximate  $q(z_{t_T}) = \mathbb{E}_{x_0\sim p_{data}(x_0)}[q(z_{t_T}\mid x_0)]$  with  $p_{prior}(z_{t_T}) = q(z_{t_T}\mid \mathbb{E}[x_0])$ , which means we let  $z_{t_T} \sim \mathrm{Beta}(\eta \alpha_{t_T}\mathbb{E}[x_0],\eta (1 - \alpha_{t_T}\mathbb{E}[x_0]))$ . To sample  $z_{t_{1:T-1}}$ , we use the remaining terms in (21), which are scaled and shifted beta distributions that are specified as in (8) and can be sampled as in (6) and (9).
KLUB for Discretized Beta Diffusion. An optimized generator is expected to make the " $x_0$ -free" joint distribution over all  $T$  steps of the discretized reverse beta diffusion chain, expressed as  $p_{\theta}(z_{t_{1:T}})$ , to approach that of the discretized forward chain, expressed as  $q(z_{t_{1:T}})$ . Thus an optimized  $\theta$  is desired to minimize the KL divergence  $\mathrm{KL}(p_{\theta}(z_{t_{1:T}})||q(z_{t_{1:T}}))$ . While this KL loss is in general intractable to compute, it can also be bounded using the KLUB shown as follows.
Lemma 6 (KLUB for discretized diffusion chain).  $\mathrm{KL}(p_{\theta}(z_{t_{1:T}})||q(z_{t_{1:T}}))$  is upper bounded by
$$
\operatorname {K L U B} = \mathbb {E} _ {x _ {0} \sim p _ {\text {d a t a}} (x _ {0})} \left[ \operatorname {K L} \left(p _ {\text {p r i o r}} \left(z _ {t _ {T}}\right) \mid \mid q \left(z _ {t _ {T}} \mid x _ {0}\right)\right) \right] + \sum_ {j = 2} ^ {T} \widetilde {\operatorname {K L U B}} _ {t _ {j - 1}, t _ {j}} \quad \text {w h e r e} \tag {22}
$$
$$
\widetilde {\mathrm {K L U B}} _ {s, t} = \mathbb {E} _ {(z _ {t}, x _ {0}) \sim p _ {\theta} (z _ {t} \mid x _ {0}) p _ {d a t a} (x _ {0})} [ \mathrm {K L U B} (s, z _ {t}, x _ {0}) ], \quad \mathrm {K L U B} (s, z _ {t}, x _ {0}) = \mathrm {K L} (p _ {\theta} (z _ {s} \mid z _ {t}) | | q (z _ {s} \mid z _ {t}, x _ {0})) ].
$$
We provide the proof in Appendix B. We note Lemma 6 is a general statement applicable for any diffusion models with a discrete forward chain  $q(z_{t_{1:T}} \mid x_0) = \prod_{j=1}^{T} q(z_{t_j} \mid z_{t_{j-1}}, x_0)$  and a discrete reverse chain  $p_\theta(z_{t_{1:T}}) = p_{prior}(z_{t_T}) \prod_{j=2}^{T} p_\theta(z_{t_{j-1}} \mid z_{t_j})$ . To estimate the KLUB in (22), however, during training, one would need to sample  $z_{t_j} \sim p_\theta(z_{t_j} \mid x_0) \propto p(x_0 \mid z_{t_j}) p_\theta(z_{t_j})$ , which is often infeasible. If we replace  $z_{t_j} \sim p_\theta(z_{t_j} \mid x_0)$  with  $z_{t_j} \sim q(z_{t_j} \mid x_0)$ , then  $\widetilde{\mathrm{KLUB}}_{s,t}$  becomes the same as  $\mathrm{KLUB}_{s,t}$  given by (11), and  $\mathrm{KLUB}_t$  given by (15) can be considered to remedy the impact of approximating  $p_\theta(z_{t_j})$  with  $q(z_{t_j})$ . Therefore, we can consider the combination of  $\mathrm{KLUB}_{s,t}$  given by (11) and  $\mathrm{KLUB}_t$  given by (15) as a computationally viable solution to compute  $\widetilde{\mathrm{KLUB}}_{s,t}$ , which hence justifies the use of the loss in (18) to optimize the discretized reverse beta diffusion chain. We summarize the training and sampling algorithms of beta diffusion in Algorithms 1 and 2, respectively.
2.5 Argument-swapped KLUBs and Negative ELBOs
We note that in theory, instead of the KL divergences in (11) and (15),  $f_{\theta}$  can also be optimized using two argument-swapped KL divergences:  $\mathrm{KL}(q(z_t,z_s)||p_\theta (z_s|z_t)q(z_t))$  and  $\mathrm{KL}(q(z_t')||p_\theta (z_t'))$ . By the same analysis, these KL divergences can also be bounded by KLUBs and log-beta divergences that are equivalent to the previous ones, but with swapped arguments. The argument-swapped KLUBs and log-beta divergences will be shown to be closely related to optimizing a discretized beta reverse
diffusion chain via -ELBO, but they do not guarantee an optimal solution that satisfies (14) in beta diffusion and are found to provide clearly inferior empirical performance.
Negative ELBO for Discretized Beta Diffusion. As an alternative to KLUB, one can also consider following the convention in diffusion modeling to minimize the negative ELBO, expressed as
$$
\begin{array}{l} - \mathbb {E} _ {x _ {0} \sim p _ {d a t a} (x _ {0})} \ln p _ {\theta} (x _ {0}) \leq - \mathrm {E L B O} = \mathbb {E} _ {x _ {0} \sim p _ {d a t a} (x _ {0})} \mathbb {E} _ {q (z _ {t _ {1: T}} | x _ {0})} \left[ - \ln \frac {p (x _ {0} | z _ {t _ {1 : T}}) p _ {\theta} (z _ {t _ {1 : T}})}{q (z _ {t _ {1 : T}} | x _ {0})} \right] \\ = -\mathbb{E}_{x_{0}}\mathbb{E}_{q(z_{t_{1}}\mid x_{0})}\ln p(x_{0}\mid z_{t_{1}}) + \mathbb{E}_{x_{0}}\mathbf{KL}[q(z_{t_{T}}\mid x_{0})||p_{prior}(z_{t_{T}})] + \sum_{j = 2}^{T}\mathbb{E}_{x_{0}}\mathbb{E}_{q(z_{t_{j}}\mid x_{0})}[L(t_{j - 1},z_{t_{j}},x_{0})], \\ \end{array}
$$
where the first two terms are often ignored and the focus is placed on the remaining  $T - 2$  terms as
$$
\begin{array}{l} L \left(t _ {j - 1}, z _ {t _ {j}}, x _ {0}\right) = \mathrm {K L} \left(q \left(z _ {t _ {j - 1}} \mid z _ {t _ {j}}, x _ {0}\right) \mid \mid q \left(z _ {t _ {j - 1}} \mid z _ {t _ {j}}, \hat {x} _ {0} = f _ {\theta} \left(z _ {t _ {j}}, t _ {j}\right)\right)\right) \\ = D _ {\ln B (a, b)} \left\{\left[ \eta \left(\alpha_ {t _ {j - 1}} - \alpha_ {t _ {j}}\right) f _ {\theta} \left(z _ {t _ {j}}, t _ {j}\right), \eta \left(1 - \alpha_ {t _ {j - 1}} f _ {\theta} \left(z _ {t _ {j}}, t _ {j}\right)\right) \right], \left[ \eta \left(\alpha_ {t _ {j - 1}} - \alpha_ {t _ {j}}\right) x _ {0}, \eta \left(1 - \alpha_ {t _ {j - 1}} x _ {0}\right) \right] \right\}. \tag {23} \\ \end{array}
$$
Lemma 7 (-ELBO and argument-swapped KLUB). Optimizing the generator  $f_{\theta}$  with -ELBO is equivalent to using an upper-bound for the augment-swapped KL divergence  $KL(q(z_{t_{1:T}}) || p_{\theta}(z_{t_{1:T}}))$ .
The proof in Appendix B relies on the convex nature of both the KL divergence and the negative logarithmic function. We find for beta diffusion, optimizing  $\mathrm{KL}(p_{\theta}(z_{t_{1:T}})||q(z_{t_{1:T}}))$  via the proposed KLUBs is clearly preferred to optimizing  $\mathrm{KL}(q(z_{t_{1:T}})||p_{\theta}(z_{t_{1:T}}))$  via -ELBOs (i.e., augment-swapped KLUBs) and leads to stable and satisfactory performance.
KLUBs and (Weighted) Negative ELBOs for Gaussian Diffusion. We note KLUBs are directly applicable to Gaussian diffusion, but they may not result in new optimization algorithms for Gaussian diffusion that drastically differ from the weighted ELBO, which weighs the KL terms using the corresponding SNRs. Moreover, whether the default or argument-swapped KLUBs are used typically does not make any difference in Gaussian diffusion and would result in the same squared error-based Bregman divergence. We provide the derivation of the (weighted) ELBOs from the lens of KLUBs in Appendix C, providing theoretical support for Gaussian diffusion to use the SNR weighted ELBO [23, 35, 20], which was often considered as a heuristic but crucial modification of ELBO.
3 Related Work, Limitations, and Future Directions
Various diffusion processes, including Gaussian, categorical, Poisson, and beta diffusions, employ specific distributions in both forward and reverse sampling. Gaussian diffusion starts at  $\mathcal{N}(0,1)$  in its reverse process, while both Poisson and beta diffusion start at 0. Beta diffusion's reverse sampling is a monotonically non-decreasing process, similar to Poisson diffusion, but while Poisson diffusion takes count-valued discrete jumps, beta diffusion takes probability-valued continuous jumps. A future direction involves extending beta diffusion to encompass the exponential family [6, 65, 45, 7].
Several recent works have explored alternative diffusion processes closely related to Gaussian diffusion. Cold diffusion by Bansal et al. [5] builds models around arbitrary image transformations instead of Gaussian corruption, but it still relies on  $L_{1}$  loss, resembling Gaussian diffusion's squared Euclidean distance. Rissanen et al. [52] propose an inverse heat dispersion-based diffusion process that reverses the heat equation using inductive biases in Gaussian diffusion-like models. Soft diffusion by Daras et al. [12] uses linear corruption processes like Gaussian blur and masking. Blurring diffusion by Hoogeboom and Salimans [25] shows that blurring (or heat dissipation) can be equivalently defined using a Gaussian diffusion process with non-isotropic noise and proposes to incorporate blurring into Gaussian diffusion. These alternative diffusion processes share similarities with Gaussian diffusion in loss definition and the use of Gaussian-based reverse diffusion for generation. By contrast, beta diffusion is distinct from all of them in forward diffusion, training loss, and reverse diffusion.
A concurrent work by Avdeyev et al. [3] utilizes the Jacobi diffusion process for discrete data diffusion models. Unlike Gaussian diffusion's SDE definition, the Jacobi diffusion process in Avdeyev et al. [3] is defined by the SDE  $dx = \frac{s}{2} [a(1 - x) - bx]dt + \sqrt{sx(1 - x)} dw$ , with  $x \in [0,1]$  and  $s, a, b > 0$ . The stationary distribution is a univariate beta distribution  $\mathrm{Beta}(a,b)$ . Beta diffusion and the Jacobi process are related to the beta distribution, but they differ in several aspects: Beta diffusion ends its forward process at  $\mathrm{Beta}(0,\eta)$ , a unit point mass at 0, not a  $\mathrm{Beta}(a,b)$  random variable. The marginal distribution of beta diffusion at time  $t$  is expressed as  $q(z_{t} \mid x_{0}) \sim \mathrm{Beta}(\eta \alpha_{t}x_{0},\eta (1 - \alpha_{t}x_{0}))$ , while the Jacobi diffusion process involves an infinite sum. Potential connections between beta diffusion and the Jacobi process under specific parameterizations are worth further investigation.
Several recent studies have been actively exploring the adaptation of diffusion models to constrained scenarios [13, 40, 41, 17], where data is bounded within specific ranges or constrained to particular manifolds. These approaches are all rooted in the framework of Gaussian diffusion, which involves the incorporation of additive Gaussian noise. In sharp contrast, beta diffusion introduces a distinct perspective by incorporating multiplicative noise, leading to the emergence of an inherently hypercubic-constrained diffusion model that offers new development and exploration opportunities.
Classifier-free guidance (CFG), often used in conjunction with heuristic clipping, is a widely used technique to perform conditional generation with Gaussian diffusion [22, 41]. Beta diffusion offers the potential for a seamless integration of CFG by directly applying it to the logit space, the operating space of its  $f_{\theta}$  network, thereby eliminating the necessity for heuristic clipping.
To adapt Gaussian diffusion for high-resolution images, a common approach is to perform diffusion within the latent space of an auto-encoder [53, 62]. A promising avenue to explore is the incorporation of sigmoid or tanh activation functions in the encoder's final layer. This modification would establish a bounded latent space conducive to applying beta diffusion, ultimately leading to the development of latent beta diffusion tailored for high-resolution image generation.
One limitation of beta diffusion is that its training is computationally expensive and data-intensive, akin to Gaussian diffusion. Specifically, with four Nvidia RTX A5000 GPUs, beta diffusion and Gaussian diffusion (VP-EDM) both take approximately 1.46 seconds to process 1000 images of size  $32 \times 32 \times 3$ . Processing 200 million CIFAR-10 images, the default number required to reproduce the results of VP-EDM, would thus take over 80 hours. Several recent works have explored different techniques to make Gaussian diffusion faster and/or more data efficient in training [68, 20, 76, 71]. It is worth exploring how to adapt these methods to enhance the training efficiency of beta diffusion.
Beta diffusion has comparable sampling costs to Gaussian diffusion with the same NFE. However, various methods have been developed to accelerate the generation of Gaussian diffusion, including combining it with VAEs, GANs, or conditional transport [77] for faster generation [72, 47, 78, 69], distilling the reverse diffusion chains [43, 55, 76], utilizing reinforcement learning [16], and transforming the SDE associated with Gaussian diffusion into an ODE, followed by fast ODE solvers [58, 39, 42, 75, 34]. Given these existing acceleration techniques for Gaussian diffusion, it is worth exploring their generalization to enhance the sampling efficiency of beta diffusion.
Beta diffusion raises concerns regarding potential negative societal impact when trained on image datasets curated with ill intentions. This issue is not exclusive to beta diffusion but applies to diffusion-based generative models as a whole. It is crucial to address how we can leverage these models for the betterment of society while mitigating any potential negative consequences.
The training and sampling algorithms for beta diffusion are described in detail in Algorithms 1 and 2, respectively, in the Appendix. Our experiments, conducted on two synthetic data and the CIFAR10 images, primarily aim to showcase beta diffusion's effectiveness in generating range-bounded data. We also underscore the superiority of KLUBs over negative ELBOs as effective optimization objectives for optimizing beta diffusion. Additionally, we highlight the differences between beta and Gaussian diffusion, specifically in whether the data are generated through additive or multiplicative transforms and their ability to model the mixture of range-bounded distributions with disjoint supports.
We compare the performance of "Gauss ELBO," "Beta ELBO," and "Beta KLUB," which respectively correspond to a Gaussian diffusion model optimized with the SNR weighted negative ELBO [23, 35], a beta diffusion model optimized with the proposed KLUB loss defined in (18) but with the two arguments inside each KL term swapped, and a beta diffusion model optimized with the proposed KLUB loss defined in (18). On CIFAR-10, we also evaluate beta diffusion alongside a range of non-Gaussian or Gaussian-like diffusion models for comparison.
4.1 Synthetic Data
We consider a discrete distribution that consists of an equal mixture of five unit point masses located at  $x_0 \in \mathcal{D} = \{1/7, 2/7, 3/7, 4/7, 5/7\}$ . We would like to highlight that a unit point mass can also be seen as an extreme case of range-bounded data, where the range is zero. Despite being simple,
Figure 3: Comparison of the true PMF, marked in red square  $\blacksquare$ , and the empirical PMFs of three different methods—Gauss ELBO, Beta ELBO, and Beta KLUB—calculated over 100 equal-sized bins between 0 and 1. Each empirical PMF is marked in solid dot  $\bullet$ .
this data could be challenging to model by a continuous distribution, as it would require the generator to concentrate its continuous-valued generations on these five discrete points.
We follow previous works to choose the beta linear diffusion schedule as  $\alpha_{t} = e^{-\frac{1}{2}\beta_{d}t^{2} - \beta_{\mathrm{min}}t}$ , where  $\beta_{d} = 19.9$  and  $\beta_{\mathrm{min}} = 0.1$ . This schedule, widely used by Gaussian diffusion [23, 61, 34], is applied consistently across all experiments conducted on synthetic data. We set  $\eta = 10000$ ,  $\pi = 0.95$ , and  $\omega = 0.5$ . As the data already falls within the range of 0 to 1, necessitating neither scaling nor shifting, we set  $S_{\text{scale}} = 1$  and  $S_{\text{shift}} = 0$ . We use the same structured generator  $f_{\theta}$  for both Gaussian and beta diffusion. We choose 20-dimensional sinusoidal position embeddings [63], with the positions set as  $1000t$ . The network is an MLP structured as (21-256)-ReLU-(256-256)-ReLU-(256-1). We utilize the Adam optimizer with a learning rate of 5e-4 and a mini-batch size of 1000.
For data generation, we set  $\mathrm{NFE} = 200$ . We provide the generation results in Figure 3, which shows the true probability mass function (PMF) of the discrete distribution and the empirical PMFs over 100 equal-sized bins between 0 and 1. Each empirical PMF is computed based on  $100\mathrm{k}$  random data points generated by the model trained after  $400\mathrm{k}$  iterations. It is clear from Figure 3 that "Gauss ELBO" is the worst in terms of mis-aligning the data supports and placing its data into zero-density regions; "Beta ELBO" is the worst in terms of systematically overestimating the density at smaller-valued supports; whereas "Beta KLUB" reaches the best compromise between accurately identifying the data supports and maintaining correct density ratios between different supports.
In Appendix E, we further provide quantitative performance comparisons between different diffusion models and conduct an ablation study between KLUB and its two variants for beta diffusion: "KLUB Conditional" and "KLUB Marginal," corresponding to  $\omega = 1$  and  $\omega = 0$ , respectively, in the loss given by (18). Additionally, we evaluate beta diffusion on another synthetic data, which comes from a mixture of range-bounded continuous distributions and point masses supported on disjoint regions.
4.2 Experiments on Image Generation
We employ the CIFAR-10 dataset and build upon VP-EDM [34] as the foundation of our codebase. Our initial foray into applying beta diffusion to generative modeling of natural images closely mirrors the settings of Gaussian diffusion, including the choice of the generator's network architecture. We introduce a sigmoid diffusion schedule defined as  $\alpha_{t} = 1 / (1 + e^{-c_{0} - (c_{1} - c_{0})t})$ , which has been observed to offer greater flexibility than the beta linear schedule for image generation. This schedule bears resemblance to the sigmoid-based one introduced for Gaussian diffusion [35, 29]. We configure the parameters for beta diffusion as follows:  $c_{0} = 10$ ,  $c_{1} = -13$ ,  $S_{shift} = 0.6$ ,  $S_{calc} = 0.39$ ,  $\eta = 10000$ ,  $\omega = 0.99$ , and  $\pi = 0.95$ . We utilize the Adam optimizer with a learning rate of 2e-4. We use EDM's data augmentation approach, but restrict augmented images to a 0-1 range before scaling and shifting. We use the beta diffusion model trained on 200M images to calculate the FID [21]. Explanations regarding the intuition behind these parameter selections can be located in Appendix G.
Below we provide numerical comparison of beta diffusion with not only Gaussian diffusion models but also alternative non-Gaussian or Gaussian-like ones. We also conduct an ablation study to compare KLUB and negative ELBO across different NFE,  $\eta$ , and  $B$ . A broad spectrum of diffusion models is encompassed in Table 1, which shows that beta diffusion outperforms all non-Gaussian diffusion models on CIFAR10, including Cold Diffusion [5] and Inverse Heat Dispersion [52], as well as categorical and count-based diffusion models [2, 8, 10]. In comparison to Gaussian diffusion and Gaussian+blurring diffusion, beta diffusion surpasses VDM [35], Soft Diffusion [12], and Blurring Diffusion [25]. While it may fall slightly short of DDPM [23], improved DDPM [46], TPDM+ [78], VP-EDM [34], it remains a competitive alternative that uses non-Gaussian based diffusion.
Table 1: Comparison of the FID scores of various diffusion models trained on CIFAR-10.
Diffusion Space	Model	FID (↓)
Gaussian	DDPM [23]	3.17
VDM [35]	4.00
Improved DDPM [46]	2.90
TDPM+ [78]	2.83
VP-EDM [34]	1.97
Gaussian+Blurring	Soft Diffusion [12]	3.86
Blurring Diffusion [25]	3.17
Deterministic	Cold Diffusion (image reconstruction) [5]	80.08 (deblurring)
Inverse Heat Dispersion [52]	8.92 ( inpainting)
18.96
Categorical	D3PM Gauss+Logistic [2]	7.34
τLDR-10 [8]	3.74
Count	JUMP (Poisson Diffusion) [10]	4.80
Range-bounded	Beta Diffusion	3.06